Incomplete WebResponse?

  • Thread starter Thread starter Chris Coddington
  • Start date Start date
C

Chris Coddington

I make a request, stuff it into a stringbuffer, then append it to a
textbox. It works on many sites (e.g. yahoo.com), but not on many
others (e.g. amazon.com). Any insight that could be provided would be
greatly appreciated.

Relevant code:

Sub GetPage()
Try

Dim loRequest As HttpWebRequest
Dim loResponse As HttpWebResponse
Dim loSR As StreamReader
Dim loSB As New StringBuilder

loRequest = CType(WebRequest.Create("http://www.amazon.com"),
HttpWebRequest)
loResponse = CType(loRequest.GetResponse(), HttpWebResponse)
loSR = New StreamReader(loResponse.GetResponseStream,
Encoding.ASCII)
loSB.Append(loSR.ReadToEnd)
loSR.Close()

txtStatus.AppendText(loSB.ToString)

Catch ex As Exception
MessageBox.Show(ex.ToString)
End Try

End Sub
 
Hi Chris,

Probably an easy one,
A lot of site pages exist as seperate documents.
A frame containerdocument and framedocuments.

So when you try to get them as you do, you get the frame container, try to
find the seperate frames one by one.

I hope this helps?

Cor
 
When I say that it doesn't work, I mean that I get a response from the
webserver but I don't get the complete page. What am I missing?
 
I'll get a random number of bytes (never consistent) and I'll never
get the closing body and html tags. It's as though I only get half the
html. I have tinkered with the timeout but it never throws an
exception - of any kind.
 
Hi Chris,

And it is sure a HTML page not something as a bytearrea?

Maybe you can download the file doing a webclient download to see the file.
\\\
Dim wbc As New System.Net.WebClient
wbc.DownloadFile(Uri, Filename)
///
Than you can see the file normaly

Otherwise I think that we are just guessing

I hope this helps?

Cor
 
Hello Cor,

Well, that worked. But I'd like to eliminate the need to use the
local filesystem. Any other ways to handle this?

thx - Chris
 
HI Chris,

Is the file a real HTML page with a start HTML tag and an end HTML tag?

Cor
 
Yes, I get the entire page. When I use any of the other methods of
pulling it down that return a stream, I don't get the entire page.
I've tried:

WebClient.OpenRead
WebClient.DownloadData
WebRequest.GetResponse

Only WebClient.DownloadFile works consistently. The other three
methods work on other sites, but for whatever reason, amazon.com gives
me a problem. The symptoms:

1. The result comes back quickly.
2. No exceptions are thrown.
3. The page is never complete.
4. The cutoff point is never in the same spot in the page.
5. The size of the stream is never the same.
6. When "sniffing" the underlying protocol using Ethereal, it cuts off
just as it does in the software.
7. The same problem occurs on other computers that I have.

Arrrrrrgh! Why just amazon.com?

thx - Chris
 
Hi Chris,

This did work for me (see that I have used the RichTextbox, the textbox does
not)

Will you tell if you with this keeps the problems?

Cor
\\\
Dim myReg As Net.HttpWebRequest = _
DirectCast(Net.WebRequest.Create("http://www.amazon.com"), _
Net.HttpWebRequest)
Dim myResp As Net.HttpWebResponse = _
DirectCast(myReg.GetResponse(), Net.HttpWebResponse)
Dim myStream As IO.Stream = myResp.GetResponseStream()
Dim myreader As New IO.StreamReader(myStream)
Dim mystring As String = myreader.ReadToEnd()
Me.RichTextBox1.Text = mystring
myResp.Close()
///
 
Cor - you are a freaking genious! GENIOUS! I would like to kindly
repay you for your help. Shoot me an email plz!

I swapped out my textbox for the richtext box and whammo it worked.

thx - Chris
 
Back
Top