time consuming issue with a HttpWebRequest and StreamReader

[z. f.]

Hi,

i have a vb.net windows application that makes HTTP requests with the framework's HttpWebReqeuest object.
i have about 10 different requests that takes between 30 to 500 for the Request.GetResponse function to execute.
this is acceptable.
problem is that for 1 of the 10 urls i request, when i call the

responseBody = reader.ReadToEnd() & ""

it takes about 11.50 seconds (!) to execute and return the responseBody.
for all the resources i request including the misbehaved one, the return content length is between 1000 and 40,000 bytes, and for the misbehaved one 8063 bytes. (should not cause any problems)

the misbehaved request is from a dot.net aspx page.
the request takes about 46 milliseconds to return a 8063 bytes of text/html (not binary)

i have other aspx resources in my list that are behaving OK. for example a 92,470 bytes response takes about 30 milliseconds to execute and for the ReadToEnd another 671 milliseconds.


the reader definition looks like that:
reader = New StreamReader(res.GetResponseStream(), System.Text.Encoding.GetEncoding("utf-8"), True)


i hope someone will have some direction to look for the problem source.

TIA, z.

 

[Joerg Jooss]

Hi,

i have a vb.net windows application that makes HTTP requests with the
framework's HttpWebReqeuest object. i have about 10 different
requests that takes between 30 to 500 for the Request.GetResponse
function to execute. this is acceptable. problem is that for 1 of
the 10 urls i request, when i call the

responseBody = reader.ReadToEnd() & ""

Out of curiosity -- why do append an empty string?

it takes about 11.50 seconds (!) to execute and return the
responseBody. for all the resources i request including the
misbehaved one, the return content length is between 1000 and 40,000
bytes, and for the misbehaved one 8063 bytes. (should not cause any
problems)

How did you obtain this value? By measuring the number of bytes
received or by relying on the Content-Length header?

the misbehaved request is from a dot.net aspx page.
the request takes about 46 milliseconds to return a 8063 bytes of
text/html (not binary)

So that's basiscally the time it takes to execute GetReponseStream(),
right?

i have other aspx resources in my list that are behaving OK. for
example a 92,470 bytes response takes about 30 milliseconds to
execute and for the ReadToEnd another 671 milliseconds.


the reader definition looks like that:
reader = New StreamReader(res.GetResponseStream(),
System.Text.Encoding.GetEncoding("utf-8"), True)


i hope someone will have some direction to look for the problem
source.

Have a look at this thread for a discussion of decoding-on-the-fly
(what you do) versus "raw" reading -- maybe a reading first and
decoding later delivers better results:

http://groups.google.de/groups?hl=de&lr=&client=firefox-a&threadm=MPG.1c
31f46243ba52bd98bafb%40msnews.microsoft.com&rnum=1&prev=/groups%3Fq%3Djo
oss%2Bskeet%2BStreamReader%26hl%3Dde%26lr%3D%26client%3Dfirefox-a%26selm
%3DMPG.1c31f46243ba52bd98bafb%2540msnews.microsoft.com%26rnum%3D1

Cheers,

 

[z. f.]

thanks, i think that the direction of the parsing might be the point of the problem, though 11.50 seconds to execute on a 8000 bytes is not acceptable.
other questions - see inline:

Hi,

i have a vb.net windows application that makes HTTP requests with the
framework's HttpWebReqeuest object. i have about 10 different
requests that takes between 30 to 500 for the Request.GetResponse
function to execute. this is acceptable. problem is that for 1 of
the 10 urls i request, when i call the

responseBody = reader.ReadToEnd() & ""

Out of curiosity -- why do append an empty string?

i don't remember the reason i did this a year ago, but even if i remove the empty string addition the result is the same.

it takes about 11.50 seconds (!) to execute and return the
responseBody. for all the resources i request including the
misbehaved one, the return content length is between 1000 and 40,000
bytes, and for the misbehaved one 8063 bytes. (should not cause any
problems)

How did you obtain this value? By measuring the number of bytes
received or by relying on the Content-Length header?

i look for the content-length header, if i don't have one, i check the length of the data.
usually responseBody.length

the misbehaved request is from a dot.net aspx page.
the request takes about 46 milliseconds to return a 8063 bytes of
text/html (not binary)

So that's basiscally the time it takes to execute GetReponseStream(),
right?

the 46 milliseconds value if for the request.GetResponse function to execute.

i have other aspx resources in my list that are behaving OK. for
example a 92,470 bytes response takes about 30 milliseconds to
execute and for the ReadToEnd another 671 milliseconds.


the reader definition looks like that:
reader = New StreamReader(res.GetResponseStream(),
System.Text.Encoding.GetEncoding("utf-8"), True)


i hope someone will have some direction to look for the problem
source.

Have a look at this thread for a discussion of decoding-on-the-fly
(what you do) versus "raw" reading -- maybe a reading first and
decoding later delivers better results:

http://groups.google.de/groups?hl=de&lr=&client=firefox-a&threadm=MPG.1c
31f46243ba52bd98bafb%40msnews.microsoft.com&rnum=1&prev=/groups%3Fq%3Djo
oss%2Bskeet%2BStreamReader%26hl%3Dde%26lr%3D%26client%3Dfirefox-a%26selm
%3DMPG.1c31f46243ba52bd98bafb%2540msnews.microsoft.com%26rnum%3D1

Cheers,

 

[z. f.]

interesting....
i noticed that the missbehaved request had an exception in the asp.net code.
the page flushed some HTML comments and then the asp.net framework wrote the exception details.
for some reason this small page ( less than 8 KB) is taking about 11.50 (constantly) for the streamReader.ReadToEnd with a unicode encoding.
strange.



thanks, i think that the direction of the parsing might be the point of the problem, though 11.50 seconds to execute on a 8000 bytes is not acceptable.
other questions - see inline:

Hi,

i have a vb.net windows application that makes HTTP requests with the
framework's HttpWebReqeuest object. i have about 10 different
requests that takes between 30 to 500 for the Request.GetResponse
function to execute. this is acceptable. problem is that for 1 of
the 10 urls i request, when i call the

responseBody = reader.ReadToEnd() & ""

Out of curiosity -- why do append an empty string?

i don't remember the reason i did this a year ago, but even if i remove the empty string addition the result is the same.

it takes about 11.50 seconds (!) to execute and return the
responseBody. for all the resources i request including the
misbehaved one, the return content length is between 1000 and 40,000
bytes, and for the misbehaved one 8063 bytes. (should not cause any
problems)

How did you obtain this value? By measuring the number of bytes
received or by relying on the Content-Length header?

i look for the content-length header, if i don't have one, i check the length of the data.
usually responseBody.length

the misbehaved request is from a dot.net aspx page.
the request takes about 46 milliseconds to return a 8063 bytes of
text/html (not binary)

So that's basiscally the time it takes to execute GetReponseStream(),
right?

the 46 milliseconds value if for the request.GetResponse function to execute.

i have other aspx resources in my list that are behaving OK. for
example a 92,470 bytes response takes about 30 milliseconds to
execute and for the ReadToEnd another 671 milliseconds.


the reader definition looks like that:
reader = New StreamReader(res.GetResponseStream(),
System.Text.Encoding.GetEncoding("utf-8"), True)


i hope someone will have some direction to look for the problem
source.

Have a look at this thread for a discussion of decoding-on-the-fly
(what you do) versus "raw" reading -- maybe a reading first and
decoding later delivers better results:

http://groups.google.de/groups?hl=de&lr=&client=firefox-a&threadm=MPG.1c
31f46243ba52bd98bafb%40msnews.microsoft.com&rnum=1&prev=/groups%3Fq%3Djo
oss%2Bskeet%2BStreamReader%26hl%3Dde%26lr%3D%26client%3Dfirefox-a%26selm
%3DMPG.1c31f46243ba52bd98bafb%2540msnews.microsoft.com%26rnum%3D1

Cheers,

 

[Joerg Jooss]

thanks, i think that the direction of the parsing might be the point
of the problem, though 11.50 seconds to execute on a 8000 bytes is
not acceptable.

Certainly not. Maybe there's a problem with the UTF8Decoder, so trying
to receive the raw bytes first should reveal quickly whether this is
true or not.

Out of curiosity -- why do append an empty string?

i don't remember the reason i did this a year ago, but even if i
remove the empty string addition the result is the same.

That's what I expected ;-)

How did you obtain this value? By measuring the number of bytes
received or by relying on the Content-Length header?

i look for the content-length header, if i don't have one, i check
the length of the data. usually responseBody.length

OK. Did you verify that the Content-Length is correct? Sending an
incorrect Content-Length header can produce weird results depending on
the receiver's implementation.

<remark>
Don't get me, wrong, but please learn to use a newsreader. Replying to
your post is a pain!
</remark>



Cheers,

 

[Guest]

IMO there is no problem in your vb.net code that requests aspx page through
HttpWebRequest.

In general time taken by ASPX, also consists of the time it takes to get
result from database. Check the URL in question, gets the results from the
Database or not if possible check how much time the ASPX takes when you
access it using browser, for this you can enable Trace.