Requesting web pages

  • Thread starter Thread starter Richard Page
  • Start date Start date
R

Richard Page

I have built a basic web crawler in C# using the HttpWebRequest class. It
works fine except in one case where I can only get a custom "content loading
...." message from the server app (an asp.net app as it happens) , althought
the page actually contains a lot of data.

I can view it perfectly in IE6, but I cannot capture it in my crawler. I
get the same clipped response when I browse the site with Sam Spade.
Obviously IE is a bit smarter than my app, the question is why?

My HttpWebResponse object does not report any redirection and the status
code is HttpStatusCode.OK. My guess is the aspx page includes a
server.transfer statement which my app cannot follow and interprets as a
connection closed message.

Am I missing something in Http?


Richard
 
Richard Page said:
I have built a basic web crawler in C# using the HttpWebRequest class.
It works fine except in one case where I can only get a custom
Click to expand...

Please dont post the same message on multiple groups. Your other copy has already been
followed up.


--
Chad Z. Hower (a.k.a. Kudzu) - http://www.hower.org/Kudzu/
"Programming is an art form that fights back"

Develop ASP.NET applications easier and in less time:
http://www.atozed.com/IntraWeb/
 
Back
Top