a simple crawler

  • Thread starter Thread starter GeezerButler
  • Start date Start date
G

GeezerButler

Hi, I am trying to write a simple crawler. I have no previous
experience in bots so beware of stupid questions

The url i want to connect to is "http://someSiteName/landing.aspx?
prg=ABC&prg=DEF&prg=GHI"
When i type this in a browser, the server redirects to some other page
(http://someSiteName/otherPage.aspx) and the page gets opened but when
i try to connect by the WebRequest class, i am able to get the
response but it contains some custom error message of that site.

So my question is that what is it that the browser can handle but the
WebRequest class cant?
 
Hard to say for sure, but some sites block requests that don't have a
UserAgent header, for example.

--
HTH,

Kevin Spencer
Chicken Salad Surgeon
Microsoft MVP
 
It is also possible that the web browser is being redirected via javascript
rather than Http Headers..
 
Thanks folks but i wrote a windows app with a
System.Windows.Forms.WebBrowser class which made this a whole lot
easier than the console app i was trying to write
 
Back
Top