Httprequest problems

  • Thread starter Thread starter Dales
  • Start date Start date
D

Dales

I'm trying to screen scrape for some data, but the response that comes
back is the "Browser Detect" page. Is there any way to "impersonate"
a browser (IE6.0 for example)?
 
The HttpWebRequest Class has properties for "Referer", "ContentType", and "Method". If you set them to more closely resemble that of
a browser, you would most likely achieve what you are after.

If you don't know what to set, you can use your machine to make a web request, with a browser and use a sniffer to look at it.
 
Hi,

I have done more research regarding this issue and found some useful
articles for your reference.

HOW TO: Make a GET Request by Using C# .NET
http://support.microsoft.com/default.aspx?scid=kb;en-us;307023

HOW TO: Make a GET Request by Using Visual Basic .NET
http://support.microsoft.com/default.aspx?kbid=301102

We can also add headers with the specified name and value. Please refer to
the following URL.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/
frlrfsystemnetwebheadercollectionclassaddtopic2.asp
"...
Try
'Create a web request for "www.msn.com".
Dim myHttpWebRequest As HttpWebRequest =
CType(WebRequest.Create("http://www.msn.com"), HttpWebRequest)

'Get the headers associated with the request.
Dim myWebHeaderCollection As WebHeaderCollection =
myHttpWebRequest.Headers

Console.WriteLine("Configuring Webrequest to accept Danish and
English language using 'Add' method")

'Add the Accept-Language header (for Danish) in the request.
myWebHeaderCollection.Add("Accept-Language:da")

'Include English in the Accept-Langauge header.
myWebHeaderCollection.Add("Accept-Language:en;q" + ChrW(61) + "0.8")

'Get the associated response for the above request.
Dim myHttpWebResponse As HttpWebResponse =
CType(myHttpWebRequest.GetResponse(), HttpWebResponse)
..."

I hope it helps.

Best regards,

Jacob Yang
Microsoft Online Partner Support
Get Secure! ¨C www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.
 
Dales said:
I'm trying to screen scrape for some data, but the response that
comes back is the "Browser Detect" page. Is there any way to
"impersonate" a browser (IE6.0 for example)?
Click to expand...

Sure. Browser detection is based on the HTTP User-Agent header. You can
set this through HttpWebRequest's UserAgent property:

string url = "http://host/browsersniffer/";
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);

// That's what my IE 6 om Win XP with .NET 1.0 and 1.1 installed sends
request.UserAgent =
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR" +
"1.0.3705; .NET CLR 1.1.4322)";

HttpWebResponse response = (HttpWebResponse) request.GetResponse();


To find out appropriate User-Agent strings, I recommend writing a simple
ASP.NET, ASP, JSP, Servlet, PHP , [insert your favorite serverside
technology] that simply dumps all HTTP request headers. If all else
fails, clientside scripting will work as well. There are also web sites
that display live HTTP request information -- see
http://www4.ncsu.edu/~awwatkin/SHTML/dump.shtml

Cheers,
 
Back
Top