Simulating a Browser?

  • Thread starter Thread starter Davej
  • Start date Start date
Is this commonly done? I'm thinking it would be extremely useful.
Click to expand...

I think it is relative common to send HTTP requests from C# code
(WebClient or HttpWebRequest). Sometimes it fake headers to completely
look like a browser.

It is also possible to embed a web browser in a web form.

Arne
 
I think it is relative common to send HTTP requests from C#
code (WebClient or HttpWebRequest). Sometimes it fake
headers to completely look like a browser.

It is also possible to embed a web browser in a web form.

Arne
Click to expand...

Well, I'm interested in the very simple case of communicating with a
website using a barebones page it has set aside for this automated
purpose. Is that sort of thing pretty trivial to get working? Thanks.
 
Well, I'm interested in the very simple case of communicating with a
website using a barebones page it has set aside for this automated
purpose. Is that sort of thing pretty trivial to get working? Thanks.
Click to expand...

If you can use WebClient then it is rather trivial.

WebClient wc = new WebClient();
string html = wc.DownloadString(url);

Arne
 
If you can use WebClient then it is rather trivial.

WebClient wc = new WebClient();
string html = wc.DownloadString(url);

Arne
Click to expand...

So far I can see the file at the URL, but I can't get a response to a
post. The UploadString() example looks a bit too simple. Shouldn't the
data string have a format more like "name1=value1&name2=value2" ? Or
do I have to do some other setup stuff?

http://msdn.microsoft.com/en-us/library/d0d3595k(v=vs.110).aspx

Thanks.
 
So far I can see the file at the URL, but I can't get a response to a
post. The UploadString() example looks a bit too simple. Shouldn't the
data string have a format more like "name1=value1&name2=value2" ? Or
do I have to do some other setup stuff?

http://msdn.microsoft.com/en-us/library/d0d3595k(v=vs.110).aspx
Click to expand...

It is probably UploadString you need to use and you can send data
with that.

But if you want more control, then look at HttpWebRequest
(I can find an example if needed).

Arne
 
It is probably UploadString you need to use and you can send data
with that.

But if you want more control, then look at HttpWebRequest
(I can find an example if needed).

Arne
Click to expand...

I looked up HttpWebRequest but the constructor is marked "obsolete."
See...

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest(v=vs.110).aspx

I tried UploadString(), UploadData() and UploadValues() but the page
never seems to see the POST. It just responds with what looks like an
initial rendering.

Here is the UploadValues() code...

public String UploadValuesTest() {

WebClient client = new WebClient();

//Console.Write("\nPlease enter the URI to post data to :
");
string uriString = @"http://www.stlnetwork.net/
default.aspx";

// Create a new NameValueCollection instance to hold some
custom parameters to be posted to the URL.
NameValueCollection NameValPairs = new
NameValueCollection();

//<form name="form1" method="post" action="default.aspx"
id="form1">
//<input name="txtUserID" type="text" id="txtUserID" />
//<input name="txtPassword" type="password"
id="txtPassword" />
//<input type="submit" name="btnLogin" value="Login"
id="btnLogin" />

// Add necessary parameter/value pairs to the name/value
container.
//"txtUserID=Userabc txtPassword=Passwd1";
NameValPairs.Add("txtUserID", "Userabc");
NameValPairs.Add("txtPassword", "Passwd1");

// 'The Upload(String,NameValueCollection)' implicitly
method sets HTTP POST as the request method.
byte[] responseArray = client.UploadValues(uriString,
NameValPairs);

// Decode and display the response.
return Encoding.ASCII.GetString(responseArray);

}//end
 
I looked up HttpWebRequest but the constructor is marked "obsolete."
See...

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest(v=vs.110).aspx
Click to expand...

You use WebRequest.Create with a HTTP URL to create a HttpWebRequest.
I tried UploadString(), UploadData() and UploadValues() but the page
never seems to see the POST. It just responds with what looks like an
initial rendering.
Click to expand...

You need to be sure that you are POST'ing to the action URL
not the form URL.

Arne
 
You use WebRequest.Create with a HTTP URL to create a HttpWebRequest.


You need to be sure that you are POST'ing to the action URL
not the form URL.
Click to expand...

Ooops.

I just checked your comment.

//<form name="form1" method="post" action="default.aspx" id="form1">
//<input name="txtUserID" type="text" id="txtUserID" />
//<input name="txtPassword" type="password" id="txtPassword" />
//<input type="submit" name="btnLogin" value="Login" id="btnLogin" />

If that is an ASP.NET web forms page, then it becomes a bit
tricky.

You need to send the viewstate in your POST for things
to work properly.

Look at the form source and see the hidden field.

Arne
 
Ooops.

I just checked your comment.

//<form name="form1" method="post" action="default.aspx" id="form1">
//<input name="txtUserID" type="text" id="txtUserID" />
//<input name="txtPassword" type="password" id="txtPassword" />
//<input type="submit" name="btnLogin" value="Login" id="btnLogin" />

If that is an ASP.NET web forms page, then it becomes a bit
tricky.

You need to send the viewstate in your POST for things
to work properly.

Look at the form source and see the hidden field.

Arne
Click to expand...

Oh! I forgot about that hidden field! So I need to downloadData() and
then send the name/value pair of the viewstate back with the other
pairs?
 
Bingo. That was it. Much thanks! Should have been obvious to me but I
just wasn't thinking to look for hidden fields.
Click to expand...

I have some working test code now. The routines necessary to extract
the two keys in the hidden fields were rather bothersome. Now I need
to figure out how to implement the "non blocking" methods because this
process is pretty slow.
 
I have some working test code now. The routines necessary to extract
the two keys in the hidden fields were rather bothersome.
Click to expand...

Yes - calling web forms pages programmatically is just cumbersome.
Now I need
to figure out how to implement the "non blocking" methods because this
process is pretty slow.
Click to expand...

Well - you can either use threads/threadpool or go for the asynch
methods.

Arne
 
Yes - calling web forms pages programmatically is just cumbersome.


Well - you can either use threads/threadpool or go for the asynch
methods.

Arne
Click to expand...

Are there other better schemes that I should be investigating? I have
read a little about web services, but they seem to be all about
exchanging XML data. Compared to that the various WebClient methods
seem pretty powerful. I want to be able to upload and download jpg
files and some text records. It looks like the WebClient methods will
cover that. Thanks.
 
Are there other better schemes that I should be investigating?
Click to expand...

I don't think so.

For HTML parsing you could look at HTML Agility Pack.
I have
read a little about web services, but they seem to be all about
exchanging XML data.
Click to expand...

Or at least a format intended for an application and not a browser.
Compared to that the various WebClient methods
seem pretty powerful. I want to be able to upload and download jpg
files and some text records. It looks like the WebClient methods will
cover that.
Click to expand...

And if WebClient is not flexible enough then you can fallback to
HttpWebRequest.

Arne
 
Back
Top