webpage download?

  • Thread starter Thread starter Alan Zhong
  • Start date Start date
A

Alan Zhong

i wonder how to download a webpage automatically using a C# program
with WebClient class.
i tried it and it's working. but somehow the data it gets is different
than the page shown in microsoft internet explorer.
and i wonder how to get around the login problems. the C# programs
only downloads the login page, do i need to "instruct" the program to
access the cookie information on my window?
thankx
 
Alan said:
i wonder how to download a webpage automatically using a C# program
with WebClient class.
i tried it and it's working. but somehow the data it gets is different
than the page shown in microsoft internet explorer.
and i wonder how to get around the login problems. the C# programs
only downloads the login page, do i need to "instruct" the program to
access the cookie information on my window?
Click to expand...

WebClient is not sufficient to implement HTTP conversations, since it
doesn't support cookies. You must use HttpWebRequest and HttpWebResponse
instead.

Cheers,
 
thankx, i had tried to use httpwebrequest.
but one more question:
NetworkCredential(username, password); is not working (still cannot
pass the authorization).
i wonder if i need to use CookieContainer. when i visit this page
(after logged in), there is a new cookie C:\Documents and
Settings\Administrator\[email protected][1].txt
how can i retrieve the cookie infomation and apply it to the
HttpWebRequest.CookieContainer?
and is it the correct approach?
thankx a lot.
 
hi everybody, i am still trying to get this work...
here were what i did.
1) use HttpWebRequest to connection the server with username +
password. call getResponse, successfully get the response header
containing the ["Set-Cookie"] data.
2) put ["Set-Cookie"] into the headers of a newly created request with
the website i want to download under the same protected domain.
3) call getResponose again but it still redirects me to the login
page.

what did i do wrong?
the following are the codes i have:
////////////////////////// test12.cs
using System;
using System.IO;
using System.Net;
class test12
{
static void Main(string[] args)
{
String address = "xxx login page with username and password xxx";
String newaddr = "page wanted to download";

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(address);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();

request = (HttpWebRequest) WebRequest.Create(newaddr);
// response headers info:
// Header Name:Server, Value :Sun-ONE-Web-Server/6.1
// Header Name:Date, Value :Thu, 01 Apr 2004 21:12:48 GMT
// Header Name:Content-type, Value :text/html
// Header Name:Set-cookie, Value :RMID=xxxxx; expires=Friday,
// 01-Apr-2005 21:12:48 GMT; path=/; domain=.xxx.com
// Header Name:Cache-control, Value :no-cache
// Header Name:Pragma, Value :no-cache
// Header Name:Transfer-encoding, Value :chunked
request.Headers.Add("Cookie", response.Headers["Set-Cookie"]);

Console.WriteLine("old Address: " + request.Address);
response = (HttpWebResponse) request.GetResponse();
Console.WriteLine("new Address: " + request.Address);
Console.WriteLine("ResponseUri " + response.ResponseUri);

StreamReader sr = new StreamReader(response.GetResponseStream());
String result = sr.ReadToEnd();
sr.Close();

response.Close();

Console.WriteLine(result.Length);
if (result.Length > 1000)
result = result.Substring(0, 1000);
Console.WriteLine(result);
}
 
Back
Top