HttpWebRequest problem

[Abhishek Srivastava]

Hello All,

I have written a program to download a text file from a web server. The
program uses HttpWebRequest Object.

My program runs once at 10:00 AM every day. The problem is that the
HttpWebRequest object internally uses the browser cache.

So when today my program runs, it doesn't download a fresh copy. But
picks the one which is already present in the browser's cache.

So when I process the download file, most of my processing fails because
that data is already present in my DB.

If I open internet explorer, enter the same URL and hit refresh couple
of times, the new version of the text file appears. Now if I run my
application, the application also runs fine.

So clearly HttpWebRequest object is using the cache of the browser.

Is there any possible way to avoid this?

This is a real PITA for me, because my scheduled job always fails and I
have to run the application manually.

I tried to open a raw TCP connection to the site but that failed too...
because I live in a firewall protected network. So the only way to
connect to a external site is via the proxy server.

Please help me with this. I have been looking for a solution to this
problem for a long time now.

Can someone, who really knows the internals of HttpWebRequest confirm
whether the internal implementation of this class uses the browser's
cache and how to avoid this feature?

regards,
Abhishek.

 

[Morten Wennevik]

Hi Abhishek Srivastava,

If you are only going to download the file, you can use a WebClient
instead and its DownloadFile or DownloadData method.

WebClient client = new WebClient();
Byte[] pageData =
client.DownloadData("http://your.site.here/thetextfile.txt");
//or
client.DownloadFile("http://your.site.here/thetextfile.txt",
localFileName);

 

[Nicholas Paldino [.NET/C# MVP]]

Abhishek,

The HttpWebRequest object does not use the IE cache internally to return
results. It estabishes the connection to the server every time. Chances
are you are seeing the results of a cached document by the proxy, which is
giving you the results.

If you add the following headers, your proxy should acknowlegdge one of
them, and not give you a cached document or cache it internally:

Pragma: no-cache
Cache-Control: no-cache
Expires: Sat, 20 Sep 2000 01:56:59 GMT (this is some date in the past).

Hope this helps.