"proper" way to download several hundred text files from a web-site ?

  • Thread starter Thread starter Bill Woodruff
  • Start date Start date
B

Bill Woodruff

As an exercise I wrote a small C# program to download several hundred text files of guitar tab music by parsing the home page of the
site and retrieving all the links to the music text files and using 'WebRequest and a StreamReader to download them.

I followed up the call to WebRequest.Create with a call to Application.DoEvents, and called Application.DoEvents() again after the
call to ReadToEnd on the StreamReader.

What I was looking for was some kind of delay mechanism to ensure that I was not making requests to the site too frequently. Is
there any way to ask a site something like : "Can I have another helping please, or would you rather I come back later ?."

Appreciate any clues.

thanks, Bill
 
What I was looking for was some kind of delay mechanism to ensure that
I was not making requests to the site too frequently. Is there any way
to ask a site something like : "Can I have another helping please, or
would you rather I come back later ?."

Appreciate any clues.

thanks, Bill
Click to expand...

The RFC says no more than 4 connections to any given HTTP 1.0 server at any
given moment, and no more than 2 to any given HTTP 1.1 server at any
moment. As long as you don't break that rule (and I think that the
WebRequest class respects IE's settings, so if you haven't changed them
with some sort of "accellerator", those are defaults) you should be fine.

If you want to introduce more delay, then Thread.Sleep() is always good, as
long as you don't do it in UI threads (but you wouldn't do work in UI
threads, would you? that would be bad).

Mark
 
Back
Top