Very Fast Multithreaded URL Fetching

  • Thread starter Thread starter Arsen V.
  • Start date Start date
A

Arsen V.

Hello,

Does anyone know if an object or how to create one, that will allow me to
fetch up to 10 URLs (containing XML-feed data) in an extremelly fast server
side fashion?

If the request is taking longer than it should, the object would need to be
able to timeout on the spot (without waiting). The timeout value would need
to be acurate to about 50 milliseconds. Usually, I would want the timeout to
be around 1.5 seconds.

Thanks in advance,
Arsen
 
Arsen,

Suppose the first question is why does it need to be multithreaded?

Running it in a thread possibly fair enough but a simple loop should
suffice, no point complicating something that doesnt require it ;-)

Look at WebRequest class. You can set the timeout as a property. Feed the
WebRequest.Create(URI) from values in a loop from an array of 10 that holds
your urls.

Tam
 
There are a couple of ways of doing this. I think the simplest is to use the
HttpWebRequest object. In your loop you would do:

foreach (string url in urls)
{
HttpWebRequest request = new HttpWebRequest(...);
request.BeginGetResponse(...);
}

that will queue up all the requests. In the call to BeginGetResponse(), you
can specify a callback that will be executed when the requests finish. In
that function you'll get EndGetResponse() which will get you a WebResponse
object with the data you want.

--
Eric Gunnerson

Visit the C# product team at http://www.csharp.net
Eric's blog is at http://weblogs.asp.net/ericgu/

This posting is provided "AS IS" with no warranties, and confers no rights.
 
Hi Eric,

Thanks for the suggestion.

Would I be able to achive performance of the Web Application Stress tool
using the Async HttpWebRequest or do I have to use sockets in multiple
threads?

Also, in your example how do I implement the timeout (needs to be able to
timeout in 500ms - 2000ms).

Also, how do I wait for either the timeout or all of the Async requests
completing?

Thanks,
Arsen
 
Arsen V. said:
Also, in your example how do I implement the timeout (needs to be able to
timeout in 500ms - 2000ms).
Click to expand...

Multiple threads is much easier, and use blocking sockets. Sure async is
faster - but unless you have a realy dog of a machine even blocking sockets
can easily saturate a 100 MB ethernet.


--
Chad Z. Hower (a.k.a. Kudzu) - http://www.hower.org/Kudzu/
"Programming is an art form that fights back"


ELKNews - Get your free copy at http://www.atozedsoftware.com
 
Back
Top