Fetching URLs in Parallel

  • Thread starter Thread starter Arsen V.
  • Start date Start date
A

Arsen V.

Hello,

How does an application like the Web Application Stress tool work? How does
it issue requests so fast without being blocked by slow executing ones?

What technique can I use to create an object that would fetch up to 10 URLs
in parallel and be able to timeout and terminate all the requests within 1.5
seconds?

This object would be used on the server side and would be called many times
per second.

Is there a component like this for sale?

Thanks,
Arsen
 
Spawn new threads based on the number of virtual users. That way no thread
blocks another. On a single machine, you can still experience some slow down
compared to multiple machines, but you are still likely to spawn more
requests than real users. Hope this helps.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

*************************************************
Think outside the box!
*************************************************
 
Hi,

But how to instantenuously kill threads that are blocking after the 1.5
timeout?

Thanks,
Arsen
 
Hi Angelos,

What is the sure way to "kill" a thread even if it is currently blocking?

I know that Abort() will simply tell the thread to start terminating, but it
may not happen instantly.

Thanks,
Arsen
 
If you are using the WebRequest class to fetch the URLs, then the
WebRequest class has a Timeout property.
 
Hi Scott,

The WebRequest's timeout property is not "instantenuous". In other words,
even if I specify timeout of 1500 milliseconds, the requests can take a long
time - 5 seconds or longer. I think this has to do with gracefully
terminating everything related to the thread. I just want to drop it if it's
taking too long.

Thanks,
Arsen
 
We had this problem in the .NET Terrarium. If you are lullygagging in
unmanaged code somewhere, then the thread won't abort. The only
sure fire way to make this work is to kill your thread using Win32 APIs.
I don't recommend this approach unless you really need, as in our case
we did. Instead think of models where you terminate the thread and set
some state related to the thread and let it go. It'll get cleaned up later.
If it returns after 5 seconds, then who cares, it won't affect your
application when it finally does die or completes.
 
Hi Justin,

The problem is that this component would need to be called from an ASP.NET
application getting dozens of hits per second. It would be pretty bad to
have all these "orphaned" threads executing.

Is there some .NET component that allows to fetch URLs in parallel and
control the timeout in a very precise way?

If it would be just a Windows Forms apps, I would not worry that much about
letting threads run and terminate. However, since this is going to be high
volume component I think it would cause performance problems.

What do you think?

Thanks,
Arsen
 
Hi Arsen:

There is no built in class to do this. Sometimes you just have to put
together a prototype and stress test it to see what works best in your
environment.
 
Hi Arsen,

Not sure whether there're any such 3rd party components. But generally, I
think creating multi-threads and request the page with WebRequest class is
ok. The Timeout property of webrequest is the length of time, in
milliseconds, until the request times out and then throw out a
WebException, so there maybe some deplay from the request timeout until we
abort the test thread. But this is still a reliable means.

In addition, as you mentioned that
=========================
The problem is that this component would need to be called from an ASP.NET
application getting dozens of hits per second.
===========================
do you mean you'll do such a stress test in an asp.net web application? If
so, I don't think this is a good idea since asp.net web application will
has limited num of free threads by default and those free threads are
cached for some internal calls from the other asp.net workerthreads so as
to avoid deaklock. If we perform such stress test which may create large
number of separate threads, that'll make the application have very poor
performance. Is there any particular requirement that the stress test need
to be done in a web application rather than a destop app?

Thanks.

Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)
 
Hi Steven,

The application must be a web ASP.NET app. It will be kind of a META-SEARCH.
It will connect to multiple sources in parallel and fetch results from each
source.

It cannot be a desktop app.

I was experimenting with using async socket communication and
ManualResetEvent with WaitAll(timeout) to guarantee a very precise timeout
control. However, I was hoping that there is a way to do this without using
sockets directly.

I also do not think that it is a good to spawn multiple threads from one
ASP.NET request. So what is the solution? Does ASYNC IO still span multiple
threads?

Thanks,
Arsen
 
Arsen V. said:
I was experimenting with using async socket communication and
ManualResetEvent with WaitAll(timeout) to guarantee a very precise timeout
control. However, I was hoping that there is a way to do this without using
sockets directly.
Click to expand...


Can't you do Async requests and accomplish the same thing with the HttpWebRequest?

I also do not think that it is a good to spawn multiple threads from one
ASP.NET request.
Click to expand...

Why not?
 
Back
Top