working with queues...

  • Thread starter Thread starter Ricardo
  • Start date Start date
R

Ricardo

Hi...

I'm doing a program that analise a web page for a string, if found it
register that page, and if it found any links on that page, it follows the
link and searsh it for the string, and so on.

The problem is that I want to delimit this process to only 5 pages to be
processed at a time. I think of using a queue, and when the one of the 5
treads end processing the page, pick another link on the queue and start
processing again.

But how can I do that??? I need some mechanism that tells me when a thread
finished so I can start another...

Do you all have any idea that can help me???


[]s...
 
Well, your threads could check the queue when they are finished doing an
item, and if there is still data, process it.

Or you could write your own full threadpool.

However, why you want to limit to five threads? Why don't you use the
default System.Threading.ThreadPool?

-mike
MVP
 
how this threadpool works??


Michael Giagnocavo said:
Well, your threads could check the queue when they are finished doing an
item, and if there is still data, process it.

Or you could write your own full threadpool.

However, why you want to limit to five threads? Why don't you use the
default System.Threading.ThreadPool?

-mike
MVP

Ricardo said:
Hi...

I'm doing a program that analise a web page for a string, if found it
register that page, and if it found any links on that page, it follows the
link and searsh it for the string, and so on.

The problem is that I want to delimit this process to only 5 pages to be
processed at a time. I think of using a queue, and when the one of the 5
treads end processing the page, pick another link on the queue and start
processing again.

But how can I do that??? I need some mechanism that tells me when a thread
finished so I can start another...

Do you all have any idea that can help me???


[]s...
Click to expand...
Click to expand...
 
The ThreadPool works by maintaining it's own internal queue of work items
that need to be completed. You can add as many items as you would like and
they will be processed as threads become available for them. MS has done
loads of performance testing on the optimum number of threads to run in the
default pool, and so the default pool should work fine for any general solution
thread based computing problems (such as the one you proposed).


--
Justin Rogers
DigiTec Web Consultants, LLC.
Blog: http://weblogs.asp.net/justin_rogers

Ricardo said:
how this threadpool works??


Michael Giagnocavo said:
Well, your threads could check the queue when they are finished doing an
item, and if there is still data, process it.

Or you could write your own full threadpool.

However, why you want to limit to five threads? Why don't you use the
default System.Threading.ThreadPool?

-mike
MVP

Ricardo said:
Hi...

I'm doing a program that analise a web page for a string, if found it
register that page, and if it found any links on that page, it follows the
link and searsh it for the string, and so on.

The problem is that I want to delimit this process to only 5 pages to be
processed at a time. I think of using a queue, and when the one of the 5
treads end processing the page, pick another link on the queue and start
processing again.

But how can I do that??? I need some mechanism that tells me when a thread
finished so I can start another...

Do you all have any idea that can help me???


[]s...
Click to expand...
Click to expand...
Click to expand...
 
Here is one way...
1) create a class that contains a thread. The class may have a public
..Start() method.
2) the thread start calls a private method that is your loop that reads a
shared queue. The queue is sync'd.
3) create 5 instances of this class and add to a collection (i.e. array,
array list, etc.) This is in your main control object.
4) Enum the list and start all threads.
5) You probably have another thread that is your producer to put items into
the queue. This may also be your control object.
5) put 5 stop objects into the queue when you want them to shutdown.

Cheers.
 
Back
Top