Threading.....

[TampaWebDevelopment]

I am writing a FTP backup service for my servers. Right now, it works
well, but it is single threaded and takes a couple of hours to
complete a backup of the websites directory, even to a server on the
same network. I want to spin off 8 threads to handle the uploading.
When one files is finished uploading, a thread becomes available and a
new one is opened.

So, say that I have the following code..... This is just Pseudo
Code...

For x As Integer = 0 To UBound(arrayOfFiles)

'-- Old Code Here Was Something Like This.....
'-- ftpConnection.UploadFile(arrayOfFiles(x), strRemotePath)

'-- What I want to do here is just continue with this single thread
and just add all these to a Queue....
'-- I will then have a System.Timers.Timer look for items in the
global queue and upload them..

'-- So, I wrote my own queue, inheriting from a something in
System.Collections and my new code will be like this..

Queue.Add(arrayOfFiles(x), strRemotePath)

Next

The question is... What should my code look like in the Timer? I think
I might need to use some Events and Delegates, maybe a custom event
like Queue.Item(x).UploadFile ???

I just need a little sample on multi-threading for this type of
purpose and I am good. All the samples that I have found thus far are
either junk code, or just didn't fit into what I was trying to
accomplish.

Any help is appreciated. Thanks.

 

[Mr. Arnold]

I am writing a FTP backup service for my servers. Right now, it works
well, but it is single threaded and takes a couple of hours to
complete a backup of the websites directory, even to a server on the
same network. I want to spin off 8 threads to handle the uploading.
When one files is finished uploading, a thread becomes available and a
new one is opened.

So, say that I have the following code..... This is just Pseudo
Code...

For x As Integer = 0 To UBound(arrayOfFiles)

'-- Old Code Here Was Something Like This.....
'-- ftpConnection.UploadFile(arrayOfFiles(x), strRemotePath)

'-- What I want to do here is just continue with this single thread
and just add all these to a Queue....
'-- I will then have a System.Timers.Timer look for items in the
global queue and upload them..

Why not use the Thread.Sleep(waittime) in a while loop?

'-- So, I wrote my own queue, inheriting from a something in
System.Collections and my new code will be like this..

Queue.Add(arrayOfFiles(x), strRemotePath)

Next

The question is... What should my code look like in the Timer? I think
I might need to use some Events and Delegates, maybe a custom event
like Queue.Item(x).UploadFile ???

You can user a Timer with Timer.Start/Timer.Stop or Thread.Sleep

You would be using the Timer.AddressOf(TimerElapsed_method) or
Thread.AddressOf(StartThread_method) the delegate method for each

I just need a little sample on multi-threading for this type of
purpose and I am good. All the samples that I have found thus far are
either junk code, or just didn't fit into what I was trying to
accomplish.

You have to cut your own path, as you might not find an example with exactly
what you're looking for. You should be able to find some examples that use
System.Timers and System.Threading namespaces out there on Google.

What's the big deal anyway? The Timer_Elapsed or the Thread.Sleep wait time
has elapsed. You run the methods in the delegates, they work with the queue
until complete, and they go back to sleep until the time has elapsed again.

I like using Thread.Sleep, because you better awareness of the a Thread
Exception in a While True loop using Thread.Sleep, like Thread exceptions.

You also might want to look at FileSystemWatch, look it up.

 

[Phill W.]

I am writing a FTP backup service for my servers. Right now, it works
well, but it is single threaded and takes a couple of hours to
complete a backup of the websites directory, even to a server on the
same network. I want to spin off 8 threads to handle the uploading.

Having 8 threads working their way through the /same/ list of files
could very well be /slower/ than having a single Thread on its own
because of the contention caused by having lots of things accessing the
same list of things to do.

Threads work /much/ better if you can give them a [simple] job to do,
pat them on the head and forget all about them; just let them get on
with it.

Now, if you could have, say, one Thread per /Server/ you might get some
improvement, because each Server's list of files will be self-contained
within the Thread so it doesn't need to "talk" to anything else.

Anyway ...

'-- What I want to do here is just continue with this single thread
and just add all these to a Queue....
'-- I will then have a System.Timers.Timer look for items in the
global queue and upload them..

'-- So, I wrote my own queue, inheriting from a something in
System.Collections and my new code will be like this..

Queue.Add(arrayOfFiles(x), strRemotePath)

OK; A Queue is a good idea but I don't think you'd want a Timer in the
Threads. Timers introduce delays and you want these things processed
just as fast as possible.

You'll need to synchronise access to the Queue, have each Thread pull an
item off the Queue, process it and loop round for another one. If the
Thread runs out of things to do, it should quietly die off.

HTH,
Phill W.

 

[Chris Mullins [MVP]]

I am writing a FTP backup service for my servers. Right now, it works
well, but it is single threaded and takes a couple of hours to
complete a backup of the websites directory, even to a server on the
same network. I want to spin off 8 threads to handle the uploading.
When one files is finished uploading, a thread becomes available and a
new one is opened.

Sounds like a decent plan. I could make a case for using Async I/O instead,
but in this case, it would probably complicate the problem for very little
benefit.

Create a class that some number of threads in it, and has a "PostWorkItem"
method, where the workItem is a directory.

When a thread finishes, it moves on to the next available work item. The
code below I just wrote (I haven't run it, or even tried to compile it!),
but it should be enough to get you started. You'll need to improve the
shutdown logic and other related stuff, and add in some status / progress
updates, a Count Property, a Remining Items property, and a bunch of other
similar stuff. Don't forget to do basic locking around the Queue when you
access it!


Imports System.Threading

Public Class Uploader
Private _syncroot As New Object
Private _shutdownEvent As New ManualResetEvent(False)
Private _workItems As New Queue(Of String)
Private _workerThreads As New List(Of Thread)

Public Sub Uploader()
For i As Integer = 0 To 8
Dim newThread As New Thread(AddressOf DoWork)
newThread.Name = "Worker Thread " & i.ToString()
newThread.Start()
_workerThreads.Add(newThread)
Next
End Sub

Public Sub PostWorkItem(ByVal directoryToUpload As String)
SyncLock _syncroot
_workItems.Enqueue(directoryToUpload)
End SyncLock
End Sub

Public Sub Shutdown()
_shutdownEvent.Set()
ForEach Thread t In _workerThreads
t.Join()
Next
End Sub

Private Sub DoWork()
While Not _shutdownEvent.WaitOne(1000, False)
Dim directoryToUpload As String = String.Empty
SyncLock _syncroot
If _workItems.Count > 0 Then
directoryToUpload = _workItems.Dequeue()
Else
directoryToUpload = Nothing
End If
End SyncLock

If directoryToUpload <> Nothing Then
'*** Do long running FTP Process here
End If
End While
End Sub

End Class

 

[Chris Mullins [MVP]]

I am writing a FTP backup service for my servers. Right now, it works
well, but it is single threaded and takes a couple of hours to
complete a backup of the websites directory, even to a server on the
same network.

'-- I will then have a System.Timers.Timer look for items in the
global queue and upload them..

You don't really want to use a Timer for this. When the timer gives you a
callback, that callback is running on a ThreadPool thread. There are a
limited number of these threads (varying with the number of processors in
your system, and the version of the .Net framework you're running), and if
you tie all these threads up (aka: Induce Threadpool Starvation), your
application won't work right.

Threadpool threads are ideal for handling small, compute-bound tasks.
They're a poor choice for performing long-running I/O bound tasks.

Given that you're going to be "owning" this thread for a few hours, you're
not hitting target use case for the ThreadPool thread. Your remining choices
are:
1 - Async I/O
2 - Your own threads

In this case, owning and managing your own threads (as I did in the sample I
just posted) is probably the easiest way to go. Async I/O would probably get
you better performance, but it generally takes a while for people to wrap
their heads around.