C# Write lines to file at high volume, multiple threads

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

Hi -

I am attempting to write lines to a file at high volume, multiple threads.
Here is my scenario:

(initial "WriteToFile" object created via a parent multithreaded process,
which receives files, can call custom code for each file received - much like
BizTalk)

1. Receive multiple files, each file received creates a "WriteToFile" object
2. Each "WriteToFile" object parses it's file, creates a line, and attempts
to write that line to a specified file. Most or all of the "WriteToFile"
objects will be attempting to write to the same file.
3. If a "WriteToFile" object can't write it's data to the file, it should
throw an exception, handle it, and try again.

Questions:
1. Would this be better handled as a Windows service which reads files from
a queue and processes them serially?

2. If I didn't want to use a queue or service approach, what could I use to
check whether a file is accessible or not? Ie: Is some other process writing
to it?

3. Any code suggestions for a high volume file-write service or process?

Thaks for your help in advance!
 
Dameon said:
Hi -

I am attempting to write lines to a file at high volume, multiple threads.
Here is my scenario:

(initial "WriteToFile" object created via a parent multithreaded process,
which receives files, can call custom code for each file received - much
like
BizTalk)

1. Receive multiple files, each file received creates a "WriteToFile"
object
2. Each "WriteToFile" object parses it's file, creates a line, and
attempts
to write that line to a specified file. Most or all of the "WriteToFile"
objects will be attempting to write to the same file.
3. If a "WriteToFile" object can't write it's data to the file, it should
throw an exception, handle it, and try again.

Questions:
1. Would this be better handled as a Windows service which reads files
from
a queue and processes them serially?

2. If I didn't want to use a queue or service approach, what could I use
to
check whether a file is accessible or not? Ie: Is some other process
writing
to it?

3. Any code suggestions for a high volume file-write service or process?
Click to expand...

I think if you want a high volume file writing, you want a single thread
writing to the file and the WriteToFile objects should parse it's file,
create a line and pass that line to the file writer thread by adding it to a
Queue of lines to be written.
 
I'd use file proxy object servicing the queue of requests

It can sit on separate thread and be waken up by monitor event by any other
posting thread. File access code then is encapsulated in proxy object. For
high volume you need to use asynch mode or file writing

If you have multiple files, you can create multiple proxies. Sending threads
can use global queue, which might distribute payload to corresponding proxy
queue. Could be done as service or as module in your application (or
plug-in).

As about locking, if only you write to the file, you can open it in
exclusive mode.

HTH
Alex
 
Back
Top