Writing into a binary file simultaneously from multithreads...

[Steve B.]

Hi,

I'm building an application that use a binary file format with a proprietary
format.
Files can grow up to several GBytes...

The mecanism of building the data is quite complex and I'd like to know if
multiple threads can -simultaneously- write into the data file...

For example, I know in advance thread one will write the first 100MB, thread
two 50 next MB, and so on. Can I write from multiple threads ?
What objects Have I to use ?
One FileStream and seeking as needing ? One FileStream and several
StreamWriters ? etc ...

Thanks in advance...
Steve

 

[Jon Skeet [C# MVP]]

I'm building an application that use a binary file format with a proprietary
format.
Files can grow up to several GBytes...

The mecanism of building the data is quite complex and I'd like to know if
multiple threads can -simultaneously- write into the data file...

For example, I know in advance thread one will write the first 100MB, thread
two 50 next MB, and so on. Can I write from multiple threads ?
What objects Have I to use ?
One FileStream and seeking as needing ? One FileStream and several
StreamWriters ? etc ...

I suspect there *are* ways of doing it from multiple threads, but it
feels like the kind of thing which could very easily go wrong. An
alternative which may well be simpler would be to have a single thread
which had a queue of write requests. The other threads would add jobs
to the queue, and then the single thread could process those requests.

 

[Steve B.]

After some tests, I found a solution which seems to work :

lock (outputStream)

{

outputStream.Seek(

dataOffset + numberObBytesWritten

SeekOrigin.Begin

);

outputStream.Write(

buffer,

0,

read

);

I'm not sure this is a very smart solution.
Your solution should look like this :

WriterClass.QueueWriteOperation(Stream s, int offset, byte[] buffer)...
That's it ?

Thanks,
Steve

 

[Jon Skeet [C# MVP]]

After some tests, I found a solution which seems to work :

lock (outputStream)
{
outputStream.Seek(
dataOffset + numberObBytesWritten
SeekOrigin.Begin
);

outputStream.Write(
buffer,
0,
read
);

I'm not sure this is a very smart solution.

It's not particularly safe, as if you have two streams pointing at the
same file, they could still both update it simultaneously.

Your solution should look like this :

WriterClass.QueueWriteOperation(Stream s, int offset, byte[] buffer)...
That's it ?

I wouldn't pass the stream - I'd create one WriterClass (or whatever)
per file I wanted to write to, and have it maintain the stream. Now,
admittedly you'd still need to make sure you didn't create two
WriterClasses for the same file, but that's probably easier to do than
the stream equivalent. (You could easily provide a factory method, in
fact.)

Note that an easier way of seeking is often to use the Position
property:

outputStream.Position = dataOffset+numberOfBytesWritten;

 

[Steve B.]

Thanks,
I'll take a deep look into this mecanism.

Steve

After some tests, I found a solution which seems to work :

lock (outputStream)
{
outputStream.Seek(
dataOffset + numberObBytesWritten
SeekOrigin.Begin
);

outputStream.Write(
buffer,
0,
read
);

I'm not sure this is a very smart solution.

It's not particularly safe, as if you have two streams pointing at the
same file, they could still both update it simultaneously.

Your solution should look like this :

WriterClass.QueueWriteOperation(Stream s, int offset, byte[] buffer)...
That's it ?

I wouldn't pass the stream - I'd create one WriterClass (or whatever)
per file I wanted to write to, and have it maintain the stream. Now,
admittedly you'd still need to make sure you didn't create two
WriterClasses for the same file, but that's probably easier to do than
the stream equivalent. (You could easily provide a factory method, in
fact.)

Note that an easier way of seeking is often to use the Position
property:

outputStream.Position = dataOffset+numberOfBytesWritten;