How to speed up reading of large disk files

  • Thread starter Thread starter Valerie Hough
  • Start date Start date
V

Valerie Hough

Sorry if this is the wrong news group for this post.

I have some large disk files (15-20 MB) that are made up of many records of
different types and sizes. I use a BinaryReader to process all the data
sequentially (i.e. with many reads operations) and load it into my .NET app.

I will have several .NET Apps running on the same computer each of which
will need to read these files but never change them.

The performance is not awful, but I would like it to be faster. I have
looked at Memory File mapping, RAM disks, etc. but I would like to know if
someone has a better idea especially if it stays within normal .NET I/O
funcationality.

Thanks in advance for your help.
Valerie Hough
 
Valerie Hough said:
[...]
The performance is not awful, but I would like it to be faster. I have
looked at Memory File mapping, RAM disks, etc. but I would like to know if
someone has a better idea especially if it stays within normal .NET I/O
funcationality.
Click to expand...

The easiest thing you can do to speed up disk i/o is to read larger portions
of the file at a time. With .NET, you either do this explicitly with large
reads, or you can explicitly set the FileStream to use buffered i/o when you
open the file, and specify a large buffer size.

I know it's not a very sophisticated technique, but I've had good success
with it. I try to use buffers that are multiples of the default page size
(4096 bytes), and have found through experimentation that buffers larger
than two or three pages reach a point of diminishing returns. At that
point, to get better performance would probably require a lot more work on
my part. :)

The other thing you can do is try to make sure that you can operate on the
data directly from the buffers used to read it. If you stick another copy
operation of the data in your code, that will slow you down, especially if
you're dealing with a lot of data.

My apologies if the above suggestions seem obvious and not useful. :)

Pete
 
Back
Top