How to read large text file ?

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

Hi, All
I have text file ASCII with record length of 388 bytes, no record delimeter
and size of the file is 562477780 bytes and 1449685 records all togeter.
How can i read such file record by record ?

Please, help
 
elena said:
Hi, All
I have text file ASCII with record length of 388 bytes, no record delimeter
and size of the file is 562477780 bytes and 1449685 records all togeter.
How can i read such file record by record ?

Please, help
Click to expand...

First of all, I wouldn't read this record-by-record. Disk reads are
slow, and so it is always best to minimize the number of reads if
possible. You can do this by reading in chunks... You will want your
chunk size to be a multiple of 388 (your record length). Then you can
process those records before you read your next chunk. So, in psudo
code you would do something like:

while more records
read group of records
process records
end while

Now, in code you would probably want to use the System.IO.StreamReader
class for this - the built in VB.NET file functions are very slow, and
are best avoided when working with larger files. There are examples in
the documentation of this class that show how to read a specific number
of char's at a time, so I'll direct you to the doc's for that :)

You may also want to consider not hardcoding the number of records to
process. That way, you can optimize it for speed and memory usage :)
I would consider puting the group size in an app.config file.
 
Back
Top