Speeding up array/file loading

  • Thread starter Thread starter Jarry
  • Start date Start date
J

Jarry

I have two arrays to load, taking one line to an entry from two 770,000
line files: so I have two arrays of 770000. But this can take upwards
of 5 minutes, which is simply too long. I use a stream reader to
readline from the textfiles, using the syntax
For i = 1 to 770000
myArray(i) = myStreamReader.readLine()
Next

How can I speed this process up, but try to keep the arrays that size?
All ideas welcomed.

Thanks,
Jarry
 
Jarry said:
I have two arrays to load, taking one line to an entry from two 770,000
line files: so I have two arrays of 770000. But this can take upwards
of 5 minutes, which is simply too long. I use a stream reader to
readline from the textfiles, using the syntax
For i = 1 to 770000
myArray(i) = myStreamReader.readLine()
Next

How can I speed this process up, but try to keep the arrays that size?
All ideas welcomed.

Thanks,
Jarry
Click to expand...

I had a similar problem when implementing a comprehensive "log file" class
for my software. When in Verbose mode, the logs can easily expand to over
100mb pretty quickly. The solution I came up with was to write each line of
the log file fixed width, i.e. say, 128 characters in width. Then I don't
have to "readline" (which needs to scan for a newline character), I can just
suck in 128 characters at a time, or any multiple of 128 characters. The
resulting log file is larger than it would have been without fixed width,
but I can easily display it using Virtual Mode in a list view pretty much in
real-time.

If I were you, I would at least read a large chunk of the file at a time and
do your processing in memory rather than from disk. Get your stream reader
to read in 10mb at a time into a byte array and then split the byte array by
newline into your "myArray". You'll have tricky cases around the edges of
each chunk but it will be an order of magnitude quicker to do it that way.
 
Assuming each line has a carriage-return/line-feed at the end
(and it must, or readLine wouldn't work), here's an easy way
to read in a file and split the lines by Crlf into an array
in one fell swoop.

Dim crlfs() as String = {ControlChars.CrLf}
Dim lines() as String = _
File.ReadAllText("c:\data.txt").Split(crlfs, StringSplitOptions.None)
Dim numOfLines = lines.Length

Thanks to Francesco Balena; I got this out
of one of his books, probably the VB2005 Core Reference.

Robin S.
 
RobinS said:
Dim crlfs() as String = {ControlChars.CrLf}
Dim lines() as String = _
File.ReadAllText("c:\data.txt").Split(crlfs, StringSplitOptions.None)
Dim numOfLines = lines.Length
Click to expand...

Thanks, I'm sure to try it out
 
Back
Top