Read large text files

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

Hello:
How to read large text files about 100M?How about using method
ReadtoEnd()?
I need some suggestions.

Thanks in advance.
 
gladiator said:
How to read large text files about 100M?How about using method
ReadtoEnd()?
I need some suggestions.
Click to expand...

Well, do you need to have the whole file in memory at a time, as a
single string? If so, ReadToEnd is probably okay. If you could get away
with processing a line at a time, using ReadLine would be much better
from a memory perspective.
 
i have tried to read files about 100M,but failed,out of memory error
happened. the buffer size is not enough,how do predict the buffer size in
advance?
i need some help/
TIA
 
gladiator said:
i have tried to read files about 100M,but failed,out of memory error
happened. the buffer size is not enough,how do predict the buffer size in
advance?
Click to expand...

Well, you could find the file length and use that to work out how many
characters might be in the file (depending on the encoding), but you
should really only read the whole file into memory at once if you
*really* need all of it at the same time. Do you?
 
thanks for your concerns.
the data to split into is at the fixed length,and surely i can read a bit
of it ,then discard the buffer. if i read them all in , i want to use regex
to parse it or the like to split into small strings.i if not read them as a
whole,i can read them at a fixed length at a time.
which method is better i want to know? the real data size is below 100M i
think.
so....
your suggestions.
thank in advance.
 
gladiator said:
thanks for your concerns.
the data to split into is at the fixed length,and surely i can read a bit
of it ,then discard the buffer.
Absolutely.

if i read them all in , i want to use regex
to parse it or the like to split into small strings.i if not read them as a
whole,i can read them at a fixed length at a time.
which method is better i want to know? the real data size is below 100M i
think.
Click to expand...

It sounds to me like you should either call ReadLine, or just Read
(being careful to note the return value).
 
Back
Top