Reading Huge Text File Problem

  • Thread starter Thread starter WoodenSword
  • Start date Start date
W

WoodenSword

Hi,

I am trying to read a huge text file (2GB) using
StreamReader.

I do this:
Read a line (readline())
I process the line a bit (split it with delimeter and
change fields a bit)
execute (using transaction) a insert query to Ms Sql server
Commit a transaction for 1000 insert queries.

For the first 10000 records speed is pretty good.However
speed gradually slows down significally ever after.What's
wrong?Do i have to use a buffer size in streamreader's
constructor?Do i have to use filestream instead? Is it Sql
server or disk performance problem ?

Thanx!
 
Are you building a concatenated string? If so, using the StringBuilder class
clould improve the speed.

Just an idea...
 
Hi WoodenSword,

I seldom say this, but I think this is typicaly a question for or a SQL
newsgroup.

Or the very active microsoft.public.dotnet.framework.AdoNet newsgroup

Or do you think that it is inside your VB.net code.

In that case we should have to see some code from you, to help you I think?

Cor
 
* "WoodenSword said:
I am trying to read a huge text file (2GB) using
StreamReader.

I do this:
Read a line (readline())
I process the line a bit (split it with delimeter and
change fields a bit)
execute (using transaction) a insert query to Ms Sql server
Commit a transaction for 1000 insert queries.

For the first 10000 records speed is pretty good.However
speed gradually slows down significally ever after.What's
wrong?Do i have to use a buffer size in streamreader's
constructor?Do i have to use filestream instead? Is it Sql
server or disk performance problem ?
Click to expand...

Post your code. Maybe you don't dispose unused objects and the
application runs out of memory.
 
Is the database set to Automatically Grow File?

Is the initial database free space adequate to handle all of the inserts or
is the SQL server having to pause, grow the database, insert some more
records, pause, grow the database....
 
You should comment out the database part and just do the file reading,
parsing piece to see if it is the file stream stuff or the database code.

Break it down into smaller chunks to focus on where the problem is.

Jerry
 
Back
Top