working with large files

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

When you have to read a big file (5-30MB) and throw the data into the
database, ofcourse some logics inbetween (doesn't matter) which of the ADO
methods is recommended.
1. read line by line and do the execute for each line
2. Consolidate the files into a dataset for example and use the dataset
update method
ie
for each line
.addnew in the dataset
end
dataset.Update( .Added)

I mean which one should we prefer in these situations, will it make any
difference in performance. Are there any limitations on how many rows can be
inserted using the dataset.Update method ?

Thanks much
Chris
 
You can also get all the statements you need into one large statement by
separating each one by semicolon, and then execute this once.

The Update method of the adapter does the updates one by one anyway, so I
don't imagine that would be any faster then any of the other methods. It
would still require round trips to the database, and have a lot of
processing overhead.

You should experiment to see what works best for your particular situation.
 
You will achieve far better performance if you read the file and process as
you go. Make sure to use objects that are "memory frugal" since you are
likely to do a lot of garbage collecting.

A dataset with 20MB of data in it is not a good use of datasets.

--
--- Nick Malik [Microsoft]
MCSD, CFPS, Certified Scrummaster
http://blogs.msdn.com/nickmalik

Disclaimer: Opinions expressed in this forum are my own, and not
representative of my employer.
I do not answer questions on behalf of my employer. I'm just a
programmer helping programmers.
 
Thanks guys for the feedback.

Thank you
Chris

Nick Malik said:
You will achieve far better performance if you read the file and process as
you go. Make sure to use objects that are "memory frugal" since you are
likely to do a lot of garbage collecting.

A dataset with 20MB of data in it is not a good use of datasets.

--
--- Nick Malik [Microsoft]
MCSD, CFPS, Certified Scrummaster
http://blogs.msdn.com/nickmalik

Disclaimer: Opinions expressed in this forum are my own, and not
representative of my employer.
I do not answer questions on behalf of my employer. I'm just a
programmer helping programmers.
Click to expand...
 
Back
Top