Performing a one-pass reading of a database

  • Thread starter Thread starter Joannes Vermorel
  • Start date Start date
J

Joannes Vermorel

I am coding a sampling algorithm in a library and I got IO efficiency
problems with "OleDbDataAdapter.Fill". Here is the situation, I have coded a
method:

void MySamplingAlgo(DataTable source, DataTable destination)

And my algorithm is efficient: each DataRow of "source" is read only once
and a few percentage of them are copied into "destination" (the selected
samples).

But my problem is the following: When I use OleDbAdapter (and
OleCommandBuilder) to connect to a medium MS Access 2000 table (~100mo), the
method OleDbAdapter.Fill first loads the entire table in RAM. For large
tables, this is totally inefficient since my computer runs out of RAM and
starts using virtual memory.


I would like to know
1) How can I do (while using the abstract DataTable) in order to have the
table read only once (the data should not be read outside "MySamplingAlgo")
?
2) How can I do to prevent the DotNet framework from loading in RAM the
entire table that are accessed via "OleDbAdapter" ?

Any help would be greatly appreciated,
Joannes Vermorel
____________________
http://www.vermorel.com/
 
Datatables are meant to load the entire table into memory. That is what
they
are there for.

If you want to read one row from the database at a time, use an
OleDbDataReader.
Click to expand...

Ok, thanks for the help. It seems that "OleDbDataReader" is the keyword I
was looking for.

By the way, the objects "DataRow" and "IDataRecord" (obtained respectively
from a "DbDataAdapter" and from a "DbDataReader") are semantically very
close from each other. But it seems that there is no convertion methods
between "DataRow" and "IDataRecord" provided in DotNet.

Does someone know the reason ?

Joannes Vermorel
____________________
http://www.vermorel.com
 
Back
Top