Incrementally saving a DataTable

  • Thread starter Thread starter dvestal
  • Start date Start date
D

dvestal

For purposes of this question, suppose I have a DataTable, and I add
one row per second, over the course of several hours.

I have to:
- Keep this data in memory, and
- Not lose more than the last second's worth of data in the event of a
computer crash.
- Satisfy the first two requirements reasonably quickly, and without
slowing down over time.

To avoid losing data, I want to save the DataTable to a file, but I
don't know how to easily do this without saving the entire set of
data, and this will not scale well as the DataTable grows large. Any
ideas on how to accomplish this?
 
To avoid losing data, I want to save the DataTable to a file, but I
don't know how to easily do this without saving the entire set of
data, and this will not scale well as the DataTable grows large. Any
ideas on how to accomplish this?
Click to expand...

Depends on the actual data types.
For simple stuff like values transmitted from measuring devices a CSV
file might suffice. Unlike XML you can append to it without having to
care about the structure.
 
Depends on the actual data types.
For simple stuff like values transmitted from measuring devices a CSV
file might suffice. Unlike XML you can append to it without having to
care about the structure.
Click to expand...

I tried to simplify the example to make it easier to express, but the
actual problem involves a DataSet with a dozen or more tables. Maybe
I oversimplified. Anyway, I don't think a flat file works for me.
 
I tried to simplify the example to make it easier to express, but the
actual problem involves a DataSet with a dozen or more tables. Maybe
I oversimplified. Anyway, I don't think a flat file works for me.
Click to expand...

How about a flat file for each table then?
Of course this involves writing a small recovery tool which restores the
dataset and table relations.
 
The ds.WriteXML(pathname);

is very easy to use, it removes your old file so it looks that this is the
only code row you need.

However, your computer can crash while writing or whatever, then you loose
your original on disk and the one in memory, as I read it right not the best
thing that can happen to you.

Therefore you should first move the original file to a temp folder and when
the writeXML process is ready remove that one (or leave it, that is your
choice)

Here beneath are some links to classes you need, you need more, while there
are alternatives for these

http://msdn.microsoft.com/en-us/library/system.io.fileinfo.moveto(VS.71).aspx

http://msdn.microsoft.com/en-us/library/system.io.path.aspx

Cor
 
For purposes of this question, suppose I have a DataTable, and I add
one row per second, over the course of several hours.

I have to:
- Keep this data in memory, and
- Not lose more than the last second's worth of data in the event of a
computer crash.
- Satisfy the first two requirements reasonably quickly, and without
slowing down over time.

To avoid losing data, I want to save the DataTable to a file, but I
don't know how to easily do this without saving the entire set of
data, and this will not scale well as the DataTable grows large.  Any
ideas on how to accomplish this?
Click to expand...

Avoid using DataTable, and use a journalling in-process DMBS instead
(MSSQL CE, SQLite, or Firebird).

If you really only ever add rows, then do what Julia suggested, and
store it in CSV (or a similar "appendable" format) with one file per
table.
 
Back
Top