Best choice of data storage on PDA in C# (short term/ long term)

[Daniel Passwater via DotNetMonster.com]

I am a newbie, so please be patient. I am working on an app that will extract data from an embedded system via serial connection. I need to get the data as fast as possible and then go back and assimilate it after I've disconnected.

What is the best way to store this data for both the temporary and long term?

Thanks to anyone who helps.

*****************************************
* This message was posted via http://www.dotnetmonster.com
*
* Report spam or abuse by clicking the following URL:
* http://www.dotnetmonster.com/Uwe/Abuse.aspx?aid=84b2448f9fb84bc1b6bab3aee06821c8
*****************************************

 

[Jordan Irwin via DotNetMonster.com]

Most of my disconnected applications use flatfiles with indexes for large amounts of data. Flatfiles allow you to send only the data you need, with very little overhead. Indexing does put some overhead after the transfer to the device, but it isn't required. All indexing would do is make accessing the information quicker.

If you don't normally use flatfiles, here are some basic examples:

Data from embedded system (I'm going to use a table)

PersonTable
[ID] [LastName] [FirstName]
123 Doe John
567 Doe Jane

Now, how do we transfer this information to the device quickly? Create a flatfile, or string in a flatfile format:

123|Doe|John
567|Doe|Jane

Using pipes, | , I divide my columns with little overhead. You can turn this into one giant string, with Newlines as record dividers if you'd like. That would provide an easy way to encrypt and send the data. As for multiple tables, you could send a string for each table, or use an identifier for each table. Sometimes, a number (or tablename) is used at the beginning of each record to identify the table it belongs to.

1|123|Doe|John
1|567|Doe|Jane

OR

Person|123|Doe|John
Person|567|Doe|Jane

But, this adds alot of extra information. You could provide a divider row to seperate your tables, something that would easily be recognizable as the end/beginning of a table,

~PersonTable~
123|Doe|John
567|Doe|Jane
~NextTable~

In the end, your results would be far smaller that using XML, or most other transfer techniques. The only disadvange is that both applications need to know how to handle the data once it's in this format. Some of my uses have decreases data size by up to 80%. That's a nice cut in transfer amount.


Good luck, and I hope this helps provide some answers.

*****************************************
* A copy of the whole thread can be found at:
* http://www.dotnetmonster.com/Uwe/Forum.aspx/dotnet-compact-framework/9134
*
* Report spam or abuse by clicking the following URL:
* http://www.dotnetmonster.com/Uwe/Abuse.aspx?aid=7f4653516b0e42969b0fa915b15d5ccf
*****************************************

 

[Boris Nienke]

Most of my disconnected applications use flatfiles with indexes for large
amounts of data. Flatfiles allow you to send only the data you need,
with very little overhead. Indexing does put some overhead after the
transfer to the device, but it isn't required. All indexing would do is
make accessing the information quicker.

123|Doe|John
567|Doe|Jane

could you explain how you build and use indexes on flat files?

Boris

 

[Jordan Irwin via DotNetMonster.com]

I can explain the basics,


The index is another file that stores very basic information about your flatfile. In the most simple form, it is a list of PrimaryKeys and there location within the flatfiles.

For instance (using our previous example),

FlatFile

123|Doe|John
567|Doe|Jane


Index

123 0
567 14


What's this mean? Well, the index list the primary key of each Person and their byte location within the flatfile (Don't forget, this is a BYTE count. So if you're writing strings to the file you must also count the NEWLINE and CARRAGE RETURN, +2). So, if you were searching for Person 567 you could search the index file for the primary key 567:

Algorithm:

Open a StreamReader on the index file
Read until you find your primary key for the person (in this case, 567)
Now, open a StreamReader to your FlatFile
Seek to the location (in this case, 14)
Read the information

In this example, the index provides not much increase in speed. You could ask yourself why build this index file when I could just search the FlatFile? Well, imagine your FlatFile holding information similiar (or even more complex) than below:

123|Doe|John|123456 Somewhere Lane|This City|This State|098765-0987|Some Information|Here is some text notes that could go on and on and on and on and on and on and on, I think you get the point. This one record could span a long time.|12/01/01|43|Another note about anything, once again it could go on a long time.

Now, multiply that one record times hundreds, or thousands, of people. You now have a pretty large size file to search through. Your index only holds the neccessary information to get to specific People.


Like I said before, depending on your information your index files could be more complex (perhaps multiple indexes for a single record), or less complex (maybe you don't even need an index). It does put some overhead on the device after the transfer, but it really slices the amount of data to be transfered to the device down nicely.

Hope this helps,

*****************************************
* A copy of the whole thread can be found at:
* http://www.dotnetmonster.com/Uwe/Forum.aspx/dotnet-compact-framework/9134
*
* Report spam or abuse by clicking the following URL:
* http://www.dotnetmonster.com/Uwe/Abuse.aspx?aid=9bc0d169bcaf4d04a16d33ef72da88b7
*****************************************

 

[Boris Nienke]

FlatFile

123|Doe|John
567|Doe|Jane


Index

123 0
567 14


What's this mean? Well, the index list the primary key of each Person and their byte location within the flatfile (Don't forget, this is a BYTE count. So if you're writing strings to the file you must also count the NEWLINE and CARRAGE RETURN, +2). So, if you were searching for Person 567 you could search the index file for the primary key 567:

OK, that's clear.

Algorithm:

Open a StreamReader on the index file
Read until you find your primary key for the person (in this case, 567)
Now, open a StreamReader to your FlatFile
Seek to the location (in this case, 14)
Read the information

this seems a bit ugly. You will just save the time to read some bytes if
the flat-file has very long lines.

But you are still reading seriell...

I think such index will make sense if it is sorted by primary key. This way
you would be able to read the key with a Quick-Sort/Read algorithm.
But then you need a fixed lenght of the key-line.

But currently i don't know how to create such sorted key-files. It's easy
when you can hold it in memory as once... just build it up in memory, sort
it and then store it.

But how to create such index file and then insert a line above/beyond an
existing line? Are there ready functions for that or would i have to
- Read the lines above and write into new file
- insert new line
- Read the lines beyond and append to new file
- delete old file

That would be really slow

Boris