Random file access -con't

[Wapiti]

Thanks Ginny,

That sounds very similar to the approach I've taken with older DOS handheld
units. Reading the whole file into memory is a scary approach, but workable.
I suppose then, that at program close, you send whats in memory back to the
file? Or provide a flush option to the user?

In terms of deletions, again, that is what I have done in older DOS apps -
and I agree, it works well. When I export the file to the appropriate
format required by the host, I just don't send those records (flagged as
deleted).

Something that came to mind, does anyone know what Microsoft uses for data
storage for applications like their onboard Calendar? Its storing data
somehow, and its not all that slow. If its sqlce, then its already onboard,
right?? If its not, what is it?

Thanks again,

Mike

I use a text file format similar to CSV. I separate records by CRLF and
fields by the pipe symbol. I read in the whole file when the program loads
and parse the parts into collections of business objects primarily using
StreamReader.ReadLine, String.Split, and various members of the Convert
class. (This is indeed the slowest part, but it's many times faster than
parsing XML and it only happens once when the app loads.) Whenever I need to
add a record to a file, I add it to the memory collection and I also append
a record to the text file itself. (This is fast.) If I need to delete or
modify a record, I use a special code in a new record I append that
indicates that the record is deleted or modified. This approach works well
for reasonably small amounts of data - a few thousand records - but it
requires a desktop app that can play back the log files and update
enterprise data correctly as well as the intelligence in the device app to
correctly interpret the files. But for large amounts of data requiring a
storage card, I would certainly use SqlCe instead rather than trying to keep
all the data in memory all the time.

I think for the volume of data you're expecting that you have a number of
options. I recommend trying several and see what you like, but I don't
recommend XML unless the data volume is very small.

--
Ginny Caughey
.Net Compact Framework MVP

So, what method do you use then, for accessing your text files? In the
Compact Framework? And how many records are you accessing - do you perform
record(column) modifications with the collection data? If so, how?

Regular text files seem like they would consume a lot of time.


than,
for where

use

of

just

a

big
text file anyway?

I've never used xml - maybe now is the time to try??

Input appreciated.

Mike


"SetFilePointer" is a cool function but it does not provide TRUE random
access like the days of old.

Take a look at this thread:
http://www.xtremevbtalk.com/archive/index.php/t-179063.html

Thanks,
Noble

"Paul G. Tobey [eMVP]" <ptobey no spam AT no instrument no spam

DOT
com>

wrote in message You can call anything in any DLL anywhere on the device, but you have
to
write your own declarations. That's what P/Invoke is: Platform
Invoke.
You
declare the functions which are found in DLLs as externs and

then
call

them,
passing suitable parameters. If you feel like something is missing,
then
you haven't been involved in .NET CF development long enough, yet.

Download the OpenNETCF SDF and look through the source code. A huge
percentage of what is done there is working around the lack of managed
code
wrappers for Win32 API functions. In fact, CreateFile, WriteFile,
ReadFile
and, maybe, SetFilePointer, are in there.

Paul T.


code
that guys
are worth
of

run

on device
(as for
the

 

[Wapiti]

Sorry to keep beating this horse with a dead noodle ... but I beg to wonder
why data collection wasn't given a higher priority in CF, in terms of RAD
for the programmer.

With that out, has anyone checked out, tried or used the following:
http://www.pocketpcdn.com/libraries/opennetcf_csv.html

I'm new to this CF stuff, so haven't quite figured it out in full detail.
DataSet? Like a ado recordset? Still don't know, but hoping to shed some
light on it, unless someone can beat me to it here.

Mike

 

[Ginny Caughey [MVP]]

I don't need to flush data when the app ends since I just append records to
the log file as the transactions happen. I decided to do this so the data
could survive a soft-reset if necessary. But anyway it sounds like your
older DOS experience has prepared you well to visualize my approach. ;-)

Currently Calendar and the other Pocket Outlook apps use the native data
store, but a future version of SqlCe will replace it (and also be installed
in ROM) for the future versions of Windows Mobile. Generally SqlCe will be
faster that the current native database even today though.

--
Ginny Caughey
..Net Compact Framework MVP

Thanks Ginny,

That sounds very similar to the approach I've taken with older DOS handheld
units. Reading the whole file into memory is a scary approach, but workable.
I suppose then, that at program close, you send whats in memory back to the
file? Or provide a flush option to the user?

In terms of deletions, again, that is what I have done in older DOS apps -
and I agree, it works well. When I export the file to the appropriate
format required by the host, I just don't send those records (flagged as
deleted).

Something that came to mind, does anyone know what Microsoft uses for data
storage for applications like their onboard Calendar? Its storing data
somehow, and its not all that slow. If its sqlce, then its already onboard,
right?? If its not, what is it?

Thanks again,

Mike

I use a text file format similar to CSV. I separate records by CRLF and
fields by the pipe symbol. I read in the whole file when the program loads
and parse the parts into collections of business objects primarily using
StreamReader.ReadLine, String.Split, and various members of the Convert
class. (This is indeed the slowest part, but it's many times faster than
parsing XML and it only happens once when the app loads.) Whenever I

need

to
add a record to a file, I add it to the memory collection and I also append
a record to the text file itself. (This is fast.) If I need to delete or
modify a record, I use a special code in a new record I append that
indicates that the record is deleted or modified. This approach works well
for reasonably small amounts of data - a few thousand records - but it
requires a desktop app that can play back the log files and update
enterprise data correctly as well as the intelligence in the device app to
correctly interpret the files. But for large amounts of data requiring a
storage card, I would certainly use SqlCe instead rather than trying to keep
all the data in memory all the time.

I think for the volume of data you're expecting that you have a number of
options. I recommend trying several and see what you like, but I don't
recommend XML unless the data volume is very small.

--
Ginny Caughey
.Net Compact Framework MVP

So, what method do you use then, for accessing your text files? In the
Compact Framework? And how many records are you accessing - do you perform
record(column) modifications with the collection data? If so, how?

Regular text files seem like they would consume a lot of time.


The main problem with XML with large amounts of data is that it's
generally
slow to parse it on mobile devices as well as taking up more space than,
for
example, as CSV file. I use text files instead of XML in most cases where
I
can.

--
Ginny Caughey
.Net Compact Framework MVP



So, with that link in mind, what would anyone think regarding the

use

of
xml
to store, and otherwise manage, data for a data collection application?

The data doesn't have to be synchronized with any host database, but
extracted to a ascii flat text file.

Seems though, that xml will be slow as an ascii stream - isn't it

just

a
big
text file anyway?

I've never used xml - maybe now is the time to try??

Input appreciated.

Mike


"SetFilePointer" is a cool function but it does not provide TRUE
random
access like the days of old.

Take a look at this thread:
http://www.xtremevbtalk.com/archive/index.php/t-179063.html

Thanks,
Noble

"Paul G. Tobey [eMVP]" <ptobey no spam AT no instrument no spam DOT
com>
wrote in message You can call anything in any DLL anywhere on the device, but you
have
to
write your own declarations. That's what P/Invoke is: Platform
Invoke.
You
declare the functions which are found in DLLs as externs and then
call
them,
passing suitable parameters. If you feel like something is missing,
then
you haven't been involved in .NET CF development long enough, yet.

Download the OpenNETCF SDF and look through the source code.

A
huge

percentage of what is done there is working around the lack of
managed
code
wrappers for Win32 API functions. In fact, CreateFile, WriteFile,
ReadFile
and, maybe, SetFilePointer, are in there.

Paul T.


in
message
Certainly you *can* use direct file I/O to accomplish what you
want.
Paul's
suggestion of CreateFile/ReadFile/WriteFile would

certainly
work

as
would
the .NetCF StreamReader and StreamWriter classes which wrap
them,
although
neither of these gives you random access - you'd need to code
that
for
yourself using these functions or classes as primitives.

I guess I'll have to do some digging to figure out what you guys
are
talking
about here. Even tho Paul 'already explained it' - explain

to
me

why,
if
CF
doesn't support random access, would the basic win32 api functions
for
file
access in CF be supported?

Something doesn't sound right.


1-2 thousand records, depending on the size of each record,
isn't
a
large
amount of data for modern devices. And where the amount of data
*is*
large,
the usual solution is to store the data on a storage card.

Sounds reasonable. A storage card might be the best way of going
anyway.


I don't know off hand the exact footprint of SqlCe, but that
product
was
designed for exactly your type of scenario (and also for amounts
of
data
that are much larger.) What is the absolute minimal hardware
that
you
need
to target?

I'm testing on an Axim right now. 'Absolute minimal' is
difficult
to
define. I guess I'd have to resort to the answer, it

depends.
But

that
would be too vague so, I could set a minimum to my users. They're
going
to
be collecting data - that data could span 1 day to a weeks worth
of
data.
The records aren't large by any means, say, 10 fields per record
each.
its
a warehouse application - Issue, Receipt, Transfer and Inventory
adjustments.


There are also 3rd party database solutions available.

I've been burned before on these, but if anyone has
recommendations,
I'd
be
open to hearing them. Speed in development, sufficient

speed

in run

export

a but
my access
on

using

a

 

[Ginny Caughey [MVP]]

The DataSet is the standard way of storing data in .Net. It's similar to an
Ado recordset, but it's always disconnected from the underlying data source.
It uses data adapters (like the CSV one on www.opennetcf.org) to load and
save data. I didn't chose that approach myself because the business objects
I use are more lightweight than DataSets. Of course I also have to write the
code to load and save data for them, etc. But the main reason is that the
CSV data adapter rewrites the entire CSV file when you save the state of the
DataSet - logical if you think about it since it's not writing to a SQL
database. And I wanted to save each transaction as it happens to minimize
the possibility of transactions getting lost and customers getting angry.

If you want the standard RAD approach - it's DataSet and SqlCe in the
Compact Framework. It will get a lot more RAD with VS 2005 since there will
be better tools for creating and working with both SqlCe and DataSets.

 

[Wapiti]

Ok, so I'm looking at implementing SQLce...

Looking at the requirements on
http://www.microsoft.com/sql/ce/productinfo/sysreq.asp
"A Microsoft SQL Server 2000 Windows® CE Edition (SQL Server CE) 2.0
development environment includes up to four systems: a system running SQL
Server¹, a system running Internet Information Services (IIS)², a
development system³, and a Windows CE-based device4. You can use the same
computer for the SQL Server system, the IIS system, and the development
system."

So, in attempt to install, it appears I -NEED- SQLserver installed as well
as IIS. Even though I'm not going to be doing any synchronization??? I
just want a local device database to store my collection data onto....
please, is this not crazy? Just to store information on a handheld device
one needs so much???

I own SQLserver 7 (installed) and SQLserver 2000 (not installed). I have a
Windows 2000 Server, and my development machine is WinXP - not running IIS
nor SQLserver.

I don't need to flush data when the app ends since I just append records to
the log file as the transactions happen. I decided to do this so the data
could survive a soft-reset if necessary. But anyway it sounds like your
older DOS experience has prepared you well to visualize my approach. ;-)

Currently Calendar and the other Pocket Outlook apps use the native data
store, but a future version of SqlCe will replace it (and also be installed
in ROM) for the future versions of Windows Mobile. Generally SqlCe will be
faster that the current native database even today though.

--
Ginny Caughey
.Net Compact Framework MVP

Thanks Ginny,

That sounds very similar to the approach I've taken with older DOS handheld
units. Reading the whole file into memory is a scary approach, but workable.
I suppose then, that at program close, you send whats in memory back to the
file? Or provide a flush option to the user?

In terms of deletions, again, that is what I have done in older DOS apps -
and I agree, it works well. When I export the file to the appropriate
format required by the host, I just don't send those records (flagged as
deleted).

Something that came to mind, does anyone know what Microsoft uses for data
storage for applications like their onboard Calendar? Its storing data
somehow, and its not all that slow. If its sqlce, then its already onboard,
right?? If its not, what is it?

Thanks again,

Mike

need

app

to

correctly interpret the files. But for large amounts of data requiring a
storage card, I would certainly use SqlCe instead rather than trying

to
keep

all the data in memory all the time.

I think for the volume of data you're expecting that you have a number of
options. I recommend trying several and see what you like, but I don't
recommend XML unless the data volume is very small.

--
Ginny Caughey
.Net Compact Framework MVP



So, what method do you use then, for accessing your text files? In the
Compact Framework? And how many records are you accessing - do you
perform
record(column) modifications with the collection data? If so, how?

Regular text files seem like they would consume a lot of time.


message
The main problem with XML with large amounts of data is that it's
generally
slow to parse it on mobile devices as well as taking up more space than,
for
example, as CSV file. I use text files instead of XML in most cases
where
I
can.

--
Ginny Caughey
.Net Compact Framework MVP



So, with that link in mind, what would anyone think regarding

the
use

of
xml
to store, and otherwise manage, data for a data collection
application?

The data doesn't have to be synchronized with any host database, but
extracted to a ascii flat text file.

Seems though, that xml will be slow as an ascii stream - isn't

it
just

a
big
text file anyway?

I've never used xml - maybe now is the time to try??

Input appreciated.

Mike


"SetFilePointer" is a cool function but it does not provide TRUE
random
access like the days of old.

Take a look at this thread:
http://www.xtremevbtalk.com/archive/index.php/t-179063.html

Thanks,
Noble

"Paul G. Tobey [eMVP]" <ptobey no spam AT no instrument no

spam
DOT

com>
wrote in message You can call anything in any DLL anywhere on the device, but you
have
to
write your own declarations. That's what P/Invoke is: Platform
Invoke.
You
declare the functions which are found in DLLs as externs and then
call
them,
passing suitable parameters. If you feel like something is
missing,
then
you haven't been involved in .NET CF development long

enough,
yet.

Download the OpenNETCF SDF and look through the source code. A
huge
percentage of what is done there is working around the lack of
managed
code
wrappers for Win32 API functions. In fact, CreateFile, WriteFile,
ReadFile
and, maybe, SetFilePointer, are in there.

Paul T.


"Ginny Caughey [MVP]"

wrote
in
message
Certainly you *can* use direct file I/O to accomplish

what
you

want.
Paul's
suggestion of CreateFile/ReadFile/WriteFile would certainly
work
as
would
the .NetCF StreamReader and StreamWriter classes which wrap
them,
although
neither of these gives you random access - you'd need to code
that
for
yourself using these functions or classes as primitives.

I guess I'll have to do some digging to figure out what

you
guys

are
talking
about here. Even tho Paul 'already explained it' -

explain

to speed sqlce
is understand
that customers
run export

system

of
using apps
for

 

[Ginny Caughey [MVP]]

You don't need SQL Server to use SqlCe. SqlCe is already part of Visual
Studio.Net. When you add a reference to the SqlCe DLL in your app, the SqlCe
engine is automatically deployed to the device or emulator when you deploy
your app. It might be more details than you want, but there is a book by Rob
Tiffany called SQL Server CE Database Developmjent with the .Net Compact
Framework that covers just about everything.

--
Ginny Caughey
..Net Compact Framework MVP

Ok, so I'm looking at implementing SQLce...

Looking at the requirements on
http://www.microsoft.com/sql/ce/productinfo/sysreq.asp
"A Microsoft SQL Server 2000 Windows® CE Edition (SQL Server CE) 2.0
development environment includes up to four systems: a system running SQL
Server¹, a system running Internet Information Services (IIS)², a
development system³, and a Windows CE-based device4. You can use the same
computer for the SQL Server system, the IIS system, and the development
system."

So, in attempt to install, it appears I -NEED- SQLserver installed as well
as IIS. Even though I'm not going to be doing any synchronization??? I
just want a local device database to store my collection data onto....
please, is this not crazy? Just to store information on a handheld device
one needs so much???

I own SQLserver 7 (installed) and SQLserver 2000 (not installed). I have a
Windows 2000 Server, and my development machine is WinXP - not running IIS
nor SQLserver.

I don't need to flush data when the app ends since I just append records to
the log file as the transactions happen. I decided to do this so the data
could survive a soft-reset if necessary. But anyway it sounds like your
older DOS experience has prepared you well to visualize my approach. ;-)

Currently Calendar and the other Pocket Outlook apps use the native data
store, but a future version of SqlCe will replace it (and also be installed
in ROM) for the future versions of Windows Mobile. Generally SqlCe will be
faster that the current native database even today though.

--
Ginny Caughey
.Net Compact Framework MVP



to
the

delete

or works
well app

requiring

a

storage card, I would certainly use SqlCe instead rather than trying to
keep
all the data in memory all the time.

I think for the volume of data you're expecting that you have a

number
of

options. I recommend trying several and see what you like, but I don't
recommend XML unless the data volume is very small.

--
Ginny Caughey
.Net Compact Framework MVP



So, what method do you use then, for accessing your text files?

In
the

Compact Framework? And how many records are you accessing - do you
perform
record(column) modifications with the collection data? If so, how?

Regular text files seem like they would consume a lot of time.


message
The main problem with XML with large amounts of data is that it's
generally
slow to parse it on mobile devices as well as taking up more space
than,
for
example, as CSV file. I use text files instead of XML in most cases
where
I
can.

--
Ginny Caughey
.Net Compact Framework MVP



So, with that link in mind, what would anyone think regarding the
use
of
xml
to store, and otherwise manage, data for a data collection
application?

The data doesn't have to be synchronized with any host

database,
but

extracted to a ascii flat text file.

Seems though, that xml will be slow as an ascii stream - isn't it
just
a
big
text file anyway?

I've never used xml - maybe now is the time to try??

Input appreciated.

Mike


"SetFilePointer" is a cool function but it does not provide TRUE
random
access like the days of old.

Take a look at this thread:
http://www.xtremevbtalk.com/archive/index.php/t-179063.html

Thanks,
Noble

"Paul G. Tobey [eMVP]" <ptobey no spam AT no instrument no spam
DOT
com>
wrote in message You can call anything in any DLL anywhere on the device,

but
you

have
to
write your own declarations. That's what P/Invoke is: Platform
Invoke.
You
declare the functions which are found in DLLs as externs and
then
call
them,
passing suitable parameters. If you feel like something is
missing,
then
you haven't been involved in .NET CF development long enough,
yet.

Download the OpenNETCF SDF and look through the source

code.
A

huge
percentage of what is done there is working around the

lack

of explain

amount

of but
that system them
for wrong,
but random
file they
are

 

[Wapiti]

Ginny, your patience is admirable. Thank you - that was too simple.

And book references are excellent - thank you for that.

-m

You don't need SQL Server to use SqlCe. SqlCe is already part of Visual
Studio.Net. When you add a reference to the SqlCe DLL in your app, the SqlCe
engine is automatically deployed to the device or emulator when you deploy
your app. It might be more details than you want, but there is a book by Rob
Tiffany called SQL Server CE Database Developmjent with the .Net Compact
Framework that covers just about everything.

--
Ginny Caughey
.Net Compact Framework MVP

Ok, so I'm looking at implementing SQLce...

Looking at the requirements on
http://www.microsoft.com/sql/ce/productinfo/sysreq.asp
"A Microsoft SQL Server 2000 Windows® CE Edition (SQL Server CE) 2.0
development environment includes up to four systems: a system running SQL
Server¹, a system running Internet Information Services (IIS)², a
development system³, and a Windows CE-based device4. You can use the same
computer for the SQL Server system, the IIS system, and the development
system."

So, in attempt to install, it appears I -NEED- SQLserver installed as well
as IIS. Even though I'm not going to be doing any synchronization??? I
just want a local device database to store my collection data onto....
please, is this not crazy? Just to store information on a handheld device
one needs so much???

I own SQLserver 7 (installed) and SQLserver 2000 (not installed). I

have

a
Windows 2000 Server, and my development machine is WinXP - not running IIS
nor SQLserver.


records
to

will

be

(flagged

as for
data CRLF
and faster
than delete

but

it

requires a desktop app that can play back the log files and update
enterprise data correctly as well as the intelligence in the

device
app

to
correctly interpret the files. But for large amounts of data

requiring

a
storage card, I would certainly use SqlCe instead rather than

trying
to

keep
all the data in memory all the time.

I think for the volume of data you're expecting that you have a number
of
options. I recommend trying several and see what you like, but I don't
recommend XML unless the data volume is very small.

--
Ginny Caughey
.Net Compact Framework MVP



So, what method do you use then, for accessing your text files? In
the
Compact Framework? And how many records are you accessing - do you
perform
record(column) modifications with the collection data? If so, how?

Regular text files seem like they would consume a lot of time.

in

message
The main problem with XML with large amounts of data is that it's
generally
slow to parse it on mobile devices as well as taking up more space
than,
for
example, as CSV file. I use text files instead of XML in most cases
where
I
can.

--
Ginny Caughey
.Net Compact Framework MVP



So, with that link in mind, what would anyone think

regarding
the

use
of
xml
to store, and otherwise manage, data for a data collection
application?

The data doesn't have to be synchronized with any host database,
but
extracted to a ascii flat text file.

Seems though, that xml will be slow as an ascii stream -

isn't
it

just
a
big
text file anyway?

I've never used xml - maybe now is the time to try??

Input appreciated.

Mike


"SetFilePointer" is a cool function but it does not

provide
TRUE

random
access like the days of old.

Take a look at this thread:
http://www.xtremevbtalk.com/archive/index.php/t-179063.html

Thanks,
Noble

"Paul G. Tobey [eMVP]" <ptobey no spam AT no instrument no spam
DOT
com>
wrote in message You can call anything in any DLL anywhere on the device, but
you
have
to
write your own declarations. That's what P/Invoke is:
Platform
Invoke.
You
declare the functions which are found in DLLs as externs and
then
call
them,
passing suitable parameters. If you feel like something is
missing,
then
you haven't been involved in .NET CF development long enough,
yet.

Download the OpenNETCF SDF and look through the source code.
A
huge
percentage of what is done there is working around the

lack

of
managed
code
wrappers for Win32 API functions. In fact, CreateFile,
WriteFile,
ReadFile
and, maybe, SetFilePointer, are in there.

Paul T.


"Ginny Caughey [MVP]"

wrote
in
message
Certainly you *can* use direct file I/O to

accomplish
what

you
want.
Paul's
suggestion of CreateFile/ReadFile/WriteFile would
certainly
work
as
would
the .NetCF StreamReader and StreamWriter classes which
wrap
them,
although
neither of these gives you random access - you'd

need

to what
you amount way
of are
the to
the

running

on for
the

example,

I customers
will streamwriter
and