CF EWF and Lazy Write

  • Thread starter Thread starter brendan
  • Start date Start date
B

brendan

Hello All,

I have a componetized Application that was originally designed for XPP.
I've built an image and included my app and it runs fine booting off my
1 gig compact flash.

My question is of the design nature.

I want to use the CF because it will be in an industrial location. My
app runs as a shell not allowing the user anything but the app. screen.

My app saves data to storage, saves configurations and saves "recipes".
The unit will run 24/7 365. The only time it will be off is when there
is a power outage. At which point it better come back on with the last
settings and data that were in use the instant before the power outage.

Using EWF with RAM overlay doesn't seem to be a viable option as I may
not have a controlled shutdown. My thought was to split up my CF to
480MB (OS and App), 500MB(overlay area) and the rest ~16MB
unpartitioned for the EWF volume and working space. I would then use
Disk type overlay with my second partition being the overlay storage.
This way my data is retained until the next startup when it will be
transferred to my protected partition 1.

Of course now my second partition will get just as many writes as the
protected partition would have had I not used EWF and therefore just as
prone to eventual wearout. So I figured I'd enable the "Lazy write"
function so that the writing to the second partition would occur less
often. but if that is the case then why don't I just set up the first
partition to have lazy writes? (Is there even a way to do this?)

My flash has a guaranteed 2 million writes which would be once every
three minutes for a product life of ten years which is very ample for
my application data. But it seems that what Windows writes/tries to
write is a mystery......else I'd drop the EWF and stock up on flash
cards ; )

ttfn,
brendan
 
Hi Brendan,

Why do you want to protect your second partition?

With your setup you will get twice as much writes that you'd get with
saving
the data direct to an unprotected volume and keeping it there.

I would suggest to use EWF REGISTRY RAM Overlay to protect you windows
partition
and then save all needed data to an unprotected partition. With this
setup you will not loose any data during power outages (offcourse you
will loose data if writes are currently in progress during an power
outage).

Good luck,

Gustaf Sandmark


brendan skrev:
 
I misunderstand the way EWF works.

My first partition is the only partition that I want.
I want the unit to power off and on without loss of data so that it
"looks" like XPP but smaller more rugged and more secure.

My understanding was that if I use RAM overlay type then I will lose my
data on powerdown. Using diskoverlay I will not. I thought that I had
to create another partition to hold the data in Flash. I didn't realize
that during FBA the EWF routine creates this "undefined" partition.
Since it's undefined I can't see into it for diagnostic purposes.

If I use RAM overlay and write the data that I really need over to my
extra drive I guess I have to do that manually? Meaning I have to
create an app to save this all the time? If so what is lazy write for?

I think that I'm not really getting what EWF is really supposed to
do....

thanks,
Brendan
 
Brendan,

Check out this url.
http://msdn.microsoft.com/library/d...-us/xpehelp/html/xeconEnhancedWriteFilter.asp

It will tell all you need to know out EWF.

Regarding the data that needs to be saved I cant advice since I know
nothing about your application. If you write your own, just be sure to
be able to save to another drive: or maybe you can run your whole app.
from the unprotected drive and save data localy on the drive?

I'm not familiar with lazy writes and EWF but as far as I can imagine
this will be off no difference with RAM overlay since no data ever will
be saved to disk if not ever commited.
Lazy writes could maybe be usefull for faster disk writes/access in a
diskoverlay EWF? Please correct me if I am wrong :)

Gustaf Sandmark
 
Even with disk-based EWF, there is no guarantee that data won't be lost and
system becomes corrupted on power interruptions since some of the OS and
your app data might still be in the filesystem cache and not completely
written to disk (or to the EWF overlay if EWF is used). The best way is to
secure the power to the machines or use a UPS system. The lazy write flag
for EWF instructs EWF to cache changes to the protected volume before
writing them to the EWF overlay. This will optimize the performance since
EWF doesn't have to write to the overlay for every write you do to the
protected partition. The other more important factor is the system lazy
write. The filesystem caches writes to the disk. There is a system lazy
writer thread that runs about every sec and write parts of the cached
modified pages to the disk (the disk in this case is the non-protected
partition you're writing to or the EWF overlay). This is also done for
performance optimization. Even if you managed to disable this cache, it
doesn't guarantee no loss of data or no corruption since the power failure
might happen at the moment when the data is written to disk. Not to mention
the performance hit that you might incur.

KS


This posting is provided "AS IS" with no warranties and confers no rights.
 
Still confused...

1. If I use Disk Overlay so that all of my write data is redirected and
stored on my "overlay partition" (as created by EWF during FBA) AKA my
"overlay volume" AKA part of my Unpartitioned space then I will be
writing to my compact flash card just as often as if I had no EWF. With
the protected partition being CF and the Overlay area being CF what am
I gaining? A write to a part of the registry 5 times in partition 1 or
in the EWF Volume is 5 burns into compact flash either way.

2. If my application is the shell and I do NO writes to memory at all
is Windows XPE still chugging away trying to write logs, update
registries etc? How do I know? is there a way to see activity? What
kind of monitoring tools are there?

3. As I understand it "Lazy writes" write to cache. This is another
name for RAM (as long as page swapping is off)?
Searching MSN, Limings book, Cseri's book always the Lazy write is
unchecked. So what's it all mean?
I searched for a "System Lazy Write" but can't find such a thing. How
is the System Lazy Write different than a Lazy Write?

4. I would rather not have to modify XPP applications to run on XPE(
besides componentizing etc) Wasn't that part of the whole idea of XPE?
Test on XPP run on XPE?

5. Right now without EWF, running on CF I power up and down at will
changes are saved. All I want to is Filter my Writes so that I don't
burn out my flash. If it's Enhanced so much the better. So I add EWF
with RAM Overlay and my program still works but on powerdown my changes
are lost (as expected). So I change to Disk overlay and everything
works! Power down, power up Life is Good. Even my dog loves me. But how
do I know what is going on? How many writes to my compact flash are
occuring? I cannot afford to ship out 50 of these units and have the
compact flash cards burn out in 6 months...

6. Are there any write leveling algorithyms that are used in XPE ?

Sorry for coming across so dense but as a hardware designer I feel
really out of my element here.....

Brendan Kirby
 
Brendan,

Disk-based EWF won't help when it comes to reducing writes to the CF drive
since as you mentioned, the writes will be just redirected to somewhere
else on the CF drive, unless you have another drive on the machine where
the EWF overlay can be created on.

If you want to protect you CF against writes to increase the life of the
CF, you can use RAM-based EWF where changes to the protected partition are
directed to the RAM instead. But on power interruption or reboot all these
changes will be lost including the data that your app wrote to the CF,
unless you commit these changes from the RAM onto the CF before rebooting.
But this is not reliable since on sudden power interruption you won't even
have a chance to save the changes onto the CF.

The requirements seem conflicting with each other; you don't want to write
to the CF to reduce the wear and tear of the CF but you still want to save
your data to the CF? A solution would be to protect the CF with a RAM-based
EWF (to increase its life) and then change your app to write to another
unprotected media, but in this case you'll require another media.

The idea of RAM-based EWF is to prevent changes to the protected partition
(usually the boot partition where the OS is installed) from changes to
persist across reboots. This is helpful if you want to discard all changes
the OS or a user (or corruption on power interruptions) on next reboot
where the system will come up fresh and new as you initially deployed it.
Another usage for RAM-based EWF is to increase the life of a CF by blocking
writes to it but then you'll have to find a way to persist your data if
this is another requirement; by either writing to an unprotected partition
or committing the EWF overlay to the CF which will write all the changes
you did to the CF since you booted last time. This will commit the data
that you want to commit along with other data that you may not want such as
user changes, OS changes, etc. So basically it depends on the nature of
your app and what things it needs to persist. If you want your app to
always start from a preset configuration, you could write this config info
to a file on the CF when EWF is disabled and then enable EWF. Then on every
boot, your app checks the config file and configures itself accordingly;
this way it'll start in the same way every time. If your app needs to write
some other data while running and you want to persist these changes on
demand, you can use the EWF APIs or command line utility (ewfmgr.exe) to
commit the changes, also unfortunately committing all other changes since
last boot. Or you could in your app, disable EWF, write your data and then
enable EWF back.

KS


This posting is provided "AS IS" with no warranties and confers no rights.
 
Also need to mention that if the app does a lot of writes that needs to be
persisted then EWF won't help here in case you don't have another hard
drive for your app to writes to. You should instead go with a hard drive or
a CF with high life expectancy, probably something of an industrial grade.

If your app is not writing much to the CF but the OS is doing, you could
use tools like File Monitor and Registry Monitor from sysinternals.com to
track these writes down; remember XPe is XP Pro and you'll see the same
writes that XP does to the boot drive (depending on what features of XP you
included in the XPe image); things like the writes of the filesystem to its
own metadata, which increase when you do activities like creating/deleting
files, writes to the eventlog, IE writes to its cache, other writes to
various log files, writes to the registry hives, writes to the user
profile, etc. There is not really a list of all the OS writes that XP does
and you can't eliminate all of it. You can instead redirect the writes to
the OS partition into the RAM (if using EWF) or redirect some of these
writes to another disk drive (see
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/xpehelp/htm
l/xeconewfperformanceconsiderations.asp).

KS

This posting is provided "AS IS" with no warranties and confers no rights.
 
brendan,
1. If I use Disk Overlay so that all of my write data is redirected and
stored on my "overlay partition" (as created by EWF during FBA) AKA my
"overlay volume" AKA part of my Unpartitioned space then I will be
writing to my compact flash card just as often as if I had no EWF. With
the protected partition being CF and the Overlay area being CF what am
I gaining? A write to a part of the registry 5 times in partition 1 or
in the EWF Volume is 5 burns into compact flash either way.
Click to expand...

How said EWF Disk mode is good for use on CF?
EWF RAM Reg mode is, though.
2. If my application is the shell and I do NO writes to memory at all
is Windows XPE still chugging away trying to write logs, update
registries etc? How do I know? is there a way to see activity? What
kind of monitoring tools are there?
Click to expand...

Unfortunately, OS does lots of write underneeze, especially in registry.
You can lower that if you properly set up your image.
Follow tips from this page: http://msdn.microsoft.com/embedded/community/community/tips/xp/ramewf/default.aspx
3. As I understand it "Lazy writes" write to cache. This is another
name for RAM (as long as page swapping is off)?
Searching MSN, Limings book, Cseri's book always the Lazy write is
unchecked. So what's it all mean?
I searched for a "System Lazy Write" but can't find such a thing. How
is the System Lazy Write different than a Lazy Write?
Click to expand...

I don't know what you refer to as "System Lazy Write".
But Lazy Writes do not make much sense with EWF enabled. You may want to turn it off.

4. I would rather not have to modify XPP applications to run on XPE(
besides componentizing etc) Wasn't that part of the whole idea of XPE?
Test on XPP run on XPE?
Click to expand...

Yup.
However, you will need to satisfy all application dependencies. Depending on the software your image may grow much and even not fit
on CF you've got or in RAM you install on target.

In general, in embedded space you'd want to optimize your software (and system of course) to minimize resource consumption, improve
performance (target may not be high end hardware), adopt the software to behave properly in embedded usage sceneriosand with some
embedded features (EWF, cloning, etc.) and etc.

But of course it all comes to the custom software specifications, system requirements, project scope and time frames :-)
XPe has been really good on fast development side.
5. Right now without EWF, running on CF I power up and down at will
changes are saved. All I want to is Filter my Writes so that I don't
burn out my flash. If it's Enhanced so much the better. So I add EWF
with RAM Overlay and my program still works but on powerdown my changes
are lost (as expected). So I change to Disk overlay and everything
works! Power down, power up Life is Good. Even my dog loves me. But how
do I know what is going on? How many writes to my compact flash are
occuring? I cannot afford to ship out 50 of these units and have the
compact flash cards burn out in 6 months...
Click to expand...

This is why you've been told to move your application data to another partition.
Then you can perfectly protect system partition with EWF RAM and not worry much about sudden power offs (at least on the system
level).
Assuming your software doesn't write data too often it shouldn't destroy the flash on the second partition too fast. Having control
over your applications code you can even improve the situation by intentionally distributing the app writes accross flash blocks to
lower wear leveling.
Also you have to make sure your application properly handle sudden shutdowns in the middle of write operation. E.g., it may be smart
to implement destination data backup and check data integrity at good system boot.
6. Are there any write leveling algorithyms that are used in XPE ?
Click to expand...


No, no at all.
However, you may be interested in some 3rd party solutions. E.g., uDOC of M-Systems. They implemented nice wear leveling algorithms
in their firmware.
 
Thank You to all,

Your replies should be published as they explain some of gaps in both
books and at MSN.

As a summary:
Disk overlay is only good if you have a hard
drive as your overlay area and then only if you are running your OS off
a CD or would like multiple overlays
RAM overlay is good for a stateless system
unless you have some means of "committing" needed data.

I still don't understand Lazy Writes. Lazy
Writes should not be used with EWF but seem to only be enabled in the
EWF configuration. Nobody uses them ..........
Liming says about Lazy Writes: " ...EWF flushes
the cache to disk in an optimized way, usually as a background
activity.... This reduces writes to the EWF volume which can be
important for flash drives."

Anyhow as was stated early and repeated often (and I just couldn't get
it) . If I use RAM Overlay and store my dat to an unprotected partition
I will get what I want. My only concern would be eating up RAM by OS
stuff. I'm guessing the tools mentioned by KS would prove this to me.
BTW All that RAM Reg does is store my EWF Volume in the OS registry
instead of it's own partition right? I'm guessing this is good for
cloning?

Thanks again,
Brendan
 
RAM-Reg based EWF is basically RAM-based EWF but the EWF configuration info
is stored in the registry instead of the EWF partition (overlay). This is
needed when running off of a media that is reported as removable media to
the OS such as many of the compact flash cards, USB and PCMCIA drives.
Removable media can't have more than one partition and therefore RAM-based
EWF and disk-based EWF can't be set up on such media since it can't create
an EWF partition.

KS


This posting is provided "AS IS" with no warranties and confers no rights.
 
Back
Top