Looking for incremental snapshot utility for NTFS, FAT32

  • Thread starter Thread starter Matt
  • Start date Start date
M

Matt

Hi,

I would like to be able to manually "snapshot" my NTFS and FAT32 file
systems (mostly on WinXP, but other Windows as well as Linux
platforms) in an incremental fashion for backup purposes. Any
solutions/ideas...other then going to Veritas and other
enterprise-pricy solutions? I am looking for a personal-use
applications for someone like me with a home-office/individual budget
(probably less then $100 on this software, something under $50 would
be preferred).

I want this most so I can do periodic personal-data-space backups
without having to copy my entire file set (between 1.5 and 10 GB,
depending on how broad a backup I want) every time. I can use
something like Acronis TrueImage to do incremental backups, but this
requires me to save an entire partition--not desirable for my
data/file backups (but it works great for my OS system images!!).

I suspect there might be more options when I'm running these file
systems with a Linux-flavored OS, but this is not much of an option
for me at the moment. I still would like to know these options,
though.

As for a usage model: I'd like said software solution to present
"snapshoted" file systems much like NetApp NFS filers/servers do with
their automatic snapshot mechanism (I just don't need it to be
automatic: I can start it with a click of a button, etc). I like how
the "snapshot" part of the file system looks like:

/snapshot/<pick-you-file-system-name>.2004-02-14/8am/<files...>

....or something like that (I have actually looked at a NetApp
toaster/filer/nfs-server for a long time, I just remember the general
concept).

Thanks for any help,
-Matt
 
Matt said:
Hi,

I would like to be able to manually "snapshot" my NTFS and FAT32 file
systems (mostly on WinXP, but other Windows as well as Linux
platforms) in an incremental fashion for backup purposes. Any
solutions/ideas...other then going to Veritas and other
enterprise-pricy solutions? I am looking for a personal-use
applications for someone like me with a home-office/individual budget
(probably less then $100 on this software, something under $50 would
be preferred).
Click to expand...


Well, the Backup tool built-in to Windows XP has a snapshot function
integrated with incremental and differential backup functions which should
do what you need for XP.
 
Is this what yoiu're looking for ?
http://www.symantec.com/specprog/goback/
Click to expand...

At first glance of http://www.symantec.com/goback/features.html, it
appears that GoBack does not allow me to easily peer into a set (ie,
multiple at the same time) of "snapshoted" file systems like I can via
a NetApp fileserver/toaster/etc (for more info see the snapshot
filesystem definition in my original post). GoBack seems to make me
restore the drive to see the older stuff. It might have a "view/mount
the recovery file" gizmo, but I doubt it presents it natively like I
want.

Nonetheless, it looks like a viable option. That and Nik's reference
to WinXP backup utility (presumably ntbackup?).

-Matt
 
Matt,

There is a reason why NetApp can do this kind of thing so easily, yet it
seems so hard with Windows.

Both FAT32 and NTFS are "update-in-place" filesystems where file data is
overwritten in it's existing locations on disk. Also, both of these
filesystems have fixed locations for metadata such as block maps, directory
entries and pointer blocks ("indirect inodes" for Unix people).

In order to maintain a "snapshot", products that work with traditional
filesystems, such as:

GoBack
CDP's Persistant Storage Manager
Volume Replicator (was Compaq, now HP)

and similar products for Unix, also products that work in storage arrays
that can't assume anything about the filesystem since they are presenting
raw LUN's, such as:

EMC Snapview
HDS QuickShadowCopy

all have one thing in common - they have to maintain a temporary storage
area to actively copy blocks out to as they are changed. i.e. every write to
a block becomes a read (either from read cache or off disk) and two writes
(one to write the old data to the snapshot area, the other two write the new
data to the original block location).

This means that for most systems maintaining a snapshot incurs a performance
penalty, and if you want to keep more than one the performance penalty grows
exponentially with each additional snapshot maintained. We call these
"active" snapshots since to keep them available requires additional activity
either by an OS shim or by the storage array processor. One of the issues
with active snapshots is that when the snapshot storage area fills up the
snapshot can no longer be maintained and becomes invalid - so if you're
running a report or doing a backup from the snapshot at that moment the
operation will fail. In an active snapshot system the active filesytem has a
higher resource priority than the snapshots.

Windows System Restore is a different beast entirely - I won't go into it
here but suffice it to say it's not useful for what you're trying to do.

NetApp can maintain multiple, on-line and readable, snapshot like you want
because their patented WAFL filesystem has no fixed locations for metadata
and never updates blocks in-place when writing to a file. i.e. snapshots are
simply a pointer to an older consistent filesystem image that happens to
share some physical blocks with the active filesystem, but no active copying
is required. We call this a "passive" snapshot. Passive snapshots have zero
performance overhead and no risk of operations against the snapshot failing,
i.e. snapshots have a higher resource priority than the active filesystem.

NetApp can also produce iSCSI or FCP LUN's that are really container files
on top of WAFL. You can create an NTFS filesystem on these LUN's and due to
the virtualisation gain the passive snapshot capability that you are
seeking.

The only other system that I am aware of that offers passive snapshots, and
at the raw block level so it can apply to NTFS as well, is the StorageTek
SVA ("Iceberg") produce, which uses a virtualisation process very similar to
using NetApp for block. Except StoTek does compression that has a
performance penalty not just because of the compression processing but
because there is no longer a 1:1 relationship between real and virtual disk
blocks.

Hope all this is of some help. It is POSSIBLE for someone to produce a
system like you want - maintaining block indirection tables is all that's
required - but the overheads in doing this locally on a Windows box given
the nature of NTFS and FAT32 are very high.
 
Hi Thanatos,

Hope all this is of some help. It is POSSIBLE for someone to produce a
system like you want - maintaining block indirection tables is all that's
required - but the overheads in doing this locally on a Windows box given
the nature of NTFS and FAT32 are very high.
Click to expand...

Your post (not all of it shown here) makes a lot of sense to me. If
I'm reading things correctly, your post describes the challenges of
making an "on-the-fly"/real-time filesystem snapshot.

For what it's worth, I personally do not have necessary requirement
for the real-time portion of the snapshot capability. Rather, my main
focus is to save space for my disk-to-disk backups (mostly of just my
personal systems, but it could branch out into other things). These
backups need not be performed in an automated fashion, nor must they
run in real time on a production system. I'm happy to shut all my
data-access applications down, and run a backup "snapshot" for a
period of minutes to hours (probably a max of 2 hours, maybe more if I
stick to strickly running this at night).

My biggest problem is storage space. My currently-sized 5-10GB
backups (that will soon be growing) end up eating up a lot of space on
my 120GB external drives each time I do a full-copy snapshot. Since I
have no economic mechanism for incremental file-system snapshots, I
must copy over the entire directory structure every time. I wish I
could get something that could do these incrementals on FAT32/NTFS and
thus dramatically reduce my storage capacity needs for my backup
retention disk media. I care not if this is done real-time or
automatically; I just want to save storage space.

Does that change how one is to approach this?
-Matt
 
Hi Thanatos,



Your post (not all of it shown here) makes a lot of sense to me. If
I'm reading things correctly, your post describes the challenges of
making an "on-the-fly"/real-time filesystem snapshot.

For what it's worth, I personally do not have necessary requirement
for the real-time portion of the snapshot capability. Rather, my main
focus is to save space for my disk-to-disk backups (mostly of just my
personal systems, but it could branch out into other things). These
backups need not be performed in an automated fashion, nor must they
run in real time on a production system. I'm happy to shut all my
data-access applications down, and run a backup "snapshot" for a
period of minutes to hours (probably a max of 2 hours, maybe more if I
stick to strickly running this at night).

My biggest problem is storage space. My currently-sized 5-10GB
backups (that will soon be growing) end up eating up a lot of space on
my 120GB external drives each time I do a full-copy snapshot. Since I
have no economic mechanism for incremental file-system snapshots, I
must copy over the entire directory structure every time. I wish I
could get something that could do these incrementals on FAT32/NTFS and
thus dramatically reduce my storage capacity needs for my backup
retention disk media. I care not if this is done real-time or
automatically; I just want to save storage space.

Does that change how one is to approach this?
Click to expand...

See what the entry-level network storage boxes from NetApp and the EMC
file servers that Dell sells cost.
 
See what the entry-level network storage boxes from NetApp and the EMC
file servers that Dell sells cost.
Click to expand...

Such solutions will not autonomously run from any machine I have,
namely my WinXP laptop, but I appreciate the "big picture" thinking on
this.

In general, I suspect that one possibly solution to this problem would
be to create software that can compare one file system to another, and
essentially save the differences between the 2 file systems in a
"delta" space. One could save only those files that have been updated
for instance, or even save just the parts of the files that have
changed (something that could significantly reduce space for big
files). Additionally, and more-generally (ie, in a
file-system-independent fashion), one could do a block-level
comparison.

Not that these operations are "off line" and are not easily done (at
least for NTFS/FAT32) for online data. But that's not my
requirement/goal, nor would it be for what I think would be a HUGE
market of users like me, all with desktops/laptops wanting to keep
consistent data protection (from both logical and physical problems)
going without having to eat up so much disk space...and without having
to use tape. Disk-to-disk all the way. And all these users/market
have no problems going "offline" with the backup/incremental snapshot
because all it is their personal/home-office machine. No
multi-user/network serving.

In any case, I smell market opportunity here. Maybe I just go do a
market analysis and then go make the darn software myself and make a
few $$. I sure as heck can't find any solutions as a user.

Anybody want to join me? While I've done a lot of sw design and
coding in the past, I like to spend more of my time on the vision and
requirements specs side of things nowadays. I would love to work with
a designer/coder. I guess I have to figure out if there's that much
market opportunity (per demand, competition, and other barriers) to
make some reasonable dough. I have little idea at this point. I work
in the enterprise-storage industry, but that does not necessarily mean
much.

-Matt
 
Matt said:
In general, I suspect that one possibly solution to this problem
would be to create software that can compare one file system to
another, and essentially save the differences between the 2 file
Click to expand...

Such software already exists, it's called "Unison". (Note that
there's also a Usenet reader for Mac OS X of the same name.)
 
Back
Top