FAT32 or NTFS?

[cquirke (MVP Win9x)]

The jpegs will affect the equation here. The average size of a picture,
depending on its source may be anywhere between 150KB for web photos to
2-3MB for those downloaded from a digital camera. If we consider the former,
it would not be incorrect to assume that the wastage in such a scenario,
with many small files, would be considerable

Well, do the maths. If files are 150k, and cluster space is 32k
rather than 4k, then the biggest wasteage would be 32k-4k = 28k per
file. Assume actual wastage is half that, i.e. 14k, per file. That
boils down to under 10%; significant, but not carnage.

OTOH, if it's more like 2M per file, this drops to an under 0.7% shrug

As suggested by your figures about start menu shortcuts.

Those are 500-*byte* files, not 150 000 - 2 048 000 bytes
What will really waste space would be IE web cache...

Other considerations of the original poster were "resilience, recovery, ease
of repair".

Yep - and once again, NTFS isn't quite what it seems.

resilience --- As far as FAT partitions go, don't forget to wave goodbye to
your all data and the OS, when the disk becomes over full. This has happened
to me once already.

Haven't seen that, unless you are using disk compression. Read up on
the shuffling NTFS does when it fills up... I can see the same sort of
widening of critical windows there, as it re-balances between MFT and
data space. Problems may also arise when software that assumes
limitless memry (as bounded only by free swap space) hits the wall.

In such situations, it becomes tricky to save the OS, and depending on
the seriousness of the crash, data loss possibilities are considerable.

I've done a lot of data recovery on FATxx, and that's the point; it is
possible to recover lost data in such cases.

recovery and ease of repair --- NTFS file systems don't need a scandisk add
on per se, simply because they don't need one. Data verification
technologies imbedded in the file system ensure that data is only written on
to the disk, when it is verifyible. If it cannot be read back, the
transation is simply rolled back.

Read: The damaged data is stimply and irreversably destroyed.

This is what I mean; NTFS "doesn't need" Scandisk simply because it
behaves as if Scandisk was running all the time, with the most
ruthless trade-off between your data and file system integrity.

That is NOT what I consider "data safety".

Which brings me to this - In an NTFS file system, a transaction is
either performed completely or not performed at all. It's 1 or 0.

Exactly. Some of us would prefer to recover truncated or partial
data, but you don't have that option with NTFS.

Consider a power outage occuring during a defrag process
of a FAT drive. The possibilities of errors is high on a FAT partition

Sure; that alerts you to what files are damaged - but at least you
have a chance at recovery of partials. With NTFS, it's simply gone
without a trace, and no way of getting it back.

One more thing, you mention that "the tiny file data would be held entirely
within the directory entry metadata." Wouldn't this be stored in the MFT?
metafiles as such, (as I have read anyway) refer to the files created when
an NTFS partition is first created (sixteen), and contain volume & cluster
information; strictly speaking unavailable to the OS. Under NTFS, there is
no specific difference between a file and a collection of attributes,

Specifically, file data and attribute metadata are held in the same
way, as streams. Streams are effectively cluster chains containing
content that hang off a dir-level entry; a structure that's familiar
from working in PICK.

including data contained in the file itself. When the space required for all
the attributes is smaller than the size of the MFT record itself, then the
data attribute is stored within the MFT record itself. Please clarify what
distinction you consider between MFT and metadata.

By metadata, I was referring to that within the directory level, not
within the streams. But you do raise an interesting point - that
bloated metadata are not "free" if it effectively means two or more
cluster chains to look at in addition to the dir level.

Earlier on, someone made the claim that NTFS keeps redundant copies of
the dir structure. What I read suggests it's only the first few
entries that are held in that way - in effect, a similar result as
predictable locations for these core structures as applies in FATxx
(where this knowledge is the "redundant copy").

------------ ----- ---- --- -- - - - -

Our senses are our UI to reality

 

[Alexander Grigoriev]

You misunderstand NTFS journalling. It doesn't delete file data behind your
back. It doesn't check the _data_ being written, too.

What is really protected by journalling is metadata. It guarantees, that in
whatever moment your system fails, the metadata is always consistent, and
your disk structure is correct. It doesn't protect you against data loss
because it was not flushed from write-back cache.

 

[cquirke (MVP Win9x)]

On Tue, 11 May 2004 03:14:10 GMT, "Alexander Grigoriev"

You misunderstand NTFS journalling. It doesn't delete file data behind your
back. It doesn't check the _data_ being written, too.

Actually, I understant it all too well.

What is really protected by journalling is metadata. It guarantees, that in
whatever moment your system fails, the metadata is always consistent, and
your disk structure is correct. It doesn't protect you against data loss
because it was not flushed from write-back cache.

That's *exactly* what I understand.

Maintaining file system integrity is nice, but it is not data
preservation. When the data you want is partial, forcing an
unambiguous file system state not only deletes this data, but also
removes any variance cues you'd have used to recover it.

Kill, bury, deny.

-------------------- ----- ---- --- -- - - - -

Tip Of The Day:
To disable the 'Tip of the Day' feature...

 

[J. Clarke]

On Tue, 11 May 2004 03:14:10 GMT, "Alexander Grigoriev"


Actually, I understant it all too well.


That's *exactly* what I understand.

Maintaining file system integrity is nice, but it is not data
preservation. When the data you want is partial, forcing an
unambiguous file system state not only deletes this data, but also
removes any variance cues you'd have used to recover it.

If this actually has an effect on your life then you need to fix whatever is
causing your system to keep crashing during writes.

 

[Al Dykes]

If this actually has an effect on your life then you need to fix whatever is
causing your system to keep crashing during writes.

I find it hard to believe the guy who insists that NTFS "deletes data"
has any experience managing large amounts of data in a business
environment, or experience with NTFS, for that matter. I also don't
understand the recovery scenario he finds so necessary.

If I spend an hour working on a huge Photoshop image (as much as 100
MB) and get a power hit and my system reboots do I go back to my
original and spend an hour redoing it or do I turn my system over to a
techie to (maybe) find all the clusters, while I have a coffee break,
and he's got more important things to do ? If power hits were common
I'd get a UPS. If my system crashed more than once I'd get a new
system.

For a business database you can't work on the basis of trying to
recover clusters of data. The auditors would go apeshit. I use a
database like Oracle or MSSQL and depend on it's journalling (there's
that word again! ) to maintain consistancy. If my raid dies I either
restore from backups or swich over to the mirror's raid system. What I
use for backup is based on a risk analysis of the overall business.

If I'm collecting irreplacable real-time data I write an application
that flushes buffers frequently, and has a highly replicated
infrastructure, and do a business risk assesment that makes sure the
(very low) chance of data loss is in line with the business impact of
the loss, if it happens.

Can the Poster state a scenario where the need to do
sector/cluster-level recovery is a skill I need unless I do recovery
as a full-time service ? There's a place for companies like Ontrack. They
recover data for people that haven't done their backups.

 

[cquirke (MVP Win9x)]

On Tue, 11 May 2004 06:25:51 -0400, "J. Clarke"

If this actually has an effect on your life then you need to fix whatever is
causing your system to keep crashing during writes.

Not a matter of "keep crashing during writes". If things never went
wrong, it wouldn't matter what file system one used, as it would
always work exactly as designed, right?

If it happens *once*, I might want a chance of data recovery, and I
won't get that with NTFS.

-------------------- ----- ---- --- -- - - - -

Tip Of The Day:
To disable the 'Tip of the Day' feature...

 

[chrisv]

I lived in the St. Louis area when I was growing up, and it was a lot
hotter than Texas.

I was in Chicago one summer and it was a lot hotter there in August
than in Texas.

What you left out is that most of Texas does not suffer from snowfall.
Our cool season is very mild and makes for great outdoor activity.

I'll take the heat - you can have the snow.

Well, Texas is a big state. The areas close to the ocean will have
more moderate temperatures than the interior of the state.

 

[Al Dykes]

On Tue, 11 May 2004 06:25:51 -0400, "J. Clarke"


Not a matter of "keep crashing during writes". If things never went
wrong, it wouldn't matter what file system one used, as it would
always work exactly as designed, right?

If it happens *once*, I might want a chance of data recovery, and I
won't get that with NTFS.

You keep saying that. Your use of the word "might" tells me you don't
work where a dollar value can be placed on loss of data, or skilled
staff's time.

I bet that the data recovery experts can recover a corrupted
NTFS disk just as easily as a FAT32 disk.

 

[Alias]

Well, Texas is a big state. The areas close to the ocean will have
more moderate temperatures than the interior of the state.

He obviously hasn't been to the Hill Country in August which is like a hot
oven full of insects.

Alias

 

[Cerridwen]

He obviously hasn't been to the Hill Country in August which is like a hot
oven full of insects.

Alias

You do know that this is an XP help newsgroups right? I thought so!
Now, go off with you chit chat, and do it elsewhere, you're wasting
bandwidth!

 

[Bob]

Well, Texas is a big state.

Texas is larger than any single European country. It is 3 times the
size of the UK, with 1/3 the population.

The areas close to the ocean

I think you mean Gulf

will have more moderate temperatures than the interior of the state.

Indeed, the sea breeze does cool things off a bit. It is routinely
hotter in Dallas (250 mile north of Houston).

However there is a caveat about official temperatures for Houston. The
major airport (IAH) where the official temperatures are taken is 25
miles north of the city proper nestled in the middle of the piney
woods where it is always cooler than the city. My backyard
temperatures always run 5F higher, sometimes up to 10F if it has been
a 100% sunny summer day. And I live next to the county line 15 miles
from the downtown area.


--

Map Of The Vast Right Wing Conspiracy:
http://www.freewebs.com/vrwc/

"You can all go to hell, and I will go to Texas."
--David Crockett

 

[Bob]

He obviously hasn't been to the Hill Country in August which is like a hot
oven full of insects.

You don't even have to go that far - it's hot as Hell in College
Station, which is about 100 miles NW of Houston downtown.


--

Map Of The Vast Right Wing Conspiracy:
http://www.freewebs.com/vrwc/

"You can all go to hell, and I will go to Texas."
--David Crockett

 

[Bob]

You do know that this is an XP help newsgroups right? I thought so!
Now, go off with you chit chat, and do it elsewhere, you're wasting
bandwidth!

You are the one wasting "bandwidth". Bandwidth would not be a problem
if you got DSL or cable.

We'll chat about any thing we want to chat about. If you don't like it
then call 1-800-EAT-SHIT.

Now you bugger off, troll.


--

Map Of The Vast Right Wing Conspiracy:
http://www.freewebs.com/vrwc/

"You can all go to hell, and I will go to Texas."
--David Crockett

 

[Unknown]

How on earth do you waste bandwidth?? You can use it but how do you waste it?
It is there to use..

 

[Mark M]

Don't get mesmerized by cluster size - it's not always
relevant, and in the circumstances discussed here, largely
irrelevant.

On large partitions the file allocation table in FAT32 can grow
relatively large when the cluster size is kept small.

Would this also apply to NTFS on a large 160 GB partition and 4 KB
clusters?

Would this be a sufficient reason to increase cluster size?

 

[Alexander Grigoriev]

It's not what will happen.

Consider expanding a file. Before any data can be written to a file, new
clusters has to be allocated, free clusters bitmap modified, directory entry
modified to reflect new extent allocated (although the file length won't be
written immediately to the directory entry). Only after that you will be
able to write new data to a file.

You're more likely to have a file with garbage data, than to get valid data
truncated. I'd say the latter case is close to impossible.

If you want your data consistent, you have to implement your own transaction
system for your application files. You can use FILE_FLAG_NO_BUFFERING, or,
for special cases, FILE_FLAG_WRITE_THROUGH.

Alex, Windows SDK/DDK MVP

 

[Alexander Grigoriev]

An NTFS bitmap describing all free 4KB clusters for such a drive would be
about 5 MB. Can be easily kept in memory.
FAT of FAT32 drive with 32 KB clusters would be 20 MB.

 

[Plato]

An NTFS bitmap describing all free 4KB clusters for such a drive would be
about 5 MB. Can be easily kept in memory.
FAT of FAT32 drive with 32 KB clusters would be 20 MB.

It's nice to keep your movies, mp3s, etc on a separate fat32 drive.
Large files anyway so no real disk space is wasted. NOT to mention its a
snap to recover them when your XP partition done in NTFS fails.

 

[DILIP]

The jpegs will affect the equation here. The average size of a picture,

Well, do the maths. If files are 150k, and cluster space is 32k
rather than 4k, then the biggest wasteage would be 32k-4k = 28k per
file. Assume actual wastage is half that, i.e. 14k, per file. That
boils down to under 10%; significant, but not carnage.

10% wastage on 10 volumes is one whole hard disk wasted. Significant? Hell
yes, it increases costs! You should've known that considering it's the job
you do.

OTOH, if it's more like 2M per file, this drops to an under 0.7% shrug


Those are 500-*byte* files, not 150 000 - 2 048 000 bytes
What will really waste space would be IE web cache...


Yep - and once again, NTFS isn't quite what it seems.


Haven't seen that, unless you are using disk compression.

Well I have seen it. The data is not recoverable, because it simply gets
corrupted. Data recovery software does little to recover completely corrupt
files.


Read up on

the shuffling NTFS does when it fills up... I can see the same sort of
widening of critical windows there, as it re-balances between MFT and
data space. Problems may also arise when software that assumes
limitless memry (as bounded only by free swap space) hits the wall.

I know about the shuffling it performs. It is not as terrible as you make
it seem. For the benefit of the others reading this thread - Basically 12%
of the initial part of the HD is set aside initially (saving the 16
metafiles) for the MFT to prevent fragmentation (it's not even used at this
point of time). As and when data is required, the MFT is adjusted. Nothing
very "problematic" or sinister in there. I have had an NTFS volume with as
less as 25MB free out of 10GB running with absolutely no problems over a
long period of time.

I've done a lot of data recovery on FATxx, and that's the point; it is
possible to recover lost data in such cases.

Lost data OK. How about corrupt data.

Read: The damaged data is stimply and irreversably destroyed.

The data remains intact. I have had power failures while working with
files and defragging. The files remain - albeit at the cost of the
changes. No issues there. I don't know what and why you're trying to
project this wrongly.

This is what I mean; NTFS "doesn't need" Scandisk simply because it
behaves as if Scandisk was running all the time, with the most
ruthless trade-off between your data and file system integrity.

I don't understand this statement. Cut the flowery language and start
explaining.

That is NOT what I consider "data safety".


Exactly. Some of us would prefer to recover truncated or partial
data, but you don't have that option with NTFS.


Sure; that alerts you to what files are damaged - but at least you
have a chance at recovery of partials. With NTFS, it's simply gone
without a trace, and no way of getting it back.


Specifically, file data and attribute metadata are held in the same
way, as streams. Streams are effectively cluster chains containing
content that hang off a dir-level entry; a structure that's familiar
from working in PICK.


By metadata, I was referring to that within the directory level, not
within the streams. But you do raise an interesting point - that
bloated metadata are not "free" if it effectively means two or more
cluster chains to look at in addition to the dir level.

Earlier on, someone made the claim that NTFS keeps redundant copies of
the dir structure. What I read suggests it's only the first few
entries that are held in that way - in effect, a similar result as
predictable locations for these core structures as applies in FATxx
(where this knowledge is the "redundant copy").

I don't know why you are propagating falsehood. Maybe because by
implementing FAT32 you get more problems to attend to, and hence more
business. Your site is as vague as the rest of your posts (I had not seen
it earlier) and the statement that NTFS is for people who need to hide data
more than recover it is just plain condescending. Don't you understand that
even a basic org would want to implement best practices; public and private
data, templates and files.

Perhaps that's why you and your biz will always cater to small/medium sized
clients. Unless you think big, you don't get big. As far as the sig goes,
regarding certainty, an uncertain person or one in two minds (with a
disclaimer hanging on his neck) will never take crucial decisions - It shows
lack of confidence in yourself. Essentially, your advice could be viewed as
a dis-service to society and the computing fraternity in general, since
you're stuck in a generation no one really cares about.

 

[Al Dykes]

10% wastage on 10 volumes is one whole hard disk wasted. Significant? Hell
yes, it increases costs! You should've known that considering it's the job
you do.

And the value of that disk is a couple hours of tech-person's time,
not to mention the additional hours necessary to "optomize" the system.
Expensive, especially if they've got more important things to do.