Deframenter That Will Create New Hard Drive?

[Will]

After seeing how much work some defragmenters will go through to do their
job, I'm wondering does anyone make a defragmenter that could be ordered to
actually construct a new drive as the result of the defragmentation? In
other words, rather than just shuffling sectors around, the fragmenter would
write out a whole new drive, with all of the sectors nicely ordered.

This would likely be a much faster way to defragment for those extreme cases
when you want to physically order files based on filename spelling, so that
consecutive files are contiguous on the drive. This is important for games
like flight simulators that need to load thousands of very small files in
very short periods of time, where disk latency accumulates across the large
numbers of files and really does affect load times.

It seems to me that this kind of defragmenter might also make an interesting
way to do an image backup of a drive while it is in use. Obviously there is
a risk if doing that for certain types of database files that are actively
in use at the time of the defragmentation, but caveat emptor and all that.

 

[CJT]

After seeing how much work some defragmenters will go through to do their
job, I'm wondering does anyone make a defragmenter that could be ordered to
actually construct a new drive as the result of the defragmentation? In
other words, rather than just shuffling sectors around, the fragmenter would
write out a whole new drive, with all of the sectors nicely ordered.

This would likely be a much faster way to defragment for those extreme cases
when you want to physically order files based on filename spelling, so that
consecutive files are contiguous on the drive. This is important for games
like flight simulators that need to load thousands of very small files in
very short periods of time, where disk latency accumulates across the large
numbers of files and really does affect load times.

It seems to me that this kind of defragmenter might also make an interesting
way to do an image backup of a drive while it is in use. Obviously there is
a risk if doing that for certain types of database files that are actively
in use at the time of the defragmentation, but caveat emptor and all that.

I've been doing that with Ghost for some time.

 

[Rod Speed]

After seeing how much work some defragmenters will go through
to do their job, I'm wondering does anyone make a defragmenter
that could be ordered to actually construct a new drive as the
result of the defragmentation? In other words, rather than just
shuffling sectors around, the fragmenter would write out a whole
new drive, with all of the sectors nicely ordered.

Some backup system produce that result with the restore.

This would likely be a much faster way to defragment for those
extreme cases when you want to physically order files based on
filename spelling, so that consecutive files are contiguous on the drive.

You dont need to defrag for that, you dont need to
move the files around, just the entrys in the directorys.

And you dont necessarily need to sort them anyway
if the directorys are properly cached in memory.

This is important for games like flight simulators that need
to load thousands of very small files in very short periods
of time, where disk latency accumulates across the large
numbers of files and really does affect load times.

Nope, that's what the OS file cache is for.

It seems to me that this kind of defragmenter might also make an
interesting way to do an image backup of a drive while it is in use.

No point, real imagers do much more, particularly incremental images.

 

[Will]

I've been doing that with Ghost for some time.

Ghost is just imaging right? In my method you defrag and then end up with
an image backup as a side effect. To do things with Ghost you need to
defrag and then image separately, taking more time.

 

[Will]

Some backup system produce that result with the restore.

True enough. It's all about how much time do you have as usual. That
method is not optimal because you must backup, then verify, then get a third
volume and restore (assuming you don't want to risk your original).

drive.

You dont need to defrag for that, you dont need to
move the files around, just the entrys in the directorys.

No, that is just wrong. Assume that you need to open 1000 small files that
are highly fragmented on the disk. Each file, because it is not contiguous
with the previous file, requires a disk seek, with a typical seek for a SATA
drive being 9 ms. For a 7200 rpm drive, once you have seeked to a track,
you have an average 4 ms of latency to get to the start sector for the file.
That is 12 ms of latency per file, and I need to open 1000 files. Simple
math tells you that you have 12,000 ms - 12 full seconds - of extra loading
time introduced in this scenario attributable to physical fragmentation of
the files on disk.

As a simple thought experiment, assume that all of the files are organized
by filename sequentially in the directory, which is your proposed solution.
The time to load that entire directory would never be more than 100 or 200
ms, and that would be a huge directory. The additional time spent by the
computer to sort even 2000 filenames in an unsorted directory is not more
than maybe another 50 ms. I'm being generous to your argument. So your
solution is based on optimizing a process that takes at most 250 ms and
saves at most maybe 50 ms, and you are ignoring the very real and very
measurable costs associated with the seek and latency for loading 1000
files.

50 ms in savings versus up to 12,000 ms in savings is not a good tradeoff.

Nope, that's what the OS file cache is for.

In a perfect world I would have a 12 GB memory cache in front of the file
system and some tool to preload that cache. Maybe next year, but not on
this year's hardware, thanks.

 

[Rod Speed]

True enough. It's all about how much time do you have as usual.
That method is not optimal because you must backup, then verify,

You dont really need to verify.

then get a third volume and restore
(assuming you don't want to risk your original).

Yes, but defragging is a complete waste of time anyway.

No, that is just wrong.
Nope.

Assume that you need to open 1000 small
files that are highly fragmented on the disk.

Small files are very unlikely to be fragmented, they normally wont
occupy more than one cluster and so by definition cant be fragmented.

Each file, because it is not contiguous with the previous file,

Thats not what fragmented means. That is poorly organised.

requires a disk seek, with a typical seek for a SATA drive being 9 ms.

You aint established that that time is significant as
part of the time involved in loading those 1K files.

For a 7200 rpm drive, once you have seeked to a track, you have
an average 4 ms of latency to get to the start sector for the file.
That is 12 ms of latency per> file, and I need to open 1000 files.

You've mangled the maths there too.

Simple math tells you that you have 12,000 ms - 12 full
seconds - of extra loading time introduced in this scenario
attributable to physical fragmentation of the files on disk.

A properly written app wont have 1K of small files.

As a simple thought experiment, assume that all of the files are
organized by filename sequentially in the directory, which is your
proposed solution.

No it isnt. I was just proposing sorting the FILE ENTRYS
IN THE DIRECTORY, not the files themselves.

The time to load that entire directory would
never be more than 100 or 200 ms,

Wrong again.

and that would be a huge directory.

And an obscenely badly written app.

The additional time spent by the computer to sort even 2000 filenames
in an unsorted directory is not more than maybe another 50 ms.

They arent sorted.

I'm being generous to your argument.

You havent even managed to grasp what I was suggesting.

So your solution is based on optimizing a process that
takes at most 250 ms and saves at most maybe 50 ms,
Wrong.

and you are ignoring the very real and very measurable costs
associated with the seek and latency for loading 1000 files.
Nope.

50 ms in savings versus up to 12,000 ms in savings is not a good tradeoff.

Only the most badly written app would attempt
to load 1K files from the drive as fast as it can.

In a perfect world I would have a 12 GB memory cache in
front of the file system and some tool to preload that cache.

You dont need a 12 GB cache when its 1K very small files.

Maybe next year, but not on this year's hardware, thanks.

Dont need any new hardware, just have whatever
is in those 1K files in a single file, stupid.

 

[Will]

You've mangled the maths there too.

Do the math correctly for us. There isn't any point to responding to your
many generalizations that have zero hard data or mathematical analysis as
backup to the points you want to make. No free rides. Do the math and
commit it to public scrutiny.

 

[Rod Speed]

Do the math correctly for us.

Just how many of you are there between those ears ?

No point in the math, any app designer with a clue wouldnt
have 1K of tiny files all loaded at once as fast as it can.

There isn't any point to responding to your many
generalizations that have zero hard data or mathematical
analysis as backup to the points you want to make.

Dont need any 'mathematical analysis' to realise that the
only thing that makes any sense at all is to have that data
in a single file instead of 1K separate tiny files if you are
going to load all that data as quickly as possible, stupid.

No free rides. Do the math and commit it to public scrutiny.

Go and **** yourself.

 

[CJT]

there is



Ghost is just imaging right? In my method you defrag and then end up with
an image backup as a side effect. To do things with Ghost you need to
defrag and then image separately, taking more time.

Not so, at least on the version I use. You can create a true image by
setting an option, but the default does a file-by-file copy.

 

[Will]

Just how many of you are there between those ears ?

No point in the math, any app designer with a clue wouldnt
have 1K of tiny files all loaded at once as fast as it can.


Dont need any 'mathematical analysis' to realise that the
only thing that makes any sense at all is to have that data
in a single file instead of 1K separate tiny files if you are
going to load all that data as quickly as possible, stupid.

Rod, you are changing the subject. I calculated the latency attributable
to opening 1000 non contiguous files on a SATA drive. You responded to
that by saying I did the math wrong in that calculation, and you supplied no
math of your own. If I did the math wrong, then do the math right and
explain what was wrong with the number I gave.

Now you are trying to dodge your own claim and instead change the topic to
how to redesign a commercial game to do things in a different way than it
does it. Okay, the game is designed badly. It opens a lot of very small
files in a very short time. Sorry. You can go convince Microsoft that
what they are doing is wrong. I'm sure they had a reason for it, but I
wasn't here to play Don Quixote with Microsoft. I was here to try to just
solve the real world problem I need to solve, which was to make the files
contiguous on the drive to avoid the latency of opening those files when
they are non contiguous.

Your solution for creating a drive that lines up the files contiguoulsly by
using a backup and restore is a good solution. No problems there. But if
you are going to say to someone that their calculation of some effect is
wrong, you should be prepared to present your own numbers, not just change
the topic to a variety of other sub topics. Otherwise your claim doesn't
have credibility. Saying that you don't need to backup your claims because
there are "other problems" doesn't establish your initial claim either.

 

[Rod Speed]

Rod, you are changing the subject.

Nope. Its stupid obsessing about how best to handle loading 1K
separate tiny files as fast as possible when only a completely
****ed app would ever do it like that when it will load MUCH
faster with all that data in a single file instead.

I calculated the latency attributable to opening
1000 non contiguous files on a SATA drive.

And that is completely irrelevant.

You responded to that by saying I did the math wrong in
that calculation, and you supplied no math of your own.

Because the detail of how to calculate it correctly is completely
irrelevant when its a completely pointless calculation in the first place.

If I did the math wrong, then do the math right and
explain what was wrong with the number I gave.

I'll rub your nose in the stupidity and irrelevancy of that calculation instead.

Now you are trying to dodge your own claim

Nope. Just rubbing your nose in the stupidity and irrelevancy of that calculation.

and instead change the topic

Another lie.

to how to redesign a commercial game to do things in a
different way than it does it. Okay, the game is designed
badly. It opens a lot of very small files in a very short time.

So its stupid to be working out how to organise the files on the
drive to minimise the time that takes, when the only sensible
thing to do is to toss that stupid approach in the bin instead.

Sorry. You can go convince Microsoft that what they are doing is wrong.

Dont need to, MS is completely irrelevant.

I'm sure they had a reason for it,

Yep, whoever did it like that doesnt actually have a ****ing clue.

And they dont do it like that anyway.

but I wasn't here to play Don Quixote with Microsoft. I was
here to try to just solve the real world problem I need to solve,
which was to make the files contiguous on the drive to avoid
the latency of opening those files when they are non contiguous.

What you want to do isnt even possible, because what
you use to reorganise the files has absolutely no way
way of knowing what order the app will load the files in.

Your solution for creating a drive that lines up the files
contiguoulsly by using a backup and restore is a good solution.

Nope, because that isnt going to be the order in which
the files will be loaded by that completely ****ed app.

No problems there. But if you are going to say to someone
that their calculation of some effect is wrong, you should be
prepared to present your own numbers,

No point when the calculation is completely irrelevant because
its never going to be possible to organise the files on the drive
since you have no way of knowing what order the app will be
loading the files in. You cant even snoop on the app, because
it wont always be loading files in the same order either.

not just change the topic to a variety of other sub topics.

Pity I never ever did that.

Otherwise your claim doesn't have credibility.

You wouldnt know what credibility was if it bit you on your lard arse.

You couldnt even manage to work out the difference between
fragmented files and contiguous files was until your nose was
rubbed in that. And I dont plan to bother to provide any cites
on the difference either.

Saying that you don't need to backup your claims because there
are "other problems" doesn't establish your initial claim either.

It doesnt need to. Your stupid calculation is completely irrelevant.

 

[JohnH]

After seeing how much work some defragmenters will go through to do their
job, I'm wondering does anyone make a defragmenter that could be ordered to
actually construct a new drive as the result of the defragmentation? In
other words, rather than just shuffling sectors around, the fragmenter would
write out a whole new drive, with all of the sectors nicely ordered.

This would likely be a much faster way to defragment for those extreme cases
when you want to physically order files based on filename spelling, so that
consecutive files are contiguous on the drive. This is important for games
like flight simulators that need to load thousands of very small files in
very short periods of time, where disk latency accumulates across the large
numbers of files and really does affect load times.

It seems to me that this kind of defragmenter might also make an interesting
way to do an image backup of a drive while it is in use. Obviously there is
a risk if doing that for certain types of database files that are actively
in use at the time of the defragmentation, but caveat emptor and all that.

Doing just a (drag and drop) file copy will accomplish that.