Long Term Hard Drive Storage

[Justin]

So I was looking at AVCHD cameras the other day. With my current MiniDV
camcorders I can store the tape for over a decade without a problem. I
have proven this with old 8mm tapes recorded in 1987, and stuck in a
Digital8 camera and Firewired into a PC ad a DV file. There I was at 9
years old.
Stunning.
So for 21 years sitting in a damn cellar, an old analog tape survived.
Apparently new digital tapes will fare better. So I'm guessing the next
time I really checkout the stuff I'm recording to MiniDV will be the
2025 or thereabouts.
AVCHD camcorders don't use tapes as we all know. Would copying the
footage to a portable Firewire/ESATA/USB2 hard drive and storing that
enclosure for a few decades be feasible? How well do hard drives last
when they're not being used? Will the magnetic information deteriorate?
What about magnetic fields?

What if one took a portable enclosure, stuffed a 200GB drive in there,
filled it with his AVCHD footage, stuck it in the factory box, with a
few Silica gel packs, maybe wrapped it with one of those lead liners
that look like trash bags and stored it (properly labeled of course)?

Is that idea stupid?
Forget about cost for now.

 

[Rod Speed]

So I was looking at AVCHD cameras the other day. With my current
MiniDV camcorders I can store the tape for over a decade without a
problem. I have proven this with old 8mm tapes recorded in 1987, and
stuck in a Digital8 camera and Firewired into a PC ad a DV file.
There I was at 9 years old.
Stunning.

So for 21 years sitting in a damn cellar, an old analog tape survived.
Apparently new digital tapes will fare better. So I'm guessing the
next time I really checkout the stuff I'm recording to MiniDV will be
the 2025 or thereabouts.

AVCHD camcorders don't use tapes as we all know. Would
copying the footage to a portable Firewire/ESATA/USB2 hard
drive and storing that enclosure for a few decades be feasible?

Yep. And writing that stuff to both a hard drive and DVDs maximises your chances.

How well do hard drives last when they're not being used?

They last fine.

Will the magnetic information deteriorate?
Nope.

What about magnetic fields?

Same thing.

What if one took a portable enclosure, stuffed a 200GB drive in there,
filled it with his AVCHD footage, stuck it in the factory box, with a
few Silica gel packs, maybe wrapped it with one of those lead liners
that look like trash bags and stored it (properly labeled of course)?

Thats a bit of overkill with the wrapping.

Is that idea stupid?

Nope, very viable.

 

[Justin]

Yep. And writing that stuff to both a hard drive and DVDs maximises your chances.


They last fine.


Same thing.


Thats a bit of overkill with the wrapping.


Nope, very viable.

Damn, that was fast!
Follow up questions... would using a laptop drive be a better idea
since they are designed to take a certain amount of abuse?

Just speaking about a worst case scenario... how strong a magnetic field

 

[Rod Speed]

Damn, that was fast!
Follow up questions... would using a laptop drive be a better idea
since they are designed to take a certain amount of abuse?

If you're likely to be a bit rough with it.

Just speaking about a worst case scenario... how strong a magnetic field

Not clear what you are asking there, maybe the post got sent before it was complete.

 

[Arno Wagner]

So I was looking at AVCHD cameras the other day. With my current MiniDV
camcorders I can store the tape for over a decade without a problem. I
have proven this with old 8mm tapes recorded in 1987, and stuck in a
Digital8 camera and Firewired into a PC ad a DV file. There I was at 9
years old.
Stunning.
So for 21 years sitting in a damn cellar, an old analog tape survived.
Apparently new digital tapes will fare better. So I'm guessing the next
time I really checkout the stuff I'm recording to MiniDV will be the
2025 or thereabouts.
AVCHD camcorders don't use tapes as we all know. Would copying the
footage to a portable Firewire/ESATA/USB2 hard drive and storing that
enclosure for a few decades be feasible?

Depends. You might run itdo component limitations, like capacitor
lifetime (5-10 years), lubrrication fluid increasing its viscosity
(no idea) and the like. HDDs are not removable media designed
for long-term storage.

How well do hard drives last when they're not being used?

I think "unknown" is accurate information here.

Will the magnetic information deteriorate?

That should not be an issue.

What about magnetic fields?

Again not an issue.

What if one took a portable enclosure, stuffed a 200GB drive in there,
filled it with his AVCHD footage, stuck it in the factory box, with a
few Silica gel packs, maybe wrapped it with one of those lead liners
that look like trash bags and stored it (properly labeled of course)?

See above.

Is that idea stupid?

Not stupid, but with a lot of unknowns.

Forget about cost for now.

Then I would strongly advise using archival tape or MOD.
Both have known long-term aging characteristics.

Arno

 

[Al Dykes]

So I was looking at AVCHD cameras the other day. With my current MiniDV
camcorders I can store the tape for over a decade without a problem. I
have proven this with old 8mm tapes recorded in 1987, and stuck in a
Digital8 camera and Firewired into a PC ad a DV file. There I was at 9
years old.
Stunning.
So for 21 years sitting in a damn cellar, an old analog tape survived.
Apparently new digital tapes will fare better. So I'm guessing the next
time I really checkout the stuff I'm recording to MiniDV will be the
2025 or thereabouts.
AVCHD camcorders don't use tapes as we all know. Would copying the
footage to a portable Firewire/ESATA/USB2 hard drive and storing that
enclosure for a few decades be feasible? How well do hard drives last
when they're not being used? Will the magnetic information deteriorate?
What about magnetic fields?

What if one took a portable enclosure, stuffed a 200GB drive in there,
filled it with his AVCHD footage, stuck it in the factory box, with a
few Silica gel packs, maybe wrapped it with one of those lead liners
that look like trash bags and stored it (properly labeled of course)?

Is that idea stupid?
Forget about cost for now.

Make two and keep one off-site.

IMO, the chance of a properly stored disk drive dying are down there
with losing it in a house fire.

Even if the chances are miniscule, it happens to *somebody*.

 

[Al Dykes]

Damn, that was fast!
Follow up questions... would using a laptop drive be a better idea
since they are designed to take a certain amount of abuse?

Just speaking about a worst case scenario... how strong a magnetic field

Unless you take the disk into the MRI exam with you, magnetic fields
are no more a problem with an unused drive than they are for a drive
inside your computer.

As for shock, unpowered disks are more shock resistant than you think.
Look you the detail spec sheet on the manufacturer's web site, you'll
find 100Gs or more acceptable.

Don't use a cheap safe. The cheap "2 hour" safe is made for saving
paper documents. It works by chemical reaction in a fire. It soaks the
papers with hot moisture that would probably kill the electronics on
the disk.

 

[Justin]

If you're likely to be a bit rough with it.


Not clear what you are asking there, maybe the post got sent before it was complete.

Yes it did.
Strange.
My question was; how strong a magnetic field would be needed to erase
the contents of a hard drive?
The poster below me (Al) said something about an MRI and we know that's
strong - stronger than anything that would happen inside the average
American home.
Except maybe an EMP blast... in which case I don't give a rat's ass
about my home videos I'll be looking for food. When civilization gets
back on its feet my DVD backup-backups will be fine.

 

[Rod Speed]

Yes it did.
Strange.

Its easy to do by hitting the wrong key when typing.

My question was; how strong a magnetic field would
be needed to erase the contents of a hard drive?

Much stronger than you will ever see.

The poster below me (Al) said something about an MRI
and we know that's strong - stronger than anything that
would happen inside the average American home.

Yep, and laptops survive those fine in airports etc.

Except maybe an EMP blast... in which case I don't give a rat's ass
about my home videos I'll be looking for food. When civilization gets
back on its feet my DVD backup-backups will be fine.

Yep. And likely the hard drive will be too.

 

[iws]

So I was looking at AVCHD cameras the other day. With my current MiniDV
camcorders I can store the tape for over a decade without a problem. I
have proven this with old 8mm tapes recorded in 1987, and stuck in a
Digital8 camera and Firewired into a PC ad a DV file. There I was at 9
years old.
Stunning.
So for 21 years sitting in a damn cellar, an old analog tape survived.
Apparently new digital tapes will fare better. So I'm guessing the next
time I really checkout the stuff I'm recording to MiniDV will be the
2025 or thereabouts.
AVCHD camcorders don't use tapes as we all know. Would copying the
footage to a portable Firewire/ESATA/USB2 hard drive and storing that
enclosure for a few decades be feasible? How well do hard drives last
when they're not being used? Will the magnetic information deteriorate?
What about magnetic fields?

What if one took a portable enclosure, stuffed a 200GB drive in there,
filled it with his AVCHD footage, stuck it in the factory box, with a
few Silica gel packs, maybe wrapped it with one of those lead liners
that look like trash bags and stored it (properly labeled of course)?

Is that idea stupid?
Forget about cost for now.

I'm not sure I'd trust any mechanical gizmo after sitting for a couple of
decades. What's wrong with recording to DVD's and recopying every few years.
Migrate to new technology as it becomes established. For example, if and
when Blue Ray becomes fully established, migrate to that. There will always
be a lengthy transition to new technology and in many cases, the same device
will be fully compatible with both old and new - e.g. CD/DVD writers. Many
current Blue Ray writers will write to CD and DVD. Hell, you can still buy
floppy drives that work in today's PC's and that technology is more than a
quarter century old. Keep in mind that you're talking digital recording and
copying with no real loss of information after many copies.

 

[Justin]

I'm not sure I'd trust any mechanical gizmo after sitting for a couple of
decades. What's wrong with recording to DVD's and recopying every few years.
Migrate to new technology as it becomes established. For example, if and
when Blue Ray becomes fully established, migrate to that. There will always
be a lengthy transition to new technology and in many cases, the same device
will be fully compatible with both old and new - e.g. CD/DVD writers. Many
current Blue Ray writers will write to CD and DVD. Hell, you can still buy
floppy drives that work in today's PC's and that technology is more than a
quarter century old. Keep in mind that you're talking digital recording and
copying with no real loss of information after many copies.

I understand the whole "copy as time goes on" and you're correct, that
is the best solution.
But, as life goes on schedules get bitter and after a while that sort of
thing slips through the cracks. I should change my air filter on the
central air system. Its just "one of those things" that should get
done, but never does.
When I took a class trip in 1994 I took those 8mm tapes and stored them.
I graduated high shcool in 1995, went to college and around 1999 I ran
into a group of people from the trip. I promised I would convert all
those tapes to VHS and we could all meet and have a "Europe 1994" party.
What happened? Finals. Internship. Summer accelerated classes.
Streaking. Personal issues. Dad's bypass. Basically life gets in the
way of things that matter.
In 2003 the same thing happened. I ran into the teacher who organized
the trip back then. I said I would edit the tapes and convert them to DVD.
Never happened. What DID happen? My Masters degree. More accelerated
classes. A lousy job that sends me to horrible places.
Fast forward again to February 2008. I run into the same teacher. Said
the same thing, convert to DVD blah blah... I got a false start because
Sony Vegas decided to putz out on me. I was reduced to Movie Maker.
Didn't get done... What happened? My decision to get out of I.T. and
into Accounting. Now for the next two years nothing but accounting
classes starting mid April.
So this time I got it done. I have all the edited DV files ready to by
written to DVD. I also have the DV files saved to my archival hard
drive and MiniDV.

 

[Dave]

What if one took a portable enclosure, stuffed a 200GB drive in there,
filled it with his AVCHD footage, stuck it in the factory box, with a
few Silica gel packs, maybe wrapped it with one of those lead liners
that look like trash bags and stored it (properly labeled of course)?

Is that idea stupid?
Forget about cost for now.

I think your biggest issue would be:

1) Finding something to read the hard disk or DVD. In a few decades
time, I doubt it will be easy to find something to read a SATA disk, in
much the same way as you would have a very hard time finding a computer
to read an 8" floppy disk. Same goes for a DVD storage - do you really
believe in a few decades time anything will read a DVD?

The SCSI interface has to date been more stable than any other.

2) Decay of electrolytic capacitors is likely to be an isssue.

3) I assume the most likely things you would want to keep over decades
would be photographs. Experience has shown images in stone have lasted
thousands of years. Getting the images carved into stone or metal would
probably be best.

4) Getting images painted by an artist would be very good and not
particularly expensive. We know many paintings have lasted centuries.
Not as long as stone/metal to date, but more practical and gives better
colour information.

5) Experience has shown many photographs have lasted well when using wet
chemistry - silver based photographic medium.

5) Nobody really knows how well digital data will store. My suggestions
for the best chance would be to

i) Keep on a few different media - optical, magnetic.
ii) Transfer it to current technology every 5-10 years. Keep the old
media anyway.
iii) Keep data in different locations.

iv) I dont think external magnetic fields would be an issue at all, but
I know mu-metal would be more suitable than lead for that purpose. But
that really would be the last of your worries.
v) I used to work developing protection systems for nuclear
electromagnetic pulse. I am aware the aim would be to explode a bomb
that would damage electronics, but not kill people. So after an EMP
strike, food might not be the issue you suspect.

On the assumption you want to store photographs, I suggest

stone > oil painting > photographs > digital storeage.

In principle, digital storage would not degrade over time, but I think
the chances of the images lasting in practice is a lot less than with
phtographs or oil painting.

 

[Al Dykes]

I wouldn't rely on tape, even in digital format, as this medium is
vulnerable to mechanical damage.

And unless you a are dealing with professional tape equipment, the
chances of getting a compatible drive hardware and restore application
that works on a machine you own years from is close to zero.

Tape storage lifetime has it's own issues. I know in the past, the
manufacturer quoted 10 years under controlled conditions.

 

[Arno Wagner]

And unless you a are dealing with professional tape equipment, the
chances of getting a compatible drive hardware and restore application
that works on a machine you own years from is close to zero.

Tape storage lifetime has it's own issues. I know in the past, the
manufacturer quoted 10 years under controlled conditions.

Tape has the advantaghe, that its characteristics are very well known.
If you use professional archival tape, it will live as long
as the manufacturer claims.

As to restore application, use GNU tar. It has been around 30
years or so, and it will still be arounf in 50. For the data
format itself, again use simple, well documented formats. ASCII,
gif/TIFF/png, PostScript, html. It may be a good idea to include
source code for converters. Stay away from any closed and/or
complicated format, such as all the Microsoft File formats.
Using them is asking to pay a huge amount to have them
professionally converted. (And I have been tempted to prepare
a company that does this for a long time now. Just so boring....)

Arno

 

[Al Dykes]

Tape has the advantaghe, that its characteristics are very well known.
If you use professional archival tape, it will live as long
as the manufacturer claims.

As to restore application, use GNU tar. It has been around 30
years or so, and it will still be arounf in 50. For the data
format itself, again use simple, well documented formats. ASCII,
gif/TIFF/png, PostScript, html. It may be a good idea to include
source code for converters. Stay away from any closed and/or
complicated format, such as all the Microsoft File formats.
Using them is asking to pay a huge amount to have them
professionally converted. (And I have been tempted to prepare
a company that does this for a long time now. Just so boring....)

Arno

Good points, all, with the addition that the backup has to be done
with tar, also.

Tar can also be used to backup to CD/DVD media. As Arno says, a tar
saveset will be understood for as long as anyone cares to care about
it.

Thought has to be given to the file formats of the files in the tar
saveset. One advantage of software over hardware is that it can run on
furure machines with just a little on the part of some programmer.

Whatever you do, test test test. Try to restore on a machine as
unlike yours as possible. A Windows tar backup can be tested on a MAC
or a Unix box.

Emulation is a big win for archivists. It's safe to say that even
after x86 hardware is found only in museums, all versions of Microsoft
Windows will be running nicely in emulation much faster than it does
natively, today.

 

[Al Dykes]

Depends. You might run itdo component limitations, like capacitor
lifetime (5-10 years), lubrrication fluid increasing its viscosity
(no idea) and the like. HDDs are not removable media designed
for long-term storage.


I think "unknown" is accurate information here.


That should not be an issue.


Again not an issue.


See above.


Not stupid, but with a lot of unknowns.


Then I would strongly advise using archival tape or MOD.
Both have known long-term aging characteristics.

MOD = Magneto-Optical Disk.

This was the gold standard for archival storage when I worked at
BigBank, which was up to 1993. We had lots in storage in Iron
Mountain and online in jukeboxes the size of large refrigerators.


http://en.wikipedia.org/wiki/Magneto-optical_drive

 

[Justin]

MOD = Magneto-Optical Disk.

This was the gold standard for archival storage when I worked at
BigBank, which was up to 1993. We had lots in storage in Iron
Mountain and online in jukeboxes the size of large refrigerators.


http://en.wikipedia.org/wiki/Magneto-optical_drive

What about DVD-RAM?

 

[Rod Speed]

Al Dykes wrote

What about DVD-RAM?

Much more viable than MOD.

 

[Rod Speed]

I think your biggest issue would be:

1) Finding something to read the hard disk or DVD.
Nope.

In a few decades time, I doubt it will be easy to find something to read a SATA disk, in much the same way as you
would have a very hard time finding a computer to read an 8" floppy disk.

Bet it will be trivial to find something to read a USB drive.

Same goes for a DVD storage - do you really believe in a few decades time anything will read a DVD?

Yep, just like the current drive read CDs just fine.

The SCSI interface has to date been more stable than any other.

Wrong, USB leaves it for dead.

2) Decay of electrolytic capacitors is likely to be an isssue.

Nope. And the DVDs protect you against that small risk.

3) I assume the most likely things you would want to keep over decades would be photographs. Experience has shown
images in stone have lasted thousands of years. Getting the images carved into stone or metal would probably be best.

His are videos, DVDs are a hell of a lot more practical.

4) Getting images painted by an artist would be very good and not particularly expensive.

Tad labor intensive with videos.

We know many paintings have lasted centuries. Not as long as stone/
metal to date, but more practical and gives better colour information.

But doesnt work for videos.

5) Experience has shown many photographs have lasted well when using wet chemistry - silver based photographic medium.

Trouble is that you dont get color that way.

5) Nobody really knows how well digital data will store.

We do know that all you really have to do is keep updating the format.

My suggestions for the best chance would be to

i) Keep on a few different media - optical, magnetic.
ii) Transfer it to current technology every 5-10 years. Keep the old media anyway.
iii) Keep data in different locations.

iv) I dont think external magnetic fields would be an issue at all,
but I know mu-metal would be more suitable than lead for that
purpose. But that really would be the last of your worries.

v) I used to work developing protection systems for nuclear
electromagnetic pulse. I am aware the aim would be to explode a bomb that would damage electronics, but not kill
people. So after an EMP strike, food might not be the issue you suspect.

Nuke attack aint gunna happen.

On the assumption you want to store photographs,

Dud assumption.

I suggest

stone > oil painting > photographs > digital storeage.

And since its video, the only thing thats practical is digital storage.

In principle, digital storage would not degrade over time, but I think the chances of the images lasting in practice
is a lot less than with phtographs or oil painting.

Its a tad unlikely that he will be around in millennia to watch the video.

 

[Arno Wagner]

Al Dykes wrote:

What about DVD-RAM?

It looks good in teory, but in practice less so. I recently
bought a drive and disks. If you are really careful, they
should last 2-3 decades, but the disk do not have cartridges
(at least for the rives you can get) and one bad drop
could possibly be enough.

Unfortunately MOD technology has not been developed further
for some years now, and it looks like it will not be. At
the moment DVD RAM seems to be the best option for small
volume long-term storage. Whether it can perform past the
10 year mark is not really clear to me.

Arno