Cheapest massive archive storage solution?

[Jeff]

I have a project that would feed a continual stream of data at a rate of
approximately 17 GB per day at a minimum. My project would require archiving
this data indefinately. I would like to know of the cheapest possible
solution to store 17 GB per day for as long of a time span as possible.

I would highly prefer to have a medium with an extremely long shelf life.
That is, I do not want to find myself 10 years from now trying to copy 62
Terrabytes of data off of piles and piles of old storage discs/cartridges
onto a new medium to preserve the data. I want it to last as long as
possible from the get go.

Another important thing to consider is the schedule of when discs/cartridges
would have to be replaced. I will not be able to physically replace storage
media any more frequently than once per 24 hour cycle. That is, the
individual discs or cartridges must store a minimum of 17 GB each. I would
highly prefer this to be on a 48 hour cycle or more, as I might not be
around every single day. That is, I'd prefer to have 34 GB or more per
disc/cartridge.

I imagine I will have multiple physical drives setup in such a way that as
soon as one drive is full, the server begins saving information to the other
drive(s). This way when I arrive to change media, I can change all of the
drives that are full while the system is still writeing data to the one
active drive. This could help offset the need for larger capacities per
unit, as I could have multiple units running live at any given moment.

I realize this is going to cost quite a bit to accomplish over the next
decade or two. However, I would like to keep the costs down as low as
possible. This is really just a hobby to me, so anything that must be
purchased will come soley out of my pocket with no commercial backing.

What is the cheapest possible way to archive 17 GB per day and save it for
decades to come? Would magnetic tape have too short of a shelf life? Would
optical discs be too expensive?


Thanks for your help,

-Jeff

 

[J.Clarke]

I have a project that would feed a continual stream of data at a rate
of approximately 17 GB per day at a minimum. My project would require
archiving this data indefinately. I would like to know of the cheapest
possible solution to store 17 GB per day for as long of a time span as
possible.

Which do you want, do you want the cheapest solution or the one that
holds the data for the longest possible span?

I would highly prefer to have a medium with an extremely long shelf
life. That is, I do not want to find myself 10 years from now trying
to copy 62 Terrabytes of data off of piles and piles of old storage
discs/cartridges onto a new medium to preserve the data. I want it to
last as long as possible from the get go.

In that case you're talking CD or magneto-optical or possibly DVD.

Another important thing to consider is the schedule of when
discs/cartridges would have to be replaced. I will not be able to
physically replace storage media any more frequently than once per 24
hour cycle. That is, the individual discs or cartridges must store a
minimum of 17 GB each.

If you have to have 17 GB minimum capacity then your best bet is
probably DLT, which is good for 10-20 years.

I would highly prefer this to be on a 48 hour
cycle or more, as I might not be around every single day. That is, I'd
prefer to have 34 GB or more per disc/cartridge.

DLT will handle that nicely.

I imagine I will have multiple physical drives setup in such a way
that as soon as one drive is full, the server begins saving
information to the other drive(s).

In that case why do you need 17 GB on each cartridge?

This way when I arrive to change
media, I can change all of the drives that are full while the system
is still writeing data to the one active drive. This could help offset
the need for larger capacities per unit, as I could have multiple
units running live at any given moment.

I realize this is going to cost quite a bit to accomplish over the
next decade or two. However, I would like to keep the costs down as
low as possible. This is really just a hobby to me, so anything that
must be purchased will come soley out of my pocket with no commercial
backing.

What is the cheapest possible way to archive 17 GB per day and save it
for decades to come? Would magnetic tape have too short of a shelf
life? Would optical discs be too expensive?

You have a tradeoff to make, and only you can know how much you are
willing to spend for what capability.

 

[Arno Wagner]

I have a project that would feed a continual stream of data at a rate of
approximately 17 GB per day at a minimum. My project would require archiving
this data indefinately. I would like to know of the cheapest possible
solution to store 17 GB per day for as long of a time span as possible.

I would highly prefer to have a medium with an extremely long shelf life.
That is, I do not want to find myself 10 years from now trying to copy 62
Terrabytes of data off of piles and piles of old storage discs/cartridges
onto a new medium to preserve the data. I want it to last as long as
possible from the get go.

Another important thing to consider is the schedule of when discs/cartridges
would have to be replaced. I will not be able to physically replace storage
media any more frequently than once per 24 hour cycle. That is, the
individual discs or cartridges must store a minimum of 17 GB each. I would
highly prefer this to be on a 48 hour cycle or more, as I might not be
around every single day. That is, I'd prefer to have 34 GB or more per
disc/cartridge.

I imagine I will have multiple physical drives setup in such a way that as
soon as one drive is full, the server begins saving information to the other
drive(s). This way when I arrive to change media, I can change all of the
drives that are full while the system is still writeing data to the one
active drive. This could help offset the need for larger capacities per
unit, as I could have multiple units running live at any given moment.

I realize this is going to cost quite a bit to accomplish over the next
decade or two. However, I would like to keep the costs down as low as
possible. This is really just a hobby to me, so anything that must be
purchased will come soley out of my pocket with no commercial backing.

What is the cheapest possible way to archive 17 GB per day and save it for
decades to come? Would magnetic tape have too short of a shelf life? Would
optical discs be too expensive?

For your requirements you need an MOD storage libary or multiple MOD
drives. Either with 2.3GB 3.5" disks or 5.25" disks. This is the
only solution that will give you the shelf-life you want. Others claim
to reach it, but fail. CD-R/DVD-R will likely last several years but
not several decades. Tapes of any sort get problematic after a decade.
Same for HDDs. In addition MOD is the only medium type where the
manufacturers are comitted to long term drive availability and
backwards compatibility, since it was explicitely designed for
professional long-term storage without a special storage
environment.

One option would be to use 10 conventional 3.5" MOD drives with
some software that switches to the next disk when one is full.
This would cost about 400 Euro per USB drive and and around 25 Euro
per disk. Say 4000 Euro basic investment plus 200 Euro/day.

If you can spend more money, you could go for an optical jukebox
like this:

http://www.promarktech.com/storage/mo/plasmon/m104.htm

Gives you up to 55 days with your storage needs before you
have to replace the cartridges. The basic equipment will be a little
expensive. A 5.4GB disk is around 55 Euro, so you can expect an
additional cost of around 200 Euro/day, just like in the above case.
I could not find a price quote for 9GB disks.

If you cannot spend these 200 Euro/day, then forget it. You will
find ceaper solutions that may keep half a decade or maybe even a
decade, but not longer.

Arno

 

[Papa]

I don't have an answer, but I do know that whatever media device you choose
will be obsolete in far too few years. For long term storage, you need
durable media - but you will also need to have on hand the media device
(tape drive, DVD drive, or whatever) because there is a very good
possibility that such a device will be unavailable a few years from now.
Computer technology is a rapidly evolving entity.

I think that most who have been involved in archiving find that copying data
from an older medium to a more modern one is a necessary and periodic
process. In other words, do not just store the data now and expect to be
able to retrieve it 10 to 15 years down the road. It probably should be
transferred to newer media much more frequently than that.

My opinion. Yours may differ.

 

[Timothy Daniels]

[..........]
I think that most who have been involved in archiving find that copying data
from an older medium to a more modern one is a necessary and periodic
process. In other words, do not just store the data now and expect to be
able to retrieve it 10 to 15 years down the road. It probably should be
transferred to newer media much more frequently than that.

True. When I left a job 12 years ago, I archived lots of my work on
5 1/4 in. floppies. I recently had a reason to access some of that stuff,
and guess what - no one I know has a 5 1/4 in. drive, much less one
that will run on Windows.

*TimDaniels*

 

[danco]

I have a project that would feed a continual stream of data at a rate of
approximately 17 GB per day at a minimum. My project would require archiving
this data indefinately. I would like to know of the cheapest possible
solution to store 17 GB per day for as long of a time span as possible.

I saw a 400 DVD changer drive at Fry's Electronics the other day.
It was selling for "just" US$1600. One places the 400 DVDs vertically
within a circular carosel (a torus). I have no idea what sort of software
is used to drive it though. I wonder if they make a +- BURNER version.

- Dan

 

[Jeff]

[..........]
I think that most who have been involved in archiving find that copying data
from an older medium to a more modern one is a necessary and periodic
process. In other words, do not just store the data now and expect to be
able to retrieve it 10 to 15 years down the road. It probably should be
transferred to newer media much more frequently than that.

True. When I left a job 12 years ago, I archived lots of my work on
5 1/4 in. floppies. I recently had a reason to access some of that stuff,
and guess what - no one I know has a 5 1/4 in. drive, much less one
that will run on Windows.

*TimDaniels*

Very true indeed. Thanks for pointing that out. I guess I won't have any
possible way of maintaining a storage media for over 10 years or so. So I
will have to be in a continual process of converting old data over to newer
storage mediums every few years. The biggest problem I see with this is that
it would be very time consuming if there were numerous cartridges
(disks/whatever). What that means to me is that I need the individual units
to be able to store massive amounts of data. If the cartridges were 17 GB
each, I would have to replace them every day. If I then had to convert them
to a newer media format 10 years down the road, that would be 3650
cartridges/disks/whatever to swap in and out of drives. That would be way
too many to be able to practically convert to a new format. It would be a
lot easier if I had a storage medium that would last a week instead of a day
(say 120 GB instead of 17 GB) on each cartridge / disk. Then I'd only have
520 disks to convert 10 years down the road.


Thanks,

Jeff

PS: The irony in this however is that I happen to have several 5.25" disk
drives sitting around in my basement, and yes they do run in Windows. None
of them are in a computer that is actually running, but it would probably
only take me 10 minutes or so to toss one in if I needed it. Of course, if
you're talking about disks you made with an Apple or Commadore or some other
funky computer, I have no idea what I'd do with them.

 

[Jeff]

I saw a 400 DVD changer drive at Fry's Electronics the other day.
It was selling for "just" US$1600. One places the 400 DVDs vertically
within a circular carosel (a torus). I have no idea what sort of software
is used to drive it though. I wonder if they make a +- BURNER version.

- Dan

I really like that idea! I could just pile on the disks once in a while and
not have to keep a close eye on it all the time. The only thing is, a DVD
only holds about 4.7 GB. I'll be needing around 17 GB per day, so that would
go through about 25 DVDs a week. After 10 years I'd have 13 thousand DVDs to
keep up with. That'd be quite a collection. I think I'll keep looking for
something that can hold more data on each disk/cartridge/unit/whatever.

Thanks though,

-Jeff

 

[Rod Speed]

[..........]
I think that most who have been involved in archiving find that copying data
from an older medium to a more modern one is a necessary and periodic
process. In other words, do not just store the data now and expect to be
able to retrieve it 10 to 15 years down the road. It probably should be
transferred to newer media much more frequently than that.

True. When I left a job 12 years ago, I archived lots of my work on
5 1/4 in. floppies. I recently had a reason to access some of that stuff,
and guess what - no one I know has a 5 1/4 in. drive, much less one
that will run on Windows.

Very true indeed. Thanks for pointing that out. I guess I won't have any
possible way of maintaining a storage media for over 10 years or so.

Dunno. That drive availability should be fine for CDs and DVDs.

The media lasting that long is more of a dubious proposition with DVDs currently.

So I will have to be in a continual process of converting
old data over to newer storage mediums every few years.

What the hell is that volume, 17GB a day ?

The biggest problem I see with this is that it would be very time
consuming if there were numerous cartridges (disks/whatever).

Yeah, you'd certainly need an automatic changer, both with the
old and the new. Likely DVD would be viable there, at a price.

What that means to me is that I need the individual units to
be able to store massive amounts of data. If the cartridges
were 17 GB each, I would have to replace them every day.

How crucial is that data if you say lose 1% of them ?

If I then had to convert them to a newer media
format 10 years down the road, that would be 3650
cartridges/disks/whatever to swap in and out of drives.

Yeah, tho changers are buyable, at a price.

That would be way too many to be able
to practically convert to a new format.

Not with a changer.

It would be a lot easier if I had a storage medium
that would last a week instead of a day (say 120
GB instead of 17 GB) on each cartridge / disk.

Yeah, and starting to be viable to use
hard drives, a new one every week etc.

Dunno that I'd like to risk them lasting 10 years tho, even
if stored ideally. Probably viable, but you could end up with
spectacular problems if say the bearings gum up over that time.

Then I'd only have 520 disks to convert 10 years down the road.

Yeah, might be viable with hard drive I spose. Tho you'd
really have to duplicate them if you cant risk any losses at all.

PS: The irony in this however is that I happen to have several 5.25" disk
drives sitting around in my basement, and yes they do run in Windows.

Yeah, me too.

None of them are in a computer that is actually running, but it would
probably only take me 10 minutes or so to toss one in if I needed it.

Yeah, very likely.

Of course, if you're talking about disks you made
with an Apple or Commadore or some other funky
computer, I have no idea what I'd do with them.

You could use them as frizbees.

 

[anonymous]

[..........]
I think that most who have been involved in archiving find that copying data
from an older medium to a more modern one is a necessary and periodic
process. In other words, do not just store the data now and expect to be
able to retrieve it 10 to 15 years down the road. It probably should be
transferred to newer media much more frequently than that.

True. When I left a job 12 years ago, I archived lots of my work on
5 1/4 in. floppies. I recently had a reason to access some of that stuff,
and guess what - no one I know has a 5 1/4 in. drive, much less one
that will run on Windows.

*TimDaniels*

I kept mine

it's in the closet.

 

[joe]

I have a project that would feed a continual stream of data at a rate of
approximately 17 GB per day at a minimum. My project would require archiving
this data indefinately. I would like to know of the cheapest possible
solution to store 17 GB per day for as long of a time span as possible.

Ok, I'll bite, what's the subject matter ?

 

[Rod Speed]

Ok, I'll bite, what's the subject matter ?

He 'runs' a brothel and wants to keep all footage captured at all
times, in case someone captured ends up in the WhiteHouse etc.

Could have got stinking rich with footage of Slick Willy.

 

[Jeff]

Ok, I'll bite, what's the subject matter ?

I want to archive a live video feed, 24/7. I think I can compress it down to
around 700 MB / hour without that much loss.

-Jeff

 

[Timothy Daniels]

I want to archive a live video feed, 24/7. I think
I can compress it down to around 700 MB / hour
without that much loss.

Got anything good of Michael Jackson? How
'bout Arnold?


*TimDaniels*

 

[Jeff]

Got anything good of Michael Jackson? How
'bout Arnold?

On occasion they might show up, but it would be rare, and it probably
wouldn't be good.

-Jeff

 

[almonious]

I really like that idea! I could just pile on the disks once in a while and
not have to keep a close eye on it all the time. The only thing is, a DVD
only holds about 4.7 GB. I'll be needing around 17 GB per day, so that would
go through about 25 DVDs a week. After 10 years I'd have 13 thousand DVDs to
keep up with. That'd be quite a collection. I think I'll keep looking for
something that can hold more data on each disk/cartridge/unit/whatever.

Thanks though,

-Jeff

Are you not forgetting that new technologies will emerge by then?
next year, you'll have double-layer DVD (X2 storage); in a couple of years,
blue-ray DVD's should appear (X7 storage IIRC), etc etc.

You'll probably change storage system every couple of years; would you keep data
on a 1.44 FDD, just because that was the mainstream mass storage 10 years ago?

How about storing 2 drives with your archives? that way, you'll have reading
hardware to go with your media.

 

[chrisv]

I want to archive a live video feed, 24/7. I think I can compress it down to
around 700 MB / hour without that much loss.

Why digital storage? Why not good old video tape?

 

[Jeff]

Why digital storage? Why not good old video tape?

I dislike analog stuff. I suppose it would work. Still, I'd be changing
video tapes every 6 or 8 hours then, and that could be a real pain.

-Jeff

 

[Rita A.. Berkowitz]

Why digital storage? Why not good old video tape?

You have instant access to any frame at any time without rewinding. Plus,
clarity is much better than what is currently offered on extended record
24-hour VCRs. Easier to access, edit, print, etc.. with digital. No more
changing tapes. You can set the default to "eat it's tail" so you won't
lose current data when you run out of storage space. I can go on, but
nothing beats digital, especially when it's a SCSI array. You know I had to
throw in the SCSI bit for my long time friendly trolls.



Rita

 

[Jeff]

Are you not forgetting that new technologies will emerge by then?
next year, you'll have double-layer DVD (X2 storage); in a couple of years,
blue-ray DVD's should appear (X7 storage IIRC), etc etc.

You'll probably change storage system every couple of years; would you keep data
on a 1.44 FDD, just because that was the mainstream mass storage 10 years ago?

How about storing 2 drives with your archives? that way, you'll have reading
hardware to go with your media.

That's true. I guess I really hadn't thought much about that. You're right,
I guess I could go with the cheapest solution this year, and in a couple of
years change over to what would later be a lower cost per gigabyte. By the
time it would be necessary to replace the old media before it falls apart, a
storage solution would likely exist that could store many of the old discs
or cartridges onto one new media unit at a lower cost.

So that steers me toward going for the cheapest solution rather than the
longest lasting solution. That way a few years from now I can copy the old
data over to the newer cheapest solution.

Any suggestions on the cheapest way of storing tons of data?


Thanks,
Jeff