Secure back up options?

[AN O'Nymous]

I've got some large files I would like to back up, approximately 50GB
every month. The data is important but not critical. It took a lot of
time to compute them. For example, I can live with a few MB of
corrupted data - I just need to re-run the computations again.

What is the best way for me to backup this data? Ideally, I'd like to
keep the data indefinitely and take it with me once I graduate.

I see two feasible options:
1) Buy a 300GB "storage" HDD and append/replace the stuff there on a
periodic basis. However, if the HDD breaks down, I lose everything. The
HDD option is also about 3.5 times more expensive on a cost-per-GB
basis than DVDs (although the HDD option is "rewritable")

2) Use winrar to make 4.7GB split archives, and burn them onto Verbatim
DVD+Rs (I've read that DVD+R's are more durable than DVD-Rs) with the
max (10%) recovery record.The problem with this technique is that
AFAIK, the entire split archive is lost if one 4.7GB "chunk" is lost.
Winrar can't just skip the data lost on a corrupted disc. So while the
odds of failure of one disc is low, the odds of failure of the entire
archive is compounded by N, where N is the number of discs. Of course,
this method is rather troublesome as well.

A RAID array is not a preferred option, as I would like to store the
backups in a physically separate location (in any case copying
everything to another HDD is sort of like a RAID-1 configuration).

 

[kony]

I've got some large files I would like to back up, approximately 50GB
every month. The data is important but not critical. It took a lot of
time to compute them. For example, I can live with a few MB of
corrupted data - I just need to re-run the computations again.

What is the best way for me to backup this data? Ideally, I'd like to
keep the data indefinitely and take it with me once I graduate.

I see two feasible options:
1) Buy a 300GB "storage" HDD and append/replace the stuff there on a
periodic basis. However, if the HDD breaks down, I lose everything. The
HDD option is also about 3.5 times more expensive on a cost-per-GB
basis than DVDs (although the HDD option is "rewritable")

One problem with the DVDs is at 4.7GB per, you're making
over 10 DVDs every month. I can't place a value on your
time but that could end up seeming quite a burden, maybe.
Ideally since you want this long-term storage, you should
also make duplicate DVDs using a different brand in case you
came upon a bad lot of discs, or one or the other wasn't
holding up well long term. This also means you have to
periodically re-check the discs so you know if/when they
start going bad.

You aren't likely to only have "a few MB of corrupted data",
more likely a far larger amount IF any. Whole DVDs or even
whole batches would be prone, or the entire HDD in the
shorter term opposed to random sectors failing over the
course of several years.

2) Use winrar to make 4.7GB split archives,

Does this data compress well? If so, use winrar (or
whatever lends itself to the task, best compression).

and burn them onto Verbatim
DVD+Rs (I've read that DVD+R's are more durable than DVD-Rs) with the
max (10%) recovery record.The problem with this technique is that
AFAIK, the entire split archive is lost if one 4.7GB "chunk" is lost.
Winrar can't just skip the data lost on a corrupted disc. So while the
odds of failure of one disc is low, the odds of failure of the entire
archive is compounded by N, where N is the number of discs. Of course,
this method is rather troublesome as well.

Yes that could be a problem. We don't know how much you can
sub-divide the data, is it always ~ 50GB chunks? You could
always save those source winrar files to HDD and then burn
two sets of DVDs using different DVD brands then if either
fails you have the 2nd. At some point you will have to
place a value on the data and on your time to better
evaluate which makes the most sense, and how long
"long-term" is, that you want to retain this data.

A RAID array is not a preferred option, as I would like to store the
backups in a physically separate location (in any case copying
everything to another HDD is sort of like a RAID-1 configuration).

Copying everything to another drive is not like a RAID1,
because it does nothing for the data inbetween copies. If
you can accept that loss of incrementally accumulated data,
so be it, but we don't know... at 50GB per month, you'd have
to be copying data quite often to have only the "few MB"
loss potential initially mentioned.

50GB is a non-trivial quantity, you might consider both the
RAID1 array for local storage AND then less frequent
duplicate DVD backup. This saves some of your time and
reduces the frequency of backups while still having
redundancy of current data. If you wanted a HDD copy in
another location, it wouldn't necessarily replace the RAID1,
rather be a suppliment to it. Not knowing the total value
of the data it's hard to speculate but even one single HDD
in an alternate location might be a little lax for long-term
data retention.... you won't know the data went bad until it
does and the more often you have to check it, the more of a
burden it becomes to recheck ALL of it as the store size
grows.

 

[Bennett Price]

Consider a DAT DDS-3 12-uncompressed 24-compressed GB tape drive.
Tapes are about $25-$30 each, drive $500-550. DAT DDS-4 20/40GB is
often available for about the same price. Take a look at, e.g.,

http://www.cdw.com/shop/search/results.aspx?key=tape+drive+dds&platform=all&x=0&y=0

As you're on a campus, perhaps the computing center can do the backups
for you for a small monthly fee; when you graduate, you can then write
the data to duplicate DVD's or similar.

Winrar can be made more robust; take a look at

http://www.par2.net/

 

[Bob]

1) Buy a 300GB "storage" HDD and append/replace the stuff there on a
periodic basis. However, if the HDD breaks down, I lose everything. The
HDD option is also about 3.5 times more expensive on a cost-per-GB
basis than DVDs (although the HDD option is "rewritable")

You are not making a valid comparison. Western Digital has its 250 GB
drive for $100 at Directron.

http://www.directron.com/wd2500sb.html

You can get Taiyo Yuden 8x -R discs for 37 cents which includes
shipping.

http://tinyurl.com/879vk

There is no advantage technical advantage between +R and -R discs.
However, -R discs are cheaper.

Burn 8x at 8x, 16x at 12x. I do not think the burn time warrants the
extra cost of 16x discs.

2) Use winrar to make 4.7GB split archives, and burn them onto Verbatim
DVD+Rs (I've read that DVD+R's are more durable than DVD-Rs)

If you want longevity use Taiyo Yuden. The last time I used Verbatim I
had 12 coasters out of the first 90 burns.

A RAID array is not a preferred option, as I would like to store the
backups in a physically separate location (in any case copying
everything to another HDD is sort of like a RAID-1 configuration).

Get Kingwin KPF-23 3-fan removable drive bays, 80-wire IDE ATA 133
flat ribbon cable and Acronis True Image to make clone drives.

 

[AN O'Nymous]

kony wrote:

Thanks Kony. Very informed advice, as usual

One problem with the DVDs is at 4.7GB per, you're making
over 10 DVDs every month. I can't place a value on your
time but that could end up seeming quite a burden, maybe.

Yes, I agree it is a real pain. Not only that, feeding that much data
through .zip & .rar compression algorithms is even more painstaking (I
use both to get as much compression as possible).

Ideally since you want this long-term storage, you should
also make duplicate DVDs using a different brand in case you
came upon a bad lot of discs, or one or the other wasn't
holding up well long term. This also means you have to
periodically re-check the discs so you know if/when they
start going bad.

Well, I do these backups once every few months (a lot of the 50GB/month
gets overwritten with more recent data), so in a sense there is some
redundancy there.

You aren't likely to only have "a few MB of corrupted data",
more likely a far larger amount IF any. Whole DVDs or even
whole batches would be prone, or the entire HDD in the
shorter term opposed to random sectors failing over the
course of several years.

I agree. If one disc is corrupted, the entire archive will probably be
irrecoverable (for practical purposes anyway).

Does this data compress well? If so, use winrar (or
whatever lends itself to the task, best compression).

Yes. I can usually get 50GB down to about 30GB or so.

Yes that could be a problem. We don't know how much you can
sub-divide the data, is it always ~ 50GB chunks?

No, it is just lots of files & subdirectories, tens of MB to hundreds
of MB in size.

You could
always save those source winrar files to HDD and then burn
two sets of DVDs using different DVD brands then if either
fails you have the 2nd. At some point you will have to
place a value on the data and on your time to better
evaluate which makes the most sense, and how long
"long-term" is, that you want to retain this data.

Yes, that would be very secure. However, as you rightly point out, also
quite troublesome. As I do perform backups once every few months, the
loss of one archive means that at most I will lose a few months worth
of computations.

Copying everything to another drive is not like a RAID1,
because it does nothing for the data inbetween copies. If
you can accept that loss of incrementally accumulated data,
so be it, but we don't know... at 50GB per month, you'd have
to be copying data quite often to have only the "few MB"
loss potential initially mentioned.

Yes, I'm OK with incremental losses if they happen. Annoying, but I can
always re-compute the results. The "few MB" loss was up to what I could
probably just sniff at. It would be equivalent to a corrupted
simulation run...probably a day to a few worth of computations.

50GB is a non-trivial quantity, you might consider both the
RAID1 array for local storage AND then less frequent
duplicate DVD backup.

Yes, I'm deciding to go for both. One benefit of a new HDD which I
forgot to mention was, of course, its random access. With a new HDD I
can probably live with backing up (say) 200GB to a DVD annually.

This saves some of your time and
reduces the frequency of backups while still having
redundancy of current data.

Yes, well spotted. Getting a new HDD in RAID1 saves time making more
frequent DVD backups.

If you wanted a HDD copy in
another location, it wouldn't necessarily replace the RAID1,
rather be a suppliment to it. Not knowing the total value
of the data it's hard to speculate but even one single HDD
in an alternate location might be a little lax for long-term
data retention.... you won't know the data went bad until it
does and the more often you have to check it, the more of a
burden it becomes to recheck ALL of it as the store size
grows.

The data can be recomputed...it just costs time.

Incidentally, do you know of any compression tools that split archives
into _independent_ 4.7GB chunks? Where if one disc fails the rest of
the archive is still retrievable?

 

[AN O'Nymous]

You are not making a valid comparison. Western Digital has its 250 GB
drive for $100 at Directron.

http://www.directron.com/wd2500sb.html

You can get Taiyo Yuden 8x -R discs for 37 cents which includes
shipping.

http://tinyurl.com/879vk

Well, I'm in the UK and the DVDs I was going to buy were the £17.59
for 50 Verbatim ones at the bottom of the page here:
http://svp.co.uk/products-list.php?bid=0&cid=103

The HDD was a 300GB £76.37 Seagate HDD.

So the cost/GB ratios were (17.59/(50*4.5)) and (76.37/300), about
3-3.5.

There is no advantage technical advantage between +R and -R discs.
However, -R discs are cheaper.
http://www.cdfreaks.com/article/113

Burn 8x at 8x, 16x at 12x. I do not think the burn time warrants the
extra cost of 16x discs.

Agreed. I would happily buy 4x discs if they were still easily found,
and cheaper than 8x. I multitask whilst burning anyway

If you want longevity use Taiyo Yuden. The last time I used Verbatim I
had 12 coasters out of the first 90 burns.

Hmmm...AFAIK there is quite a lot of disc type (+/- R and brand)
preference depending on your drive. I have a LiteOn SOHW-812S (latest
firmware) and have endless problems with the "average" Ritek media.
I've burnt about 100 Verbatim 8x DVD+Rs, not a single coaster. About
10-15 coasters with 100 Fujifilm 8x DVD+Rs. My drive is hopeless with
DVD-Rs, and many other users will corroborate that.

I have read that Pioneer drives are the best for data integrity and
will be buying one for my next system.

Get Kingwin KPF-23 3-fan removable drive bays, 80-wire IDE ATA 133
flat ribbon cable and Acronis True Image to make clone drives.

Thanks, I will look into this.

 

[AN O'Nymous]

Consider a DAT DDS-3 12-uncompressed 24-compressed GB tape drive.
Tapes are about $25-$30 each, drive $500-550. DAT DDS-4 20/40GB is
often available for about the same price. Take a look at, e.g.,

As you're on a campus, perhaps the computing center can do the backups
for you for a small monthly fee; when you graduate, you can then write
the data to duplicate DVD's or similar.

Hmm, interesting idea. I will look into that.

Winrar can be made more robust; take a look at

Thanks.

 

[kony]

Incidentally, do you know of any compression tools that split archives
into _independent_ 4.7GB chunks? Where if one disc fails the rest of
the archive is still retrievable?

I was under the impression Winrar can if you don't do "solid
archives".

 

[CBFalconer]

I've got some large files I would like to back up, approximately 50GB
every month. The data is important but not critical. It took a lot of
time to compute them. For example, I can live with a few MB of
corrupted data - I just need to re-run the computations again.

What is the best way for me to backup this data? Ideally, I'd like to
keep the data indefinitely and take it with me once I graduate.

Look into arj, <http://www.arjsoftware.com>. Free for personal
use, and it has all sorts of backup provisions, including addition
of syndromes to recover from bad or failed storage media, split up
of dumps, readback checks, etc. Learning to use it can take a bit,
but you can encapsulate the appropriate switches in an alias or
batch file.

Don't bother compressing compressed files. The possible gain is
miniscule, it increases the danger of data loss, and takes a lot of
time.

--
"If you want to post a followup via groups.google.com, don't use
the broken "Reply" link at the bottom of the article. Click on
"show options" at the top of the article, then click on the
"Reply" at the bottom of the article headers." - Keith Thompson
More details at: <http://cfaj.freeshell.org/google/>
Also see <http://www.safalra.com/special/googlegroupsreply/>

 

[CBFalconer]

I was under the impression Winrar can if you don't do "solid
archives".

Yep - arj. Just posted a reply recommending it. Especially well
suited for backups.

--
"If you want to post a followup via groups.google.com, don't use
the broken "Reply" link at the bottom of the article. Click on
"show options" at the top of the article, then click on the
"Reply" at the bottom of the article headers." - Keith Thompson
More details at: <http://cfaj.freeshell.org/google/>
Also see <http://www.safalra.com/special/googlegroupsreply/>

 

[jaster]

I've got some large files I would like to back up, approximately 50GB
every month. The data is important but not critical. It took a lot of time
to compute them. For example, I can live with a few MB of corrupted data -
I just need to re-run the computations again.

What is the best way for me to backup this data? Ideally, I'd like to keep
the data indefinitely and take it with me once I graduate.

I see two feasible options:
1) Buy a 300GB "storage" HDD and append/replace the stuff there on a
periodic basis. However, if the HDD breaks down, I lose everything. The
HDD option is also about 3.5 times more expensive on a cost-per-GB basis
than DVDs (although the HDD option is "rewritable")

2) Use winrar to make 4.7GB split archives, and burn them onto Verbatim
DVD+Rs (I've read that DVD+R's are more durable than DVD-Rs) with the max
(10%) recovery record.The problem with this technique is that AFAIK, the
entire split archive is lost if one 4.7GB "chunk" is lost. Winrar can't
just skip the data lost on a corrupted disc. So while the odds of failure
of one disc is low, the odds of failure of the entire archive is
compounded by N, where N is the number of discs. Of course, this method is
rather troublesome as well.

A RAID array is not a preferred option, as I would like to store the
backups in a physically separate location (in any case copying everything
to another HDD is sort of like a RAID-1 configuration).

Others have posted good solutions. Like you I back up 30 gb to DVDs
about every 3-6 months. I don't use winrar or proprietary compression
like winrar so files on my backup DVDs can be read by any operating system
especially ones that may not have that backup software utility. A lesson
learned from trying to restore Win98 backups to Win2k. Also I
alternate between 2 or 3 sets of DVD+/-RWs.

Maybe you could spread the backup process over a couple of days.
Schedule your backups into 2-3 DVD chunks so over 4 days one week you
backup the complete 50gb. Or portion the 50gb into 4 chunks so every
week you backup 1/4 of the 50gb. Or every 3 months make a complete
backup and every other month make incremental backups.

 

[Curious George]

Hmm, interesting idea. I will look into that.

Or for significantly less look into a DLT7000. Tapes are in that
price range or a little less and will hold 35/70gig at 5/10 MB/sec.
The only problem is "new" drives are hard to come by. You can still
get a reliable one nonetheless.

DLT8000 (40/80) is easier to get new but can cost a bit more. Probably
not worth it because your look at 2 tapes regardless.

Many ppl like VXA for budget tape, but the media is generally
expensive. Prices have come down though.

 

[Bob]

Well, I'm in the UK and the DVDs I was going to buy were the =A317.59
for 50 Verbatim ones at the bottom of the page here:
http://svp.co.uk/products-list.php?bid=3D0&cid=3D103

The HDD was a 300GB =A376.37 Seagate HDD.

You are going to have to post in US fonts for us to be able to use
your links.

http://www.cdfreaks.com/article/113

I used to be a frequent poster until it started getting silly. I can
tell you that what you read must be confirmed before you can take it
at face value. There is no substantive evidence that today's +Rs are
any better than -Rs.

Agreed. I would happily buy 4x discs if they were still easily found,
and cheaper than 8x. I multitask whilst burning anyway

The US discount retailers have them for a very attractive price.

http://www.supermediastore.com/taiyo-yuden-4x-dvd-r-media-silver-matte-spindles-100.html

Taiyo Yuden 4X DVD-R Media Silver Shiny in Tape Wrap 100 Pack
Media ID: TYG01
Sale Price: $22.99

Hmmm...AFAIK there is quite a lot of disc type (+/- R and brand)

Not if the disc is TY. Either +R or -R will work.

preference depending on your drive. I have a LiteOn SOHW-812S (latest
firmware) and have endless problems with the "average" Ritek media.
I've burnt about 100 Verbatim 8x DVD+Rs, not a single coaster. About
10-15 coasters with 100 Fujifilm 8x DVD+Rs.

FujiFikm used to brand TYs but not anymore. They now brand CMC which
is pure crap.

My drive is hopeless with
DVD-Rs, and many other users will corroborate that.

I believe you when you use anything other than TY.

Get some TY 8x -Rs and see what happens. Burn at 8x.


--

"Our country's a place of limitless hopes and
possibilities, and nowhere is that spirit more
alive than in the great nation of Texas."
--GW Bush, U.S. President from Texas

"One thing that makes George Bush such a great
president is that he does not govern according
to public opinion polls."
--John Cornyn, U.S. Senator from Texas

 

[Bob]

I agree. If one disc is corrupted, the entire archive will probably be
irrecoverable (for practical purposes anyway).

That's why I rotate 3 identical disks thru my archive system. I have
had to reach back to the third disk on more than one occasion. Without
it, I would have been Frakked with a capital "F".

Yes, that would be very secure. However, as you rightly point out, also
quite troublesome. As I do perform backups once every few months, the
loss of one archive means that at most I will lose a few months worth
of computations.

Besides a weekly clone, I run daily NTBackup using differential mode.
That way I can restore timely information, like email files.


--

"Our country's a place of limitless hopes and
possibilities, and nowhere is that spirit more
alive than in the great nation of Texas."
--GW Bush, U.S. President from Texas

"One thing that makes George Bush such a great
president is that he does not govern according
to public opinion polls."
--John Cornyn, U.S. Senator from Texas