Tape Backups are NEVER Reliable - EVER

  • Thread starter Thread starter Jolly Student
  • Start date Start date
J

Jolly Student

Okay Folks:



Here is one for all of you who thought that people could not get any dumber.



Yes, I am cross posting here but the recommendations for such are only in
the case where the subject matter concerns a bunch of groups. I think this
qualifies as such.



I work for a mid-sized company (600 employees) whose "Technology Director"
has openly said that "Tape backups are not reliable". This director had a
"consultant" come in to back up his assertion, a consultant who asked to
check his email via his "AOL" account (indeed, his email address is
something like (e-mail address removed)).



Enough jokes aside - its going to get serious and this group seems to be
spreading the rumor that "Tape Backups are Always Unreliable".



So we now we have a huge, Raid 5 server that has a pretty decent amount of
capacity and are using a company's software to that backups are quick and
slick. Cool, my life is so much easier. But thats it. . . we do NOT have
an offsite backup, we do NOT have another inhouse SDLT tape backup drive and
the entire compliment of our backup resides ONLY on this single Network
Attached Raid 5 server. Sure, its housed in a closet somewhere, but what if
we had a catastrophic failure, how about a huge fire, or a plane hitting us.



See, this "consultant" has "clients" in Manhattan who have their offices on
the 89th floor, but their Tapeless Backup servers in the basement. Errr, is
it me or do basements and the safes that may be contained therein get buried
under rubble, or are there some group of IT specialists out there who
specialize in nothing but digging out backup servers from the rubble.



As stupid as this question is, I need to basically find credible, reliable
sources of published information that basically say its really, really,
really dumb to not archive stuff onto some type of medium tape or otherwise.
This Raid 5 server that we have at our company is not a bottomless pit, but
the higher ups do not listen to me, only to the "director" who, along with
their "consultant" has them believing that the system we currently have in
place is relable.



Normally I would just shut my face since my life is a lot easier in terms of
backup, hell, set it and forget it is the name of the game. However, I know
full well that if we ever got hit with a major disaster the "director" would
be off on his vacation while the rest of us poor slobs had to restore data
from God knows where. Oh, and if we were to get hit by a brand new,
spanking virus because the "director's" kid came in and did so, well, our
Network Attached Storage pig would also suffer.



In short, I need some type of recommendation, in writing, in some type of
white paper, from some type of credible sources, that SDLT tape backup
drives, at least for the purpose of long term archiving are not "unreliable"
, they are only as "unreliable" as the poor work habits of the person who is
responsible for them.



Oh, and for the record, dear friends of mine swear by SDLT tape drives and
the like, but I cannot bring an IT manager from CitiCorp into this
discussion because, since he is a friend of mine, his opinion is not
"neutral".



Please help



Roger.
 
Okay Folks:






I work for a mid-sized company (600 employees) whose "Technology Director"
Click to expand...

As the old saying goes, if you don't like your boss, just wait (you
fill in here) and he/she will be gone. Seriously,

1. tape backup is extremely reliable if you use good (well cleaned!)
drives and quality media, stored under the right conditions.
2. your backup is only as good as your software. For a company your
size, Retrospect might be a good solution. The point here is that
"backups always work" but "restores" don't always, meaning that the
software is too complex/inadequate to do what you need.
has openly said that "Tape backups are not reliable". This director had a
"consultant" come in to back up his assertion, a consultant who asked to
check his email via his "AOL" account (indeed, his email address is
something like (e-mail address removed)).
Click to expand...

If this director pays a (high priced? with New York) clients, then he
won't listen to anything anyone on this newsgroup has to say. That
would expose him as a fool who wastes company money on useless
consultants.

Enough jokes aside - its going to get serious and this group seems to be
spreading the rumor that "Tape Backups are Always Unreliable".



So we now we have a huge, Raid 5 server that has a pretty decent amount of
capacity and are using a company's software to that backups are quick and
slick. Cool, my life is so much easier. But thats it. . . we do NOT have
an offsite backup, we do NOT have another inhouse SDLT tape backup drive and
the entire compliment of our backup resides ONLY on this single Network
Attached Raid 5 server. Sure, its housed in a closet somewhere, but what if
we had a catastrophic failure, how about a huge fire, or a plane hitting us.
Click to expand...

Never mind airplanes. How about a bad power failure which took days
to restore, or a flood or an earthquake, or a roof collapse. You're
right, though,. Your director needs to learn about "business
continuance" and "disaster recovery."
See, this "consultant" has "clients" in Manhattan who have their offices on
the 89th floor, but their Tapeless Backup servers in the basement. Errr, is
Click to expand...

I'll bet this company's IT directors is best friends with your IT
director. :)
it me or do basements and the safes that may be contained therein get buried
under rubble, or are there some group of IT specialists out there who
specialize in nothing but digging out backup servers from the rubble.
Click to expand...

Frankly, after 9/11, I don't believe it, not for any company in the
financial services industry, for example.
As stupid as this question is, I need to basically find credible, reliable
sources of published information that basically say its really, really,
really dumb to not archive stuff onto some type of medium tape or otherwise.
This Raid 5 server that we have at our company is not a bottomless pit, but
the higher ups do not listen to me, only to the "director" who, along with
their "consultant" has them believing that the system we currently have in
place is relable.
Click to expand...

See my comment above.
Normally I would just shut my face since my life is a lot easier in terms of
backup, hell, set it and forget it is the name of the game. However, I know
full well that if we ever got hit with a major disaster the "director" would
be off on his vacation while the rest of us poor slobs had to restore data
Click to expand...

You should disc cuss this with your immediate boss or supervisor and
to CYA by documenting all your concerns. it may save your ass when
that inevitable disaster occurs and you and your mates can't do the
impossible. Also, see what your boss thinks of the situation.

If you are a lone voice in the wilderness in this company, perhaps
it's time to find another job. Seriously. Next thing you know, this
same consultant will tell the IT director to outsource all the IT jobs
except his to India or China.
 
Jolly said:
Okay Folks:



Here is one for all of you who thought that people could not get any
dumber.



Yes, I am cross posting here but the recommendations for such are only in
the case where the subject matter concerns a bunch of groups. I think
this qualifies as such.



I work for a mid-sized company (600 employees) whose "Technology Director"
has openly said that "Tape backups are not reliable". This director had a
"consultant" come in to back up his assertion, a consultant who asked to
check his email via his "AOL" account (indeed, his email address is
something like (e-mail address removed)).



Enough jokes aside - its going to get serious and this group seems to be
spreading the rumor that "Tape Backups are Always Unreliable".



So we now we have a huge, Raid 5 server that has a pretty decent amount of
capacity and are using a company's software to that backups are quick and
slick. Cool, my life is so much easier. But thats it. . . we do NOT have
an offsite backup, we do NOT have another inhouse SDLT tape backup drive
and the entire compliment of our backup resides ONLY on this single
Network
Attached Raid 5 server. Sure, its housed in a closet somewhere, but what
if we had a catastrophic failure, how about a huge fire, or a plane
hitting us.



See, this "consultant" has "clients" in Manhattan who have their offices
on
the 89th floor, but their Tapeless Backup servers in the basement. Errr,
is it me or do basements and the safes that may be contained therein get
buried under rubble, or are there some group of IT specialists out there
who specialize in nothing but digging out backup servers from the rubble.



As stupid as this question is, I need to basically find credible, reliable
sources of published information that basically say its really, really,
really dumb to not archive stuff onto some type of medium tape or
otherwise. This Raid 5 server that we have at our company is not a
bottomless pit, but the higher ups do not listen to me, only to the
"director" who, along with their "consultant" has them believing that the
system we currently have in place is relable.



Normally I would just shut my face since my life is a lot easier in terms
of
backup, hell, set it and forget it is the name of the game. However, I
know full well that if we ever got hit with a major disaster the
"director" would be off on his vacation while the rest of us poor slobs
had to restore data
from God knows where. Oh, and if we were to get hit by a brand new,
spanking virus because the "director's" kid came in and did so, well, our
Network Attached Storage pig would also suffer.



In short, I need some type of recommendation, in writing, in some type of
white paper, from some type of credible sources, that SDLT tape backup
drives, at least for the purpose of long term archiving are not
"unreliable" , they are only as "unreliable" as the poor work habits of
the person who is responsible for them.



Oh, and for the record, dear friends of mine swear by SDLT tape drives and
the like, but I cannot bring an IT manager from CitiCorp into this
discussion because, since he is a friend of mine, his opinion is not
"neutral".



Please help
Click to expand...

You're fighting a political battle. If you're not a politician you're not
going to win. Best you can do is CYA.

First, write up your concerns in the form of a memo to your immediate
supervisor--don't make any value judgments about individuals, stick to the
technical issues, i.e. right now we're doing this, if this, this, or this
happens then we lose everything, we need to keep a copy of our data
off-site because of this . . .. Give him time to read it and then talk to
him about it--if he's not sympathetic to your concerns then that's pretty
much the end of it unless you want to set something up on your own without
telling anybody, which is courting personal disaster. If he is
sympathetic, then see if he can find out if his supervisor is sympathetic
and so on up as high as there is somebody who'll listen. Once you've found
the highest level in your chain of command that agrees with your concerns,
find out how much they can spend without asking permission from anybody and
then figure out a way to get an offsite backup of at least the most
critical data within that budget, present it to your supervisor and let him
run it up the chain of command for approval.

Another question--do you have a mentor in the company? That would be
someone very high up with whom you are on good terms who won't pull strings
for you but will advise you on how to get things done in that particular
business. If not, you might want to try to find one.

If you're willing to stick your neck _way_ out and if this is a publicly
traded company then buy a few shares of stock and inform whoever you need
to inform that you wish to speak at the next shareholders' meeting, then
say your piece there in front of God and everybody--write it out in simple
declarative sentences composed of words of one syllable or less in terms
that the hypothetical New Guinea headhunter whose tribe had its first
contact with civilization last week could understand. This may backfire on
you though.
 
Thanks to all who have responded with the obvious, common sense approach to
this matter, ERGO, that tape backups, or some other offsite type of backup
system is a necessity.

What I forgot to mention here is that I am dealing with "Educators" who are
really, really dense when it comes to anything having to do with technology.
Essentially, you have one director who feels that tape backups are not
reliable, ever, period and that experience is based upon his personal
feelings or the feelings of others who are just plain too lazy to set up
some type of disaster mechanism off site be it tape, hard drive, magic
F***ING oxide pixie-dust media or what have you.

I know guys and gals, I know. . . just relying on stuff on site no matter
what is just plain dumb. Even for those of you out there who may think that
tapes do fail from time to time (of course they do), you would have to say
that if you regularly check the tapes, the backup devices, perform routine
restorations and all of the other "Duhhhh-basics" you are going to be in
better shape.

My "thing" is wanting to walk off-site with a tape or two in my briefcase,
come home and stick them in my firebox on a rotating basis. If a plan
decided to land on our facilty, or if a nicely placed bolt of lightning hit
the building and used the network cable as a means to send its electron
offspring to to their worst, we would be screwed, the director would go on
break or hide their head in the sand and the grunts would have to be the
ones doing the clean up. Except that, in this case, all of us grunts are in
agreement that we will go so very high up in our chain of command that we
will let them know this was preventable and produce a plethora of
documentation despite the "director's" 'feelings that tapes are never
reliable".

In short, I know that all of you here are credible sources, Jesus, its like
telling you to keep a spare tire in the trunk of your car or to wear your
body armor when you are going into Bagdad, but what I cannot convey enough
to you guys is that I need the shit, the goods, in the form of papers.

Theres got to be some white papers out there, hell, I got calls to MIS
buddies of mine as far as the NSA asking if they have anything along the
lines of "duhhh, you better backup off site" white papers to show this
director.

I can't believe that its come to this. That there are those in our industry
that are so stupid and are permitted to be in charge of people like us.

Anyways, enough rambling (jokes about breaks, drinks, drugs, etc., will be
heartily laughed at, I assure you), but I think that if such a paper or set
of papers could be found, perhaps some other poor slobs out there besides me
will be able to sigh when a restoration is performed and not fry when they
are into their fifteenth cup of coffee while the supervisor is home banging
his mother up the butt.

Thanks for your time,

Rog
 
Okay Folks:

So we now we have a huge, Raid 5 server that has a pretty decent amount of
capacity and are using a company's software to that backups are quick and
slick. Cool, my life is so much easier. But thats it. . . we do NOT have
an offsite backup, we do NOT have another inhouse SDLT tape backup drive and
the entire compliment of our backup resides ONLY on this single Network
Attached Raid 5 server. Sure, its housed in a closet somewhere, but what if
we had a catastrophic failure, how about a huge fire, or a plane hitting us.
Click to expand...

I would worry about the following scenarios:

(1) a virus infection that deletes or replaces all files with a particular
extension (it's happened to us).

(2) a successful cracker attack which is not discovered for several days
(again, a site in our company was taken over and the attack was not
discovered for over a month - a web server no one was managing)

(3) some businesses, especially ones dealing with tax data or that are
in a contractual relationship with the Federal Government (specific
example is federal research grants) have a requirement that they be
able to produce financial data going back 7 years. I don't know what
your requirements are, but would be surprised if at some point you
would have to provide files up to a year back.

(4) disaster recovery, but you've discussed this

I suspect that somewhere there is a "best practices" document for your OS;
I would go looking at security sites since security issues often drive
the use of multi-backup tape systems.

My experience is that people who think tape drives are unreliable have only
used consumer-grade equipment, and I certainly have had the same experience
- with consumer grade equipment. I find the AIT and DLT drives I've used to
be the most reliable hardware I've ever used, and the performance of such
complicated devices as a Qualstar or ADIC changer with such drives has been
flawless, even when they've been abused.

Good luck; I doubt you'll get very far, but I'd definitely prepare a file
so that when the inevitable crash occurs blame gets put on the guilty party,
not on the innocent as so often happens in IT.

Mike Squires
 
Thanks to all who have responded with the obvious, common sense approach to
this matter, ERGO, that tape backups, or some other offsite type of backup
system is a necessity.

What I forgot to mention here is that I am dealing with "Educators" who are
Click to expand...

My favorite "educators" did the following:

(1) tried to save money by installing a single-ended SCSI drive in an HP
Snake (HV differential); result was a fire and a new $1,000 motherboard.

(2) ordered a Sun with 4GB of memory and 2 GB of disk; SunOS would not
install since it defaults to 2X memory for swap.

That's not to say I've never done anything stupid, but...

The strongest argument for tape backup (or some kind of multiple backup
strategy, where there are monthly full backups and weekly/daily incrementals)
is recovering from virus/worm or cracker attacks. In some cases organizations
have had to go back several months to find uncontaminated data. Then there are
the viruses that delete all files with a particular name or extension.

I would look for a "best practices" guide from the OS vendor, or from one of
the reputable security sites. I'm sure CA or Veritas has a lot of horror
stories, but I doubt material from a backup software vendor will help much.

Just document what you've done, and wait for the inevitable crash.

Mike Squires
 
Never design a backup or recovery strategy on the basis of all the worst
case wives' tales that you can dig up. Design a security, backup or
recovery strategy based on cost risk trade-offs.
 
~Okay Folks:
~
~
~
~Here is one for all of you who thought that people could not get any dumber.
~
~
~
~Yes, I am cross posting here but the recommendations for such are only in
~the case where the subject matter concerns a bunch of groups. I think this
~qualifies as such.
~
~
~
~I work for a mid-sized company (600 employees) whose "Technology Director"
~has openly said that "Tape backups are not reliable". This director had a
~"consultant" come in to back up his assertion, a consultant who asked to
~check his email via his "AOL" account (indeed, his email address is
~something like (e-mail address removed)).

I do not see this as off topic. Backup and recovery concern PCs, Hardware and
storage.

The reliability of a backup and restore procedure depends on the what practices
have been implemented, the backup client, the backup hardware and the quality of
the backup media. The biggest problem I have seen is with the backup procedure.
Miserly attitudes towards backup and recovery will cause extreme suffering in
the event of a worst case recovery senario. It is also important to
periodically review your backup and recovery procedure to ensure that the
organization has a written documented procedure and that this procedure can
still meet the needs of the organization.

I worked for a software house that had grown from ~30 people in 1990 to hundreds
in 1995. Somewhere along the way they implemented Palindrome, An 8mm helical
scan tape tape drive that was sole sourced through Exabyte. Palindrome operated
such that the day you turned it on it would take a full backup, from then on it
performed some sort of modified incremental backup using a Tower of Hanoi tape
rotation. In theory and on paper this approach works and minimizes the number
of tapes you should need for the backup but I was never comfortable with the
never take a full backup approach. The product worked OK to restore a file or
three but was a disaster when we tried to restore an entire volume. The
principle problem we had was that the file history database outgrew what could
be accomodated by the Palindrome DOS client. This caused the client to loose
track of which file was on what tape and and the client did not seem to be aware
that it had not been able to check the entire file. The second problem with
this was that the tape drive itself was glacial. I once calculated that it
would take on the order of 60 hours to perform a full backup of our most
critical volumes. I have always suspected there were other bugs in the Tower of
Hanoi tape rotation schema that caused Palindrome to loose track of which files
were on what tape. It seemed very odd to me that the client could find a file
from three weeks ago but could not find the file from two days ago but if I
searched by backup date the file from two days ago was on the list. We were
down for several days while we junked together a partial restore of that volume.

Not long after that charlie foxtrot we dumped Palindrome and migrated to a
mirrored RAID5 that was backed up on DLT.

In the enterprise world you approach the problem not from the perspective of how
much the tapes and drives cost but from the perspective of "How long can we
afford be out of operation? and how much data needs to be restored?" If the
answer is four hours and 600GB of data then you need a backup/restore process
that can restore 150GB/hour and you look for the solution that fits the
requirement.

Data Centers running IBM mainframes use 3490 tape cartridges. When I worked in
such a data center back in the era of the similar 3480 cartridge we performed
dual full backups every Sunday morning and dual incremental backups every night.
The second set of cartridges were shipped offsite the next morning and
cartridges from a 60 day rotation were returned. We never depended on
incremental backups for more than a weeks worth of changes.

I suspect your managers attitude towards tape is the result of a bad experience
with a poorly conceived helical scan backup solution implemented with the
cheapest tapes available which were then used until the magnetic surface had
been worn down to the mylar substrate. Of course these were not reliable. I
have IBM 3480 cartridges that I have loaded and read without problems after
sitting on a shelf for ten years. I am sure that DLT has a similar lifespan.
 
Back
Top