Idiot proof hot swap backup system

[Dale Walker]

I've been given the task of setting up an idiot proof backup system
for a small business server.

My experience of tapes backups/online backups/DVD backups have all
ended up in various forms of disaster (mostly because the staff are
either too lazy, too impatient or too stupid to work out what actually
needed to be done). So, if I can set up a backup system that requires
the least amount of effort from them the better.

I had the idea of getting a SATA drive bay/caddy system and two or
three extra drive caddies and basically get one of the staff to swap
the drives out at the end of the day.

Now as these particular staff members can only just about recognise
the difference between a mouse and a keyboard, this has got to be as
idiot proof as possible.

Fumbling around in My Computer/Disk Management is not an option. (no
way am I letting them anywhere near there). The little disconnect
system tray thing will do but are there any drive bay/caddy thingumies
that have an auto disconnect/connect button on them or something that
can prevent them from being removed until they're turned off via
software?

These drives won't actually be in use apart from a set times of the
day so will the odd 'accidental' removal cause damage to the system or
drives (I really don't trust these people)?

Or am I better off getting a USB or Firewire setup if I need a bit
more protection from accidents?

Second part of the question. Assuming I can set up the above with
relative safety, What would be the best way to backup the server?

I've got the data and system files on separate partitions so I was
thinking of doing a diskimage of the system files weekly and a daily
MS backup of the data files. Total files needed to be backed up is
less than 30GB so I was thinking of two or three 300GB disks rotated
daily would give plenty of room for at least a couple of weeks worth
of backups.

Would love some comment on the pros and cons of doing things this way.
Any other ideas would be gratefully received.

 

[Rod Speed]

I've been given the task of setting up an idiot
proof backup system for a small business server.

My experience of tapes backups/online backups/DVD backups
have all ended up in various forms of disaster (mostly because
the staff are either too lazy, too impatient or too stupid to work
out what actually needed to be done). So, if I can set up a backup
system that requires the least amount of effort from them the better.

I had the idea of getting a SATA drive bay/caddy system
and two or three extra drive caddies and basically get one
of the staff to swap the drives out at the end of the day.

I wouldnt use caddys in that situation myself, too easy to
drop them and too expensive when they do get dropped.

Now as these particular staff members can only just about
recognise the difference between a mouse and a keyboard,
this has got to be as idiot proof as possible.

I'd personally have a low powered server in a fire
safe and completely automate the backup from there.

Fumbling around in My Computer/Disk Management is not an
option. (no way am I letting them anywhere near there). The
little disconnect system tray thing will do but are there any drive
bay/caddy thingumies that have an auto disconnect/connect
button on them or something that can prevent them from
being removed until they're turned off via software?

I think it would be better to avoid removal completely.

These drives won't actually be in use apart from a set times
of the day so will the odd 'accidental' removal cause damage
to the system or drives (I really don't trust these people)?

Should be ok with SATA.

Or am I better off getting a USB or Firewire setup
if I need a bit more protection from accidents?

They are rather more prone to unreliability and the
clowns may well just carry on regardless when it fails
and you only discover that when the backup is needed.

Second part of the question. Assuming I can set up the above with
relative safety, What would be the best way to backup the server?

I've got the data and system files on separate partitions
so I was thinking of doing a diskimage of the system files
weekly and a daily MS backup of the data files.

Presumably you mean an incremental backup for that daily.

Modern imagers like True Image can do incremental backups now.

So you could do a full backup weekly and an incremental daily.

Total files needed to be backed up is less than 30GB so I was
thinking of two or three 300GB disks rotated daily would give
plenty of room for at least a couple of weeks worth of backups.

Sure, but I'd avoid removable drives in that situaiton myself.

Would love some comment on the pros and cons of doing things this way.

Any removable system is going to be stuffed up by clowns who dont care.

Any other ideas would be gratefully received.

I think its best to have it permanently connected, in a fire safe.

 

[Dale Walker]

I wouldnt use caddys in that situation myself, too easy to
drop them and too expensive when they do get dropped.

I'd personally have a low powered server in a fire
safe and completely automate the backup from there.

To be honest, alongside the other reasons for going the removable
route, I'd rather take the risk of wrecking the occasional drive and
spending 200ukp on a new 300GB drive than over 2000ukp on a new server
and fireproof housing. I'll just have to make sure I re-verify all
data on the disk at very regular intervals.

I think it would be better to avoid removal completely.

I appreciate your comments but the priority here is to get the data
offsite and without the removal aspect, they have no 'backup' should
there be a fire or if the place gets robbed (OK, half of that's
covered with your fireproof housing thnigumy). We did have an on
online solution but unfortunately the ADSL specs in the area can't
cope with the upload and we had to ration what we backed up.

Should be ok with SATA.
Cool.


They are rather more prone to unreliability and the
clowns may well just carry on regardless when it fails
and you only discover that when the backup is needed.
Indeed.



Presumably you mean an incremental backup for that daily.
Yes.

Modern imagers like True Image can do incremental backups now.

Are they MS Exchange/SQL aware? Will I still need to use some other
backup system to clear the logs?

Any removable system is going to be stuffed up by clowns who dont care.


I think its best to have it permanently connected, in a fire safe.

Once we have ADSL2 in the area (3-6 months away), I think I'll chuck a
heap of extra diskspace on my server at home and get them to backup to
that (and charge them for the privilege) but until then I need a cheap
solution. It was difficult enough explaining to them why they needed
to spend money on a server if no-one was going to 'use-it' (meaning
sit in front of it). I think yet another 'redundant' PC will blow
their tops.

Thanks for your input.

 

[Mike Redrobe]

I appreciate your comments but the priority here is to get the data
offsite and without the removal aspect, they have no 'backup' should
there be a fire or if the place gets robbed

Theft is a real big problem, and offsite backups are the last thing
people consider.

Alot of backups also get used more frequently to restore some files
accidentally deleted, or an old version of the file, instead of just
disaster recovery.

E.g. it takes a month for them to realise they needed them.

 

[Peter]

I would add additional hard drive to your server and run nightly backups to
that backup disk.
In addition I would attach an external HD to your server (USB or FireWire)
and copy backup disk to external HD also on a daily basis.
External HD would then be taken to offsite. You should use more than one
external HD and rotate them.

You can substitute external HD's with a fast network connection to an
offsite server.

 

[Rod Speed]

To be honest, alongside the other reasons for going the
removable route, I'd rather take the risk of wrecking the
occasional drive and spending 200ukp on a new 300GB drive
than over 2000ukp on a new server and fireproof housing.

It doesnt cost anything like that.

I'll just have to make sure I re-verify all
data on the disk at very regular intervals.

Wont help with a dropped drive.

I appreciate your comments but the priority here is to get
the data offsite and without the removal aspect, they have
no 'backup' should there be a fire or if the place gets robbed
(OK, half of that's covered with your fireproof housing thnigumy).

All of it is actually. It isnt hard to hide that safe where some
burglar wont find it. Even pathetic druggys minimise the time
they are in the place in case the alarm system is monitored etc.

We did have an on online solution but unfortunately the ADSL specs in the
area can't cope with the upload and we had to ration what we backed up.

Even if the backup is only of changed data ?

Are they MS Exchange/SQL aware?

Imagers dont need to be.

Will I still need to use some other backup system to clear the logs?

I'd personally have that system produce just changed data that is uploaded
over the adsl system. I wouldnt bother with traditional backup at all.

Once we have ADSL2 in the area (3-6 months away),
I think I'll chuck a heap of extra diskspace on my
server at home and get them to backup to that

Yeah, only way to go if its feasible.

(and charge them for the privilege) but until then I need a cheap solution.

Yeah, that does make sense for a temporary solution.

It was difficult enough explaining to them why they needed
to spend money on a server if no-one was going to 'use-it' (meaning
sit in front of it). I think yet another 'redundant' PC will blow their tops.

Yes, and a repeating charge often slips under
the radar better than a substantial up front charge.

Thanks for your input.

No problem.

 

[Rod Speed]

Dale Walker wrote

Theft is a real big problem,

Not really. It isnt hard to hide something like that so a burglar wont find it.

and offsite backups are the last thing people consider.

Alot of backups also get used more frequently to restore some files
accidentally deleted, or an old version of the file, instead of just disaster
recovery.

E.g. it takes a month for them to realise they needed them.

Doesnt sound like his situation. Looks more like a real database.

Makes more sense with those to have a separate transaction
list that is usually quite viable to upload using bog standard adsl.

 

[Dale Walker]

It doesnt cost anything like that.

Even better! hadn't looked at the prices for a couple of months but
yes, 300GB drives are almost half that these days.

Wont help with a dropped drive.

That's why you have several and rotate. A dropped drive might ruin the
previous nights backup but you'll soon know about it next time you
comes to use it.

Well, we all know that the 1,000,000:1 shot that the server will blow
up on the day you drop the drive is usually closer to 10:1 when you
factor in Sods law but I think it's a pretty safe way to go about
things.

Even if the backup is only of changed data ?

Depends on how good the backup program is. If it can deal with backing
up changes within databases (like MS Exchange/SQL server/etc then
maybe it's OK. Without it a 4GB MS Exchange database alone is going to
take all night at the current 256Mb upload speed (taking other data
transfers/contention issues, etc into consideration.

Imagers dont need to be.

But MS Exchange uses Exchange aware backup programs to clear
transaction logs and stuff or it ends up clogging the hard drive.

I'd personally have that system produce just changed data that is uploaded
over the adsl system. I wouldnt bother with traditional backup at all.

Wouldn't work with MS Exchange. All of the cheaper online backup apps
can't deal with open database files. If you close them first then
you've still got a set of files with a few truly huge files which have
of course been changed throughout. I haven't come across any
Exchange/database aware online backup apps. If I could find one that
backed up the changes within databases then yes, an incremental
version might work, without that capability you'll always be uploading
a huge chunk of data.

 

[Dale Walker]

I would add additional hard drive to your server and run nightly backups to
that backup disk.

The server is already backed up to another PC nightly.

It's a great, simple, no-brainer solution to guard against situations
where the server died but doesn't really help in the case of fire,
natural disasters or theft as any of those situations could cause
exactly the same thing to happen to the other PC.

In addition I would attach an external HD to your server (USB or FireWire)
and copy backup disk to external HD also on a daily basis.
External HD would then be taken to offsite. You should use more than one
external HD and rotate them.

The drive bay hot swap caddy system is basically the same as a an
external hard disk but there's a 5.25" drive bay used up on the server
to house the drive instead of having to plug them in using wires
(which inevitably mean diving under tables to plug them in, being
careful not to snag them with anything, etc.)

An example can be seen here
http://www.yoursite.com/pc-183640-2105-promise-superswap-1100-beige.aspx

You can substitute external HD's with a fast network connection to an
offsite server.

Aye, but our offsite network connection is anything but fast.

 

[Rod Speed]

Even better! hadn't looked at the prices for a couple of months
but yes, 300GB drives are almost half that these days.

And its nothing like 2K for the alternative either.

That's why you have several and rotate.

That assumes the clowns actually do that rotation.

A dropped drive might ruin the previous nights backup but
you'll soon know about it next time you comes to use it.

Not necessarily.

Well, we all know that the 1,000,000:1 shot that the server will blow
up on the day you drop the drive is usually closer to 10:1 when you
factor in Sods law but I think it's a pretty safe way to go about things.

It isnt with that class of user.

Depends on how good the backup program is. If it can deal with backing
up changes within databases (like MS Exchange/SQL server/etc then
maybe it's OK. Without it a 4GB MS Exchange database alone is going to
take all night at the current 256Mb upload speed (taking other data
transfers/contention issues, etc into consideration.

Sure, I obviously meant just moving the changed data, not the whole lot.

But MS Exchange uses Exchange aware backup programs to clear
transaction logs and stuff or it ends up clogging the hard drive.

Wouldn't work with MS Exchange. All of the cheaper
online backup apps can't deal with open database files.

Sure, I wasnt talking about using a backup app, get the database
to produce the transaction data that is what needs to be backed up.

If you close them first then you've still got a set of files with a few truly
huge files which have of course been changed throughout. I haven't
come across any Exchange/database aware online backup apps.
If I could find one that backed up the changes within databases
then yes, an incremental version might work, without that
capability you'll always be uploading a huge chunk of data.

You're looking at the problem the wrong way, it isnt the backup app
that matters, its the database system itself that should be doing it.

 

[Arno Wagner]

I've been given the task of setting up an idiot proof backup system
for a small business server.

My experience of tapes backups/online backups/DVD backups have all
ended up in various forms of disaster (mostly because the staff are
either too lazy, too impatient or too stupid to work out what actually
needed to be done). So, if I can set up a backup system that requires
the least amount of effort from them the better.

I had the idea of getting a SATA drive bay/caddy system and two or
three extra drive caddies and basically get one of the staff to swap
the drives out at the end of the day.

Now as these particular staff members can only just about recognise
the difference between a mouse and a keyboard, this has got to be as
idiot proof as possible.

Fumbling around in My Computer/Disk Management is not an option. (no
way am I letting them anywhere near there). The little disconnect
system tray thing will do but are there any drive bay/caddy thingumies
that have an auto disconnect/connect button on them or something that
can prevent them from being removed until they're turned off via
software?

These drives won't actually be in use apart from a set times of the
day so will the odd 'accidental' removal cause damage to the system or
drives (I really don't trust these people)?

Or am I better off getting a USB or Firewire setup if I need a bit
more protection from accidents?

Second part of the question. Assuming I can set up the above with
relative safety, What would be the best way to backup the server?

I've got the data and system files on separate partitions so I was
thinking of doing a diskimage of the system files weekly and a daily
MS backup of the data files. Total files needed to be backed up is
less than 30GB so I was thinking of two or three 300GB disks rotated
daily would give plenty of room for at least a couple of weeks worth
of backups.

Would love some comment on the pros and cons of doing things this way.
Any other ideas would be gratefully received.

It may be a bit extreme, but I think this cannot be done except with
at least one person that can at least reliably follow instructions
and is careful with the hardware.

That said, a caddy is probably far mor risky than an USB enclosure,
since it requires more mechanical skill and possibley more careful
handling. I also do not trust SATA hotswap much at the moment. For
USB it works in my experience.

As for the backup process itself, I would advise you to completely
automatise it with scripting: During a predefined time window
the computer looks whether the backup drive is connected. If it is,
it informs the user in a visible way (e.g. pop-up) that it is now
doing a backup. When it is finished it again informs the user that
it is finished and the drive may be removed. I have no problems
imaging doing something like that with Linux or any other unix-like
OS. It should be possible with Windows also. (E.g. drive detection:
See whether a particular "ID" file is in a particular place.)

When a user pulls the plug during backup the backup is obvious invalid
and the filesystem on the pulled disk may be damaged. In Linux I would
get around this with a journalling filesystem, preferrably ext3. I made
very good experiences with it running my laptop until battery empty.
The backup would still be shot, nothing to do against that. An
alternative would be splitting the drive in two (or more) partitions:
A small one with ID information and maybe shedule ("This drive to use
Monday and Thursday, refuse and complain on other days") and one or
more large ones with the data. The filesystem on the large one(s)
can be recreated for every backup, avoiding filesystem corruption
propagation.

I would also add a disk validation, e.g. a complete surface read
every 14 days.

Arno

 

[Dale Walker]

You're looking at the problem the wrong way, it isnt the backup app
that matters, its the database system itself that should be doing it.

With MS Exchange, its MS usually MS Backup that clears out the
transaction logs. Other aware backup programs can do it too but so far
as I'm aware, there's no native non-command prompt way to do this
without somehow making a backup first.

 

[Iago]

I've been given the task of setting up an idiot proof backup system
for a small business server.

My experience of tapes backups/online backups/DVD backups have all
ended up in various forms of disaster (mostly because the staff are
either too lazy, too impatient or too stupid to work out what actually
needed to be done). So, if I can set up a backup system that requires
the least amount of effort from them the better.

Perhaps you need unattended, continuous backup service to an outside
vault? Not sure where you are, but there are many SPs to consider.

IMHO you are overengineering this. If those people are too stupid
(your wording) to be trusted moving DVDs how can they become suddenly
smart with USB drives or hot plug drives?

 

[Joe Rom King]

Dale

Take a look at Relative Rev Backup http://www.datamills.com .

This software is designed for unattended operation. To begin with, it
takes file-level incremental forever (including weeklies and
monthlies), but at the same it will restore any file or folder as if it
is done from a full backup. Therefore, if your data is 30GB, you can
expect to have 6 months of history on a 40GB disk (not including
Exchange). Also the backup time will be in the minutes range.

Then, it also manages the number hourly, daily, weekly, and monthly you
want to retain. So you do not have to worry about deleting the backups
once the disk gets full.

It is recommended to have two or even three backup disk (USB is
recommended, but make sure that they are stopped prior to swapping
because 2000 will not always flush the data. See another idea at the
end of this post) and to rotate them daily. Relative Rev Backup will
make sure to update the backup disk with the changes relative to the
last registered backup for the online backup disk.

If while swapping the disk the drive letter changes, Relative Rev
Backup will search for the new drive letter.

If one disk backup disk fails, it has a wizard to import selected
backup packs form the remaining disk, so backup history can be once
again backed up to more than a single disk.

It also has a validation feature that will test retained backup
generation for their fidelity.

As for Exchange, you may use NTBackup to prepare a backup file in a
folder that Relative Rev Backup backs up. If your Exchange is 4GB big,
and you set Relative Rev Backup to keep say 4 daily, 4 weeklies and 4
monthlies you will need 64GB just for Exchange, and together with the
50GB for the rest of the system, a 120GB disk will be good to hold six
months of backup history.

Another idea regarding the backup disk, is to connect it to a networked
XP machine. XP has a feature that allows 'surprise removal' of portable
storage. That way you do not need to worry about training the swappers,
to make sure they stop the USB device before unplugging the external
drive.

If this makes you wonder about network load (4GB Exchange, plus 1%
changes of 30GB = 7GB). With 100Mbps network that is less than 10
minutes daily.

Joe Rom King

 

[Neil Maxwell]

All of it is actually. It isnt hard to hide that safe where some
burglar wont find it. Even pathetic druggys minimise the time
they are in the place in case the alarm system is monitored etc.

There are generally 2 options for keeping a safe from walking off -
hide it or secure it.

Small safes are easy enough to hide, but if they do get found, they
can just be picked up and carried off. Even a big 1000 lb safe can be
wheeled out by 2 big guys with a dolly in a matter of minutes. Also,
safes in general are like a big flag that says "Valuables Here!", so
if they did carry it off, they'd likely be quite disappointed if it
just had a network hard drive in it.

Most decent safes can be bolted to the floor, so lag-bolting it to the
concrete in the garage will keep most burglars (kids and junkies) from
making off with it, because it takes too much time, as Rod said.

Probably the best bet is to hide it in the garage behind a bunch of
clutter, with bolt-down as insurance. It limits access a bit, but you
really shouldn't need to get into it all that much if it's dedicated
to your 2nd level backup gear. I rarely have to physically access
mine.

 

[Rod Speed]

There are generally 2 options for keeping
a safe from walking off - hide it or secure it.

No reason why you cant do both.

Small safes are easy enough to hide, but if they do
get found, they can just be picked up and carried off.

Only with the worst small safes. Thats easily fixed.

Even a big 1000 lb safe can be wheeled out
by 2 big guys with a dolly in a matter of minutes.

Thats easily fixed too.

Also, safes in general are like a big flag that says
"Valuables Here!", so if they did carry it off, they'd likely
be quite disappointed if it just had a network hard drive in it.

Their problem.

Most decent safes can be bolted to the floor,

So your original about picked up and carried off is just plain wrong.
Anyone with a clue uses safes that can be bolted to the floor.

so lag-bolting it to the concrete in the garage will keep
most burglars (kids and junkies) from making off with
it, because it takes too much time, as Rod said.

Or those who loot the normal business operations.

Different if its a bank or they keep most
valuables in the safe overnight like a jeweller etc.

 

[Folkert Rienstra]

It's a downright miracle application. And such an independent review too.

 

[Dale Walker]

It's a downright miracle application. And such an independent review too.

Who cares.

Read between the lines and it answers many questions that aren't often
asked.

Not saying I'll get the app (or comment on it's quality or
appropriateness) but at least they've identified certain areas other
people might overlook or ignore.

 

[Joe Rom King]

As my signature indicates, the post is not portrayed to be objective.

If you want an objective view, read this testimonial, may be it will
somewhat reduce your skepticism.

"Having been let down by our old tape-based backup system once to many
times I searched the internet high and low for an alternative solution.
I knew that the logical route would be a disk-to-disk backup between
the server and a remote machine, but I was concerned that transferring
so much data across the network would cause a significant drop in
performance.

Over the period of several months I downloaded and tested dozens and
dozens of different backup systems, but none of them really did what I
wanted; store multiple versions of the data from days, weeks & months
ago without needing to multiply the disk space (and cost) it required!
I was almost ready to give up when I stumbled across DataMills Relative
Rev Backup.

At last I had found an application that would store incremental backups
as if they were full backups... what they are in fact is virtual
backups! Genius!

I now backup all my data in less than 6 minutes, and can restore any
version of any file from an archive of backups that goes back months.
It is so easy and so reliable I can forget about it and get on with
more important things. Thanks DataMills!" -- David Hasell, Croydon, UK


Joe Rom King

 

[Neil Maxwell]

So your original about picked up and carried off is just plain wrong.
Anyone with a clue uses safes that can be bolted to the floor.

You might be surprised how many people don't bother bolting their
safes down. It's a lot like backing up your data - even people who
know they should be doing it don't.