RAIDING different size drives

[David Brown]

David Brown wrote


Or got something wrong.




Doesnt need to be fast with home systems given that failure is so
uncommon.

I've been describing what I see as a good backup system - not just a
home backup system (though that's relevant too). I agree with much of
your comments that some of my suggestions are overkill for many home
backup systems. For most home systems, it's enough to copy regularly to
CD/DVD and, as you say, make duplicates of your photos.

But the OP appears to have data that is particularly important to him,
it's working data rather than archived photos, and it's serious enough
that he's considering raid. That's sounds to me more like professional
or serious amateur work, and requires a bit more out of a backup system.

So I've been discussing general good backup practice - it must of course
be adapted to suit the situation, and with an appropriate balance of
protection, costs, hardware, effort, etc.

I'll leave your "Not relevant to most of the home systems being
discussed" comments unchallenged when I agree that the point is
irrelevant or overkill for home systems, even though that's not all we
are discussing (at least, not all that /I/ am discussing).

That varys with how the system is used. If the user has enough of a
clue to save new work like photos etc to more than one place, the
backup system doesnt really need to worry too much about this sort of
thing.


Its very arguable if home systems really need this, particularly if
the irreplaceable stuff like photos etc are manually written to more
than one place when they enter the system.

For photos and things that typically have multiple archives (such as
burning to CD after copying them onto the disk), as well as things that
don't change often, then I agree. But if you are also storing other
changing data, then it's useful to have snapshots over time. This is
mainly for protection against user error - overwriting or deleting
something you wanted to keep, and only realising it at a later date.

Not relevant to most of the home systems being discussed.


Not relevant to most of the home systems being discussed.

For single computer systems, this applies to the files on that system,
and then it is relevant.

Again, that depends on what the user does with the irreplaceable
stuff.

Yes.


Even that varys with how vulnerable the system is to that sort of
thing.

Agreed.


An important distinction needs to be made between irreplaceable data
and stuff that can be replaced with some effort. If the house has
burnt down, it doesnt really matter if some time has to be spend
getting stuff again when it is replaceable.

Agreed.

It also depends on the circumstances - if your house has burned down,
loss of your digital photos collection is probably a very minor concern.
But if your main PC is an easily stolen laptop, you want to make sure
you have copies that are kept in a different place.

Waffle with home systems being discussed.



Overkill with most home systems.

This I *don't* agree with. You must test that you can recover your
backup data - it's too easy to make a mistake. For a typical home
backup system this is not much more than putting a backup CD in another
computer and checking that you can see and read the files - it's
absolutely worth the effort.

What matters with the home systems is the irreplaceable data.

Agreed.

What I was thinking of here is if you have a backup arrangement that is
tied to specific hardware (such as a tape drive) or specific software -
you have to consider how to get your data back if you need to buy a new
drive and/or new software. An example might be if you have an old
system that uses iOmega ZIP drives for backup. If that machine gets
stolen, you have the challenge of finding a new ZIP drive for your
replacement machine (eBay might help), and then hope that there were no
head alignment issues with the old drive that make the disks unreadable
with another drive.

You can well say that such problems are highly unlikely. But highly
unlikely still means vaguely possible, and it's better to think of such
possibilities and then dismiss them as obviously being no problem for a
given backup solution, than to ignore them totally.

Waste of money with the home systems being discussed.

An rsync to a second PC is an automatic system with no costs - whether
it's a home system or a professional system.

Overkill with the home systems being discussed.

For home users, habit (such as copy photos to drive then CD) is a clear
procedure. It's also easy to set your email/calendar system to remind
you once a week to do your backups.

Without some sort of procedure or habit, backup regimes start out well
and quickly get forgotten.

Gross overkill with the home systems being discussed.

It should be easy enough if you have an automatic backup system - and
without it, how are you going to know that the backups you think have
been running every night actually stopped with a "disk full" error two
months ago?

 

[David Brown]

David Brown wrote







You reinstall the OS on the replacement drive and carry on
regardless. Not a shred of rocket science whatever required.

<snip>

Having no experience with windows software raid, I did not know if there
were any complications involved (such as raid setup information stored
in the system registry). I did not know if there were likely to be
risks involved, such as it being too easy to delete your good mirror by
making it part of a new raid, instead of rebuilding the raid. That's
why I asked.

But I am now happy to hear that recovery from failure in this situation
is straightforward, and therefore this sort of raid setup does offer
useful protection.

There arent really that many anymore. Essentially because write only
media is so slow and of such limited capacity.

Perhaps you missed the point. "Write-only media" is media which cannot
be read back, such as writing to unformatted tapes, or using encrypted
backups without keeping track of the key. That's why testing your
restore procedure is so vital - many people believe they have good
backup procedures, but have problems when they have to do a restore.

<http://www.linuxjournal.com/article/6435>

Sure, but thats rather less common now.


It is indeed.


You can with Win too.

How do you recommend doing a live Windows CD boot? I've seen some
suggestions on the web, but they were mostly hacks of some sort. If you
know of a good way to make a general bootable windows CD with recovery
tools, I'd be very interested. I find Linux live CDs useful for
recovering problematic windows installations, but I'd be happy to expand
my toolkit.

Yes. And you can use a Linux live CD to do that too with Win.

If Linux live CDs can access windows software raid, then that's a good
familiar safety net.

They arent alternatives.


Nope. Most use external hard drives now.


Anyone with a clue uses and external hard drive to reduce the risk of
fire or theft etc affecting both drives.

Agreed - having the second disk external is much better (I think I
mentioned that myself in an earlier post). It also means it can be
disconnected when you are not running backups (reducing wear on the
disk, and risks of any user / OS / malware issues), and you can easily
connected it to another machine.

Copying to CD or DVD is still very popular - it's a familiar solution
for many people. And perhaps the use of internet-based backup is more
common here in Norway than in many other parts of the world, since we
have a high percentage of users with fast (10 Mbit) broadband connections.

Only in the sense that hard drive failure is uncommon.

Yes, hard drive failure is uncommon - especially in comparison to other
causes of data loss such as user error.

 

[Rod Speed]

David Brown wrote

Rod Speed wrote

Having no experience with windows software raid, I did not know if there were any complications involved (such as raid
setup information stored in the system registry).

That can obviously be backed up even if it is the case.

I did not know if there were likely to be risks involved, such as it being too easy to delete your good mirror by
making it part of a new raid, instead of rebuilding the raid. That's why I asked.

You initially proclaimed that there is no point in mirroring with Win.

That was just plain wrong.

But I am now happy to hear that recovery from failure in this situation is straightforward, and therefore this sort of
raid setup does offer useful protection.

Perhaps you missed the point.
Nope.

"Write-only media" is media which cannot be read back, such as writing to unformatted tapes,

Which like I said, doesnt get used much anymore for home system backup.

or using encrypted backups without keeping track of the key.

Thats not write only media, thats a ****ed up backup.

That's why testing your restore procedure is so vital - many people believe they
have good backup procedures, but have problems when they have to do a restore.

Thats much less common now than it used to be.

<http://www.linuxjournal.com/article/6435>
<http://en.wikipedia.org/wiki/Write_Only_Memory>

None of that is any news to anyone with a clue.

How do you recommend doing a live Windows CD boot?

Barts is very convenient.

Its easier to use a Linux live CD for a Win system tho,
basically because it doesnt have to be built, just burnt.

I've seen some suggestions on the web, but they were mostly hacks of some sort.

Barts isnt.

If you know of a good way to make a general bootable windows CD with recovery tools, I'd be very interested.
Barts.

I find Linux live CDs useful for recovering problematic windows installations, but I'd be happy to expand my toolkit.

Barts is a lot more configurable and you can for example build a
bootable CD that includes various tools that you might need when
still working out what the hardware problem is if it isnt just a simple
hard drive failure etc or you prefer to continue to use the Win UI
when recovering instead of having to understand the basics of
Linux enough to be able to use that for the recovery.

There are others like the Ultimate Boot CD for Win that isnt strictly
speaking legal in the sense that some licensed software is included.
Corse you can always buy the licenses for whats included if you want.

If Linux live CDs can access windows software raid, then that's a good familiar safety net.

The main downside with that approach is that you do need to understand
the basics of Linux or choose a Linux thats very close to Win UI wise etc.

Even with a Linux thats very close to the Win UI, its not quite as convenient
to use the rather different UI with smartctl instead of say Everest which is
rather more convenient to use with a Win live CD to get a look at the state
of the hard drives when deciding what has died etc.

Agreed - having the second disk external is much better (I think I mentioned that myself in an earlier post). It also
means it can be disconnected when you are not running backups (reducing wear on the disk, and risks of any user / OS /
malware issues), and you can easily connected it to another machine.

Copying to CD or DVD is still very popular - it's a familiar solution for many people.

And is very viable for backing up only the irreplaceable data
and being prepared to wear the extra effort of a full reinstall
in the very unlikely event of say a hard drive failure or fire etc.

And perhaps the use of internet-based backup is more common here in Norway than in many other parts of the world,
since we have a high percentage of users with fast (10 Mbit) broadband connections.

It isnt all that commonly used because much of the world
uses DSL that doesnt have that fast an upstream speed.

Very viable for incremental backup but not really that viable for full backups for most.

Yes, hard drive failure is uncommon - especially in comparison to other causes of data loss such as user error.

Depends on the user.

I personally have a completely different backup scheme for code
with a high backup frequency manually initiated where I do another
copy of the current code at every major step in the process of writing
and modifying the code, to the same hard drive the code lives on,
with the code also included in the normal backup scheme as well.

That allows you to handle gracefully the all to common situation
where you decide a particular approach to the code will be useful
but it turns out to have a major downside that you didnt realise,
so you can step back to where you were before that sidetrack etc.

 

[Rod Speed]

David Brown wrote

Rod Speed wrote

I've been describing what I see as a good backup system - not just a home backup system (though that's relevant too).

And I was pointing out that fast isnt necessary for home systems.

I agree with much of your comments that some of my suggestions are overkill for many home backup systems. For most
home systems, it's enough to copy regularly to CD/DVD and, as you say, make duplicates of your photos.

The main advantage with more complete imaging backup to an
external hard drive is that you dont have to bother with go thru
the considerable reinstall and reconfigure step and that can be
non trivial with some of the more complex home systems.

But the OP appears to have data that is particularly important to him,
Yes.

it's working data rather than archived photos, and it's serious enough
that he's considering raid. That's sounds to me more like professional or serious amateur work, and requires a bit
more out of a backup system.

But like I said, you dont know that he hasnt got one already and is just
considering the raid so that he doesnt lose much work on a hardware
failure, doesnt lose what work he had done since the last backup.

That can be a complete pain in the arse even if you lose just one days
work for example, let alone a whole weeks work with weekly backup.

So I've been discussing general good backup practice - it must of course be adapted to suit the situation, and with an
appropriate balance of protection, costs, hardware, effort, etc.

And yours was mostly overkill even for small business.

I'll leave your "Not relevant to most of the home systems being
discussed" comments unchallenged when I agree that the point is irrelevant or overkill for home systems, even though
that's not all we are discussing (at least, not all that /I/ am discussing).

For photos and things that typically have multiple archives (such as
burning to CD after copying them onto the disk), as well as things
that don't change often, then I agree. But if you are also storing
other changing data, then it's useful to have snapshots over time.

Few home systems have much of that tho.

This is mainly for protection against user error - overwriting or deleting something you wanted to keep, and only
realising it at a later date.

Sure, but thats easily handled just by doing the backups
to DVD and keeping all the DVDs written. They are so
cheap that there is no reason to not keep them.

For single computer systems, this applies to the files on that system, and then it is relevant.

But not feasible for home systems.

It also depends on the circumstances - if your house has burned down,
loss of your digital photos collection is probably a very minor concern.

Dunno, that is something that I would personally ensure cant happen
by swapping one copy of backup media with someone else etc.

But if your main PC is an easily stolen laptop, you want to
make sure you have copies that are kept in a different place.

And you might as well cover the photos the same way.

This I *don't* agree with. You must test that you can recover your backup data - it's too easy to make a mistake.

There isnt really any way to make a mistake with the main approaches
now used with home systems, writing to DVDs and external hard drives.

For a typical home backup system this is not much more than putting a backup CD in another computer and checking that
you can see and read the files - it's absolutely worth the effort.

Corse you can read the files. You dont even get the effect you
used to get with floppy drives that you can get a drive that can
read and write floppys which cant be read in a different drive.

What I was thinking of here is if you have a backup arrangement that is tied to specific hardware (such as a tape
drive) or specific
software - you have to consider how to get your data back if you need to buy a new drive and/or new software. An
example might be if you have an old system that uses iOmega ZIP drives for backup. If that machine gets stolen, you
have the challenge of finding a new ZIP drive for your replacement machine (eBay might help), and then hope that there
were no head alignment issues with the old drive that make the disks unreadable with another drive.

It makes a lot more sense to use universal media like DVDs,
external hard drives and multiple USB pen drives etc.

You can well say that such problems are highly unlikely. But highly
unlikely still means vaguely possible, and it's better to think of
such possibilities and then dismiss them as obviously being no
problem for a given backup solution, than to ignore them totally.

Its better to avoid that sort of hardware completely with home systems.

An rsync to a second PC is an automatic system with no costs - whether it's a home system or a professional system.

Yes, but not worth the setup hassle with most home systems.

For home users, habit (such as copy photos to drive then CD) is a clear procedure.

That was a comment on the bit in brackets.

It's also easy to set your email/calendar system to remind you once a week to do your backups.

Without some sort of procedure or habit, backup regimes start out well and quickly get forgotten.

Sure, I was only commenting on the bit in brackets. Should have said that more clearly.

It should be easy enough if you have an automatic backup system

Most home systems dont.

- and without it, how are you going to know that the backups you think have been running every night actually stopped
with a "disk full" error two months ago?

Most home systems arent done like that.

 

[Sydney Lambe]

As I wrote, the card needs to be able tho handle the disk size
of all used disks.




Interestingly my current employer uses Linux software RAID for
the base system and then puts Windows installations in VMWare
on top of that. This works pretty well, but requires people that
understand Linux and Windows.


I agree. There is a cultural difference. Unix folks expect their
stuff to reside on servers, be backed up by the system administrator,
etc.. If they operate their own (say) Linux box, they do all that
by themselves, because they regard it as part of professional
computing. The MS crowd comes out of the "computing is easy"
crows that then sometimes becomes the "where has all my data
gone" crowd.

But even the more knowledgeable MS users tend to use
servers with system administrators today.




I habve seen a number of disk failures. I have run about 50 drives
in a climate controlled server room for 4 years, with maybe one
unexplained failure (the others were dropped in shipping). I have
seen a RAID1 of Maxtors fail from heat (helped the colleguae that
set this up to recover some data). I have seen a lot of
DeathStars die, including 2/2 of my own. I also have one 2.5"
HDD in a RAID1 that has a tendency to transient errors. It drops
out of the RAID about 2-3 times a year.

I agree that for well handled disks under good operating
conditions fail rarely, and in line with the manufacturer-stated
reliability. However many desktop systems do not provide
these conditions. And there can be surprises, like the DeathStars
or the exceedingly heat-sensitive Maxtors of some years ago.

You handle a lot of HDDs in your work. Maybe you believe HDDs are more
reliable than I do. There's so many stories in NewEgg reveiws talking of
dead drives. ISTR some figures saying 5% or whatever got RMAed.

That Google report on HDDs left me thinking they are less reliable than
I thought.

Some if your HDDs failed if they had been shipped badly. When I get a
HDD I can see the packaging but even if it looks ok I can't tell if the
courier has played basketball with it. I find out when it's too late, so
I figured data mirroring (RAID-1) would overcome a HDD failing.

 

[Sydney Lambe]

Consider a setup like this:

Disk A, 750G:
partition 1, 250G
partition 2, 500G
Disk B, 500G:
partition 1, 500G

You put your OS, programs, etc., on "C drive" on partition A.1. You
mirror partition A.2 and B.1, and put your data on this "D drive".

Disk A dies.

You have a safe copy of all your data on the mirror B.1. But your OS
drive is dead.

How do you get your data off B.1 again?

David, I'm the OP. Guess I see what you're trying to show: how data or
programs in the second partition of the oversized drive could be hard to
recover. Might be complicated if there's a separate partition for
swapfiles etc.

To avoid that problem, these two hard drives are only for data. Backing
up and using off site storage is important but am trying to prevent loss
of irreplacable data from a HDD crash.

A backup will always miss some data depending on the frequency of
backing up, so RAID-1 is sort of data mirroring.

My data is always changing because am editing and re-editing audio
files. Each editing session creates a new master so I can't just do a
once only backup of some master files to another hard drive (stored away
from the machine).

Hope that's clearer. Rod seems to have made some accurate assumptions
about my question.

 

[Sydney Lambe]

X-No-Archive: yes

I believe that with a hardware RAID solution (and I think that also
applies to pretend hardware RAID that is often supported by motherboards
and cheap cards), you have to use whole disks in the RAID. That means
the 750 GB disk will be treated as 500 GB to make the mirroring work.

Can you explain what is meant by "pretend hardware RAID"? Are some
implementations of RAID less good than others (for the same level of RAID)?

If you use software raid, you can do it on a partition level - use 250GB
of the 750 GB for the OS, software, swap, etc., and use a 500 GB
partition for mirroring with a matching partition on the other disk.

Didn't know about software RAID. Seems to fit with dynamic disks. I'm
not using dynamic disks.

 

[Rod Speed]

Sydney Lambe wrote

X-No-Archive: yes

That isnt a good idea, it prevents anyone from researching
the issues being discussed using groups.google.

Can you explain what is meant by "pretend hardware RAID"?

Its basically a way of sneering at the most basic RAID as seen on
motherboards which isnt much more than software RAID in the sense
that all of the RAID functionality is done in software, in the driver etc.
http://en.wikipedia.org/wiki/RAID#Purpose_and_basics, last part of that section.

Are some implementations of RAID less good than others (for the same level of RAID)?

Yes, most obviously when the calculations are done
in software rather than with a dedicated cpu etc.

Didn't know about software RAID. Seems to fit with dynamic disks.

No, its quite different to dynamic disks.
http://en.wikipedia.org/wiki/Dynamic_disk

 

[David Brown]

David Brown wrote


And I was pointing out that fast isnt necessary for home systems.

Fair enough.

The main advantage with more complete imaging backup to an external
hard drive is that you dont have to bother with go thru the
considerable reinstall and reconfigure step and that can be non
trivial with some of the more complex home systems.

I agree on that too.

Certainly for some users, a full /hardware/ mirror of their disks would
make life a lot easier if a disk dies, since they don't need to
re-install everything. Here we are no longer talking about the security
of your important and irreplaceable data, but the convenience of being
able to continue using the PC as before. Re-installation can be a big
issue - often home users have got nothing more than some sort of "system
restore" disk that may or may not do the job. Then there is the
installation of everything else added during use of the system.

On the other hand, many home systems could benefit from a wipe and a
re-install - suddenly, windows will run much faster without a few years
worth of accumulated junk.

But like I said, you dont know that he hasnt got one already and is
just considering the raid so that he doesnt lose much work on a
hardware failure, doesnt lose what work he had done since the last
backup.

That can be a complete pain in the arse even if you lose just one
days work for example, let alone a whole weeks work with weekly
backup.

OK. And now that the OP has replied in another thread, it looks like
this is exactly the case for him.

And yours was mostly overkill even for small business.

Maybe, maybe not. It's all a matter of balancing needs, wants, and
costs. But I prefer to have considered the questions anyway.

Sure, but thats easily handled just by doing the backups to DVD and
keeping all the DVDs written. They are so cheap that there is no
reason to not keep them.

It's something I think is important for backup systems, and as you say
it's easy to implement.

But not feasible for home systems.

You can come a long way by storing your files and/or copies of the files
on a separate partition on the same machine (or a separate external
harddisk, or even just a USB flash). You are not making an infallible
system, but you are reducing risks at a low cost.

Dunno, that is something that I would personally ensure cant happen
by swapping one copy of backup media with someone else etc.

Yes, that's an easy way to do it.

And you might as well cover the photos the same way.



There isnt really any way to make a mistake with the main approaches
now used with home systems, writing to DVDs and external hard drives.

There are *always* ways to make a mistake - people can be very ingenious
in how they get things wrong. What about labelling your backup CD's
with permanent pen on the data side? I'm sure *someone* has tried it.
I'd agree that it's harder to make mistakes with current approaches, but
it's also easier to test.

Corse you can read the files. You dont even get the effect you used
to get with floppy drives that you can get a drive that can read and
write floppys which cant be read in a different drive.


It makes a lot more sense to use universal media like DVDs, external
hard drives and multiple USB pen drives etc.

I agree - and in such cases, testing is trivial. But that doesn't mean
it should be ignored. And there are always people who like to try
something different, such an old second-hand tape drive bought cheap.

It's a simple rule - any system designed to prevent failures or loss, or
to help recover after a failure, needs to be tested so you know that it
works, and you know what to do if you get a failure.

Its better to avoid that sort of hardware completely with home
systems.

Agreed, of course.

Yes, but not worth the setup hassle with most home systems.

Probably not, unless you already understand about that sort of thing.
But it's still a zero monetary cost system.

That was a comment on the bit in brackets.

OK



Sure, I was only commenting on the bit in brackets. Should have said
that more clearly.

OK



Most home systems dont.


Most home systems arent done like that.

And in those cases, you don't need any reporting or monitoring. It's
only if things run in the background somewhere that you need the extra
reporting.

 

[David Brown]

David Brown wrote

That can obviously be backed up even if it is the case.

Registry backup is easy - restoring specific parts of the registry is
not necessarily easy.

You initially proclaimed that there is no point in mirroring with
Win.

That was just plain wrong.



Which like I said, doesnt get used much anymore for home system
backup.


Thats not write only media, thats a ****ed up backup.

You've *almost* got the point. "Write-only media" is media that cannot
be read at all - it's not something that *anyone* would knowingly use at
any time. But people *do* use it by mistake, thinking they are doing a
proper backup.


To quote from Wikipedia:

Write-only memory is the antithesis of read-only memory (ROM). By
definition, a WOM is a memory device which can be written but never
read. Since there seems to be no obvious utility for such a memory
circuit, from which data cannot be retrieved, the concept is most often
used as a joke or a metaphor for a failed memory device.


Maybe you thought I was talking about WORM drives or something like that?

Thats much less common now than it used to be.


None of that is any news to anyone with a clue.



Barts is very convenient.

Its easier to use a Linux live CD for a Win system tho, basically
because it doesnt have to be built, just burnt.


Barts isnt.


Barts.

Thanks - that is useful information. I'll have a good look at it this
evening.

Barts is a lot more configurable and you can for example build a
bootable CD that includes various tools that you might need when
still working out what the hardware problem is if it isnt just a
simple hard drive failure etc or you prefer to continue to use the
Win UI when recovering instead of having to understand the basics of
Linux enough to be able to use that for the recovery.

I am happy using a Linux live CD for a lot of hardware issues, and for
some software recovery issues (like copying data out of the machine,
reseting administrator passwords, running a ClamAV scan). But most
tools for finding and removing malware are designed to run natively on
windows - being able to run these from a live CD will be a very useful tool.

There are others like the Ultimate Boot CD for Win that isnt strictly
speaking legal in the sense that some licensed software is included.
Corse you can always buy the licenses for whats included if you
want.

I've just had a quick look at their website. It's possible that it
would be useful for some things, but I think that combination of System
Restore CD (useful for low level stuff, such as disk imaging), Knoppix
(nice for higher level stuff like finding files and copying them over to
other systems), and now Barts (especially for malware scanning) will
cover what I need.

The main downside with that approach is that you do need to
understand the basics of Linux or choose a Linux thats very close to
Win UI wise etc.

True enough. In this case, it's for my own use, and I'm happy with
Linux and/or Windows. However, there are always some things that are
easier or more familiar in one system rather than another, so it will be
useful to have a Barts CD lying around.

Even with a Linux thats very close to the Win UI, its not quite as
convenient to use the rather different UI with smartctl instead of
say Everest which is rather more convenient to use with a Win live CD
to get a look at the state of the hard drives when deciding what has
died etc.



And is very viable for backing up only the irreplaceable data and
being prepared to wear the extra effort of a full reinstall in the
very unlikely event of say a hard drive failure or fire etc.


It isnt all that commonly used because much of the world uses DSL
that doesnt have that fast an upstream speed.

Very viable for incremental backup but not really that viable for
full backups for most.

Here in Norway, fibre based systems (or cable in cities) is increasingly
common. I think something like 60-70% of the households in my town
(about 3500 people, I believe) have fibre for TV, telephone and Internet
access - with the minimum Internet speed being 10 Mbps up and down. You
can copy a DVD's worth of data in about an hour and a half at that
speed, which is a lot of photos.

Depends on the user.

I personally have a completely different backup scheme for code with
a high backup frequency manually initiated where I do another copy of
the current code at every major step in the process of writing and
modifying the code, to the same hard drive the code lives on, with
the code also included in the normal backup scheme as well.

That allows you to handle gracefully the all to common situation
where you decide a particular approach to the code will be useful but
it turns out to have a major downside that you didnt realise, so you
can step back to where you were before that sidetrack etc.

You need something like that for coding - some sort of source code
management is invaluable. I use subversion on a server, but the best
solution depends on the kind of coding you do, how you like to work, etc.

 

[David Brown]

David, I'm the OP. Guess I see what you're trying to show: how data or
programs in the second partition of the oversized drive could be hard to
recover. Might be complicated if there's a separate partition for
swapfiles etc.

Yes, it might be complicated (Rod has explained how it could be done).

To avoid that problem, these two hard drives are only for data. Backing
up and using off site storage is important but am trying to prevent loss
of irreplacable data from a HDD crash.

A backup will always miss some data depending on the frequency of
backing up, so RAID-1 is sort of data mirroring.

As long as you have a good backup system in addition to the raid, you
have things covered. Raid mirroring will give you the short-term
security between backups, backups will give you long-term security
against user error or other disasters.

My data is always changing because am editing and re-editing audio
files. Each editing session creates a new master so I can't just do a
once only backup of some master files to another hard drive (stored away
from the machine).

Hope that's clearer. Rod seems to have made some accurate assumptions
about my question.

And I seem to have made a few inaccurate assumptions. Either way, it's
important that you think about what could happen to your data, and how
you can get it back if something goes wrong.

 

[David Brown]

X-No-Archive: yes




Can you explain what is meant by "pretend hardware RAID"? Are some
implementations of RAID less good than others (for the same level of RAID)?

Some cards are advertised as having "hardware RAID", but it's really
just some BIOS settings that work together with special drivers in the
OS. This may have some advantages over a pure software raid solution
for some uses (I believe non-server Windows cannot be installed on a
pure software raid, but perhaps it can on such a pseudo hardware raid -
maybe Rod could comment here?). But unlike a true hardware RAID
solution, all the mirroring and/or parity calculations are handled by
the host processor rather than a dedicated RAID device. That can have
implications for speed, and perhaps also for recovery if you need to
move the disks to a different computer. I can't give any details here,
just ideas of things to consider.

Didn't know about software RAID. Seems to fit with dynamic disks. I'm
not using dynamic disks.

"Dynamic disks" is, I think, the windows term for "software raid".

 

[Arno]

You handle a lot of HDDs in your work. Maybe you believe HDDs are more
reliable than I do. There's so many stories in NewEgg reveiws talking of
dead drives. ISTR some figures saying 5% or whatever got RMAed.

If you run drives not well (e.g. too hot) they do die more often.
I have seen my share of dead drives.

That Google report on HDDs left me thinking they are less reliable than
I thought.

I have some issues with that report (basically some things they do
are scientifically unsound and some explanations are missing), but
yes.

Some if your HDDs failed if they had been shipped badly. When I get a
HDD I can see the packaging but even if it looks ok I can't tell if the
courier has played basketball with it. I find out when it's too late, so
I figured data mirroring (RAID-1) would overcome a HDD failing.

That is whaty I do at home too.

Arno

 

[Arno]

X-No-Archive: yes

Can you explain what is meant by "pretend hardware RAID"? Are some
implementations of RAID less good than others (for the same level of RAID)?

Also called FakeRAID. This is baiscally software RAID in the BIOS
and the worst alternative, because it has the incompatibility
issues and problem when the controller fails that hardware RAID has,
is less transparent than true hardware RAID and has none of the
advantages of hardware RAID.

Either use OS-level software RAID (my preference) or true hardware
RAID. It is expensive though. With a <200EUR/USD controller you can
almost be certain it is FakeRAID.

Didn't know about software RAID. Seems to fit with dynamic disks. I'm
not using dynamic disks.

Dynamic disks are basically software RAID, just that Microsoft is to
arrogant to uses the names everybody else uses and has to do their own
thing, which is typically a bit more obscure, a bit less reliable and
a bit less well though out what the rest of the community does.

Arno

 

[Rod Speed]

Some cards are advertised as having "hardware RAID", but it's really
just some BIOS settings that work together with special drivers in the
OS. This may have some advantages over a pure software raid solution
for some uses (I believe non-server Windows cannot be installed on a
pure software raid, but perhaps it can on such a pseudo hardware raid
- maybe Rod could comment here?).

Nothing to stop you using a server Win on a desktop to get the mirroring etc.

But unlike a true hardware RAID solution, all the mirroring and/or parity calculations are handled by the host
processor rather than a dedicated RAID device. That can have implications for speed, and perhaps also for recovery if
you need to move the disks to a different computer. I can't give any details here, just ideas of things to consider.

"Dynamic disks" is, I think, the windows term for "software raid".

Its more a superset, dynamic disks are more than
just software raid, most obviously with spanning etc.

 

[Rod Speed]

Registry backup is easy - restoring specific parts of the registry is not necessarily easy.

You dont need to restore specific parts on a hard drive failure with the OS not mirrored.

And it isnt hard to restore specific parts either.

You've *almost* got the point.

I've always had it entirely.

"Write-only media" is media that cannot be read at all - it's not something that *anyone* would knowingly use at any
time. But people *do* use it by mistake, thinking they are doing a proper backup.

Not anymore with home systems, because they
dont use the media that has that possibility anymore.

To quote from Wikipedia:

Write-only memory is the antithesis of read-only memory (ROM). By
definition, a WOM is a memory device which can be written but never
read. Since there seems to be no obvious utility for such a memory
circuit, from which data cannot be retrieved, the concept is most
often used as a joke or a metaphor for a failed memory device.

So isnt relevant to the backup being discussed.

Maybe you thought I was talking about WORM drives or something like that?

Nope. I was rubbing your nose in the fact that the media that can produce
that result with backup isnt used for home system backup anymore except
by dinosaurs that are crippling along with media way past its useby date.

Its very unlikely that the OP is one of those.

Thanks - that is useful information. I'll have a good look at it this evening.

I am happy using a Linux live CD for a lot of hardware issues, and for
some software recovery issues (like copying data out of the machine,
reseting administrator passwords, running a ClamAV scan). But most
tools for finding and removing malware are designed to run natively on
windows - being able to run these from a live CD will be a very useful tool.
Yep.

I've just had a quick look at their website. It's possible that it
would be useful for some things, but I think that combination of
System Restore CD (useful for low level stuff, such as disk imaging),
Knoppix (nice for higher level stuff like finding files and copying
them over to other systems), and now Barts (especially for malware
scanning) will cover what I need.

Yeah, its more for those who want that config stuff done for them.

Even just building a Barts CD is a bit offputting for some.

True enough. In this case, it's for my own use, and I'm happy with Linux and/or Windows. However, there are always
some things that are easier or more familiar in one system rather than another, so it will be useful to have a Barts
CD lying around.

Here in Norway, fibre based systems (or cable in cities) is increasingly common.

Yeah, we're proposing to do that to 90% of the population in Australia.

Remains to be seen if the Senate will buy the A$43B bill for it tho.

I think something like 60-70% of the households in my town (about 3500 people, I believe) have fibre for TV, telephone
and Internet access - with the minimum Internet speed being
10 Mbps up and down.

We're talking about 100Mb and the Japs are moving to 1Gb.

You can copy a DVD's worth of data in about an
hour and a half at that speed, which is a lot of photos.

But not too many HD movies.

You need something like that for coding - some sort of source code
management is invaluable. I use subversion on a server, but the best
solution depends on the kind of coding you do, how you like to work, etc.

Yeah, I much prefer a manual approach with a copy
done when you complete a substantial chunk of work
etc with that happening multiple times a day.

 

[Rod Speed]

David Brown wrote

Rod Speed wrote

Fair enough.

I agree on that too.

Certainly for some users, a full /hardware/ mirror of their disks
would make life a lot easier if a disk dies, since they don't need to re-install everything. Here we are no longer
talking about the
security of your important and irreplaceable data, but the
convenience of being able to continue using the PC as before. Re-installation can be a big issue - often home users
have got
nothing more than some sort of "system restore" disk that may or may not do the job. Then there is the installation
of everything else added during use of the system.

On the other hand, many home systems could benefit from a wipe and a re-install - suddenly, windows will run much
faster without a few years worth of accumulated junk.

That hasnt been true for a while now with Win.

And the repair install which preserves the settings and restore
points has eliminated most need for a clean install too now.

And the files and settings transfer wizard has made clean
installs much easier now too, again preserving the settings.

The main problem with an image is that you normally lose stuff
like email etc unless you have more than the default config.

OK. And now that the OP has replied in another thread, it looks like this is exactly the case for him.

Maybe, maybe not. It's all a matter of balancing needs, wants, and costs. But I prefer to have considered the
questions anyway.

I dont bother with stuff like signing things out etc for small business.

It's something I think is important for backup systems, and as you say it's easy to implement.

You can come a long way by storing your files and/or copies of the
files on a separate partition on the same machine (or a separate
external harddisk, or even just a USB flash). You are not making an infallible system, but you are reducing risks at
a low cost.

I dont believe that separate partitions give anything useful.

And external hard drives are much less reliable than internal ones.

They are better for backup, but not for the originals.

Yes, that's an easy way to do it.

There are *always* ways to make a mistake -

Not in that sense of the backup not actually happening.

people can be very ingenious in how they get things wrong. What about labelling your backup CD's with permanent pen
on the data side? I'm sure *someone* has tried it.

Sure, but testing isnt the best way to avoid that.

I'd agree that it's harder to make mistakes with
current approaches, but it's also easier to test.

Testing can be quite dangerous tho.

I agree - and in such cases, testing is trivial. But that doesn't mean it should be ignored.

Pointless bothering if the media is used routinely for other
than backup, you know it works, not need to prove that again.

And there are always people who like to try something different, such an old second-hand tape drive bought cheap.

Its impossible to adequately prescribe proceedures that cover
all weird stuff that might happen with that sort of approach.

It's a simple rule - any system designed to prevent failures or loss,
or to help recover after a failure, needs to be tested so you know
that it works, and you know what to do if you get a failure.

Trouble is that with home systems where the failure rate is so low
that many never ever see a hard drive failure, there is no chance
that they will be able to remember what to do in case of failure.

What makes sense is to get them to backup the irreplaceable
stuff properly and ask for assistence when the shit does hit the fan.

They are mostly quite incapable of deciding if the hard drive
is dying, let alone able to restore onto a replacement.

Thats the other big advantage of mirroring, you dont normally get
an unusuable system which can encourage those who dont know
what they are doing to thrash around and **** the system completely.

Agreed, of course.

Probably not, unless you already understand about that sort of thing.
But it's still a zero monetary cost system.

Sure, but is too hard for most home users to setup.

They have enough trouble networking their home PCs.

 

[David Brown]

David Brown wrote

That hasnt been true for a while now with Win.

And the repair install which preserves the settings and restore
points has eliminated most need for a clean install too now.

And the files and settings transfer wizard has made clean installs
much easier now too, again preserving the settings.

I'm not sure I agree entirely. Certainly the situation has improved,
and certainly it's possible to have windows systems running for years
without suffering from any "ageing" effects - I have a PC at home that
has been running W2K for at least 8 years.

But I have also seen plenty of home PCs that are completely bogged down
in trial software, adware, malware, "security" software, software that
comes with new hardware (it seems that every printer, every camera,
every MP3 player comes with its own massive suite of software, and
non-experts think they need it), and so on.

Much of this can be uninstalled, and with enough work with registry
cleaners, malware removers, etc., you can tidy up the system a lot. But
often it's simply faster, easier, and more effective to do a re-install.
And if the root cause is malware infections, it can be best to do a
clean re-install without preserve any settings or software.

The main problem with an image is that you normally lose stuff like
email etc unless you have more than the default config.

Yes, email is the big issue. Everything else is typically saved as
simple files, and is easy to find and backup or copy to a new
installation or new PC. But email is often a problem to preserve, with
many people using Outlook Express and POP3. Their email history is
locked into a non-standard format mailbox, with important settings
scattered about the registry and other files. I'm not always convinced
by the privacy policies of online email systems (gmail et.al.), but they
are far preferable to POP3 (I prefer IMAP on a server, but IMAP servers
are rare in homes).

I dont believe that separate partitions give anything useful.

I do - regardless of any real or perceived reliability benefits of
having separate file systems.

I find it so much easier to keep a machine organised when all my data is
on a separate partition. Pathnames such as "Documents and
Settings\david\My Documents" just make me cringe.

And external hard drives are much less reliable than internal ones.

They are better for backup, but not for the originals.

I'd agree on that.

Not in that sense of the backup not actually happening.


Sure, but testing isnt the best way to avoid that.

Testing shows you that you've got a problem - then you can figure out
how to avoid it.

Testing can be quite dangerous tho.

Testing should *not* be dangerous. Checking that your raid works as you
expect can be difficult if you can't do it all from software (I have no
idea if windows software raid can simulate an error or not).

I can't believe I'm having a discussion here with someone actively
recommending that people don't test their systems!

I'll grant you that simple backup systems don't need *much* testing.
And I'm also happy with the idea that most people need help fixing their
system after a disaster. But that doesn't mean you shouldn't do some
quick tests yourself.

 

[Rod Speed]

David Brown wrote

Rod Speed wrote

I'm not sure I agree entirely.

Its true anyway.

Certainly the situation has improved, and certainly it's possible to have windows systems running for years without
suffering from any "ageing" effects - I have a PC at home that has been running W2K for at least 8 years.

But I have also seen plenty of home PCs that are completely bogged down in trial software, adware, malware, "security"
software,

Thats a different matter to your original.

software that comes with new hardware (it seems that every printer,
every camera, every MP3 player comes with its own massive suite of software, and non-experts think they need it), and
so on.

Those dont slow anything down.

Much of this can be uninstalled, and with enough work with registry
cleaners, malware removers, etc., you can tidy up the system a lot.

You dont need to with most non Win9x systems.

But often it's simply faster, easier, and more effective to do a re-install.

Different matter entirely to your original.

And if the root cause is malware infections, it can be best to do a clean re-install without preserve any settings or
software.

Anyone with a clue doesnt get malware infections.

Yes, email is the big issue.

Only for some. Quite a few dont bother to keep old email.

Everything else is typically saved as simple files, and is easy to find and backup or copy to a new installation or
new PC. But email is often a problem to preserve,

Not if you know what you are doing.

with many people using Outlook Express and POP3.

Its trivial to retain if you know what you are doing.

Their email history is locked into a non-standard format mailbox, with important settings scattered about the registry
and other files.

Still easy enough to handle if you know what you are doing.

I'm not always convinced by the privacy policies of online email systems (gmail et.al.), but they are far preferable
to POP3 (I prefer IMAP on a server, but IMAP servers are rare in homes).

And most dont need much 'privacy' with home systems.

I do

More fool you.

- regardless of any real or perceived reliability benefits of having separate file systems.

There are not real ones.

I find it so much easier to keep a machine organised when all my data is on a separate partition.

More fool you.

Pathnames such as "Documents and Settings\david\My Documents" just make me cringe.

More fool you.

I'd agree on that.

Testing shows you that you've got a problem - then you can figure out how to avoid it.

Only fools do stupid stuff like that.

Testing should *not* be dangerous.

Then there's the real world...

Checking that your raid works as you expect can be difficult if you can't do it all from software (I have no idea if
windows software raid can simulate an error or not).

We were discussing testing of backup, not raid.

I can't believe I'm having a discussion here with someone actively recommending that people don't test their systems!

I never ever said anything even remotely resembling anything like that.

I'll grant you that simple backup systems don't need *much* testing.

They dont need any, just check that the backup is actually
writing to the backup media and it taking something like a
normal time to do that and that the result is an appropriate size.

Thats not testing, thats just monitoring the backup.

And I'm also happy with the idea that most people need help fixing their system after a disaster. But that doesn't
mean you shouldn't do some quick tests yourself.

Too dangerous for most.

I dont believe that it even makes much sense for the average
home user to attempt their own backup. Its too hard for them
to ensure that they really are backup up everything that matter.

Its too late when a hard drive dies to discover that they havent
been backing up the encryption keys or digital certificates etc
and no real way for the average home user to test that safely.

 

[David Brown]

David Brown wrote


More fool you.


More fool you.

Eventually there's a limit to how long I'm going to continue this
thread. I know you have a lot of experience and knowledge, and there's
plenty that I can benefit from (the tip about Barts being an example).
But I'm fed up trying to apologise because /you/ can't understand
sarcasm and exaggeration (I'm British - it's part of our culture and
language), and I'm fed up with you repeatedly changing the discussion so
you can cut-and-paste "you're wrong" at every paragraph. Telling me I'm
foolish for my personal preferences of how I use a computer is the last
straw.

Maybe you think yourself so high-and-mighty and infallible that you'll
plonk me for this. If not, perhaps we'll meet again on another threat
and get off on a better foot.

mvh.,

David