Force Rebuild RAID = Bad Idea

[lcoughey]

I think that every technician, especially me, tend to learn things the hard way. But, I always want to try and share my mistakes so that others won't have to follow my path. In this case, I'd like to share a common mistake made by many technicians before they call on the services of Recovery Force.

When a RAID drive goes offline, assuming that the RAID controller is doing its job right, the remaining drives will continue to function by using parity calculation so that they user does not experience any server down time. Unfortunately, if the failed drive is not replaced right away and another drive fails, the server instantly goes offline. Sadly, it is not uncommon for RAID controllers (especially older Dell servers) to knock a healthy drive offline.

So, the technician comes onsite and sees that the server is offline and after testing and confirming that all drives are healthy and follows on to force the RAID back online, assuming that all 4 drives went offline at the exact same time. Unfortunately, this is very rare and it is usually days, weeks or even months since the first drive failed. By forcing all the drives back online, the RAID controller recalculates all parity blocks and scrambles most of the files that were created or modified since the first drive failure. Without a backup, there is absolutely no going back to under the damage caused by the rebuild.

How does one avoid this?

1. Always have a current backup, even if the data is stored on a RAID.
2. If a single drive is offline, make sure that the accessible data is backed up and readable before replacing the failed drive.
3. If the server is offline because of multiple drives being offline, always get a full sector-by-sector copy of every drive and recover the data virtually with the copies, rather than the originals. Set the originals aside and never reuse them until you are 100% certain that the data is 100% recovered.

It is best not to take chances with client's data and recommend that technicians contact Recovery Force or their data recovery partner for free advice on how to approach the failure.

[RolandJS]

Do you want me to link this to college cafe's computer forum? Is the info still applicable to present raid setups?

[lcoughey]

You are welcome to link to it, yes.