4 ms·

> the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Well there's one issue that another whole drive will fail and you'r

by eck 12y ago

> the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Well there's one issue that another whole drive will fail and you're screwed. The other issue is that modern disks have an unrecoverable read error rate compared to their size such that a total cover-to-cover read -- necessary on every remaining disk to rebuild a RAID5 -- is kinda unreliable, even with a supposedly healthy disk.

ghshephard 12y ago
I'm interested - do you have a citation for that? I'm wondering if manufacturers of large drives accommodate for the statistically increased chance of a cover-cover failure (based on having so much data) by increasing their redundancy data/checksums to keep it constant.
- ryan-c 12y ago
  That is based off of the BER (bit error rate) of the drive, as stated by the manufacturer. It's usually spec'd at 1e-14 or 1e-15 - that is to say one in every hundred trillion or one in every quadrillion bits. This works out to 12.5TB or 125TB respectively. http://www.enterprisestorageforum.com/storage-hardware/selecting-a-disk-drive-how-not-to-do-research-1.html http://www.enterprisestorageforum.com/storage-hardware/selec...
- cmurf 12y ago
  The problem is due to a common misconfiguration. Consumer drives have longer recoveries potentially, upwards of minutes, trying to recover a flaky sector. The OS sees this as an unresponsive drive, and eventually does a link reset. This is 30 seconds by default on Linux, not sure about others. The link reset prevents the drive from reporting an explicit read error along with the affected sector LBA. That information is needed for RAID to know what data to rebuild from parity, send that up to the app layer, and also send a good copy back to the sector that reported the read error. So eventually there's an accumulation of these, and in case of a drive failure and another drive that produces a URE, poof, imploded array. Now, you can recover from this, sorta, but it's tedious and requires a sort of skill to do it. So most people give up. Ergo RAID is not a backup. Backup your RAIDs. And make sure drive SCT ERC is shorter than the kernel's SCSI/ATA command timer. Ideally shorten the drive timeout. If that can't be done (consumer drive) then increase the kernel's command timer. Both of these are per device settings.
- rasz_pl 12y ago
  fun fact: Commodore Amiga floppies used 32bit XOR checksum = 3% of undetected two bit errors!! http://www.techtravels.org/amiga/amigablog/?p=280 http://www.techtravels.org/amiga/amigablog/?p=280 This is probably because Amiga didnt have real hardware floppy-disk controller, just a general IO (CIA) chips, and read raw serial datastream into ram, all the decoding was done in software. Similar to Apple II 2x 8bit XOR checksum.
static_noise 12y ago
> Well there's one issue that another whole drive will fail and you're screwed. You do know that in a RAID6 two drives may fail without causing data loss? In any way you need one or more hotspares ready plugged in in order to keep the time window short. RAID6 is not perfect but the probability of 3 drives failing during the rebuilt window is much lower than the probability of 2 drives failing. (One also has to consider errors such as memory corruption, chip failures or catastrophic failure to the power supply where no RAID level will protect you from.) A RAID1 built of two RAID6s may be necessary to avoid performance drops during rebuild. In the case of multiple failures a RAID6+1 setup will protect you from at least 4 hard drives failing in the rebuild time window. > The other issue is that modern disks have an unrecoverable read error rate compared to their size such that a total cover-to-cover read -- necessary on every remaining disk to rebuild a RAID5 -- is kinda unreliable, even with a supposedly healthy disk. This is another reason for a RAID6. Not only does it recover when one or two disks fail. It also recognizes and recovers broken blocks when one disk returns the wrong data. You scrub the disk weekly and remap broken sectors or swap out (soon to break) harddrives.

by eck 12y ago

ghshephard 12y ago

I'm interested - do you have a citation for that? I'm wondering if manufacturers of large drives accommodate for the statistically increased chance of a cover-cover failure (based on having so much data) by increasing their redundancy data/checksums to keep it constant.

ryan-c 12y ago
That is based off of the BER (bit error rate) of the drive, as stated by the manufacturer. It's usually spec'd at 1e-14 or 1e-15 - that is to say one in every hundred trillion or one in every quadrillion bits. This works out to 12.5TB or 125TB respectively. http://www.enterprisestorageforum.com/storage-hardware/selecting-a-disk-drive-how-not-to-do-research-1.html http://www.enterprisestorageforum.com/storage-hardware/selec...
cmurf 12y ago
The problem is due to a common misconfiguration. Consumer drives have longer recoveries potentially, upwards of minutes, trying to recover a flaky sector. The OS sees this as an unresponsive drive, and eventually does a link reset. This is 30 seconds by default on Linux, not sure about others. The link reset prevents the drive from reporting an explicit read error along with the affected sector LBA. That information is needed for RAID to know what data to rebuild from parity, send that up to the app layer, and also send a good copy back to the sector that reported the read error. So eventually there's an accumulation of these, and in case of a drive failure and another drive that produces a URE, poof, imploded array. Now, you can recover from this, sorta, but it's tedious and requires a sort of skill to do it. So most people give up. Ergo RAID is not a backup. Backup your RAIDs. And make sure drive SCT ERC is shorter than the kernel's SCSI/ATA command timer. Ideally shorten the drive timeout. If that can't be done (consumer drive) then increase the kernel's command timer. Both of these are per device settings.
rasz_pl 12y ago
fun fact: Commodore Amiga floppies used 32bit XOR checksum = 3% of undetected two bit errors!! http://www.techtravels.org/amiga/amigablog/?p=280 http://www.techtravels.org/amiga/amigablog/?p=280 This is probably because Amiga didnt have real hardware floppy-disk controller, just a general IO (CIA) chips, and read raw serial datastream into ram, all the decoding was done in software. Similar to Apple II 2x 8bit XOR checksum.

static_noise 12y ago

> Well there's one issue that another whole drive will fail and you're screwed. You do know that in a RAID6 two drives may fail without causing data loss? In any way you need one or more hotspares ready plugged in in order to keep the time window short. RAID6 is not perfect but the probability of 3 drives failing during the rebuilt window is much lower than the probability of 2 drives failing. (One also has to consider errors such as memory corruption, chip failures or catastrophic failure to the power supply where no RAID level will protect you from.) A RAID1 built of two RAID6s may be necessary to avoid performance drops during rebuild. In the case of multiple failures a RAID6+1 setup will protect you from at least 4 hard drives failing in the rebuild time window. > The other issue is that modern disks have an unrecoverable read error rate compared to their size such that a total cover-to-cover read -- necessary on every remaining disk to rebuild a RAID5 -- is kinda unreliable, even with a supposedly healthy disk. This is another reason for a RAID6. Not only does it recover when one or two disks fail. It also recognizes and recovers broken blocks when one disk returns the wrong data. You scrub the disk weekly and remap broken sectors or swap out (soon to break) harddrives.