3 ms·

I'm interested - do you have a citation for that? I'm wondering if manufacturers of large drives accommodate for the statistically increased chance of a cover-

I'm interested - do you have a citation for that? I'm wondering if manufacturers of large drives accommodate for the statistically increased chance of a cover-cover failure (based on having so much data) by increasing their redundancy data/checksums to keep it constant.

ryan-c 12y ago
That is based off of the BER (bit error rate) of the drive, as stated by the manufacturer. It's usually spec'd at 1e-14 or 1e-15 - that is to say one in every hundred trillion or one in every quadrillion bits. This works out to 12.5TB or 125TB respectively. http://www.enterprisestorageforum.com/storage-hardware/selecting-a-disk-drive-how-not-to-do-research-1.html http://www.enterprisestorageforum.com/storage-hardware/selec...
cmurf 12y ago
The problem is due to a common misconfiguration. Consumer drives have longer recoveries potentially, upwards of minutes, trying to recover a flaky sector. The OS sees this as an unresponsive drive, and eventually does a link reset. This is 30 seconds by default on Linux, not sure about others. The link reset prevents the drive from reporting an explicit read error along with the affected sector LBA. That information is needed for RAID to know what data to rebuild from parity, send that up to the app layer, and also send a good copy back to the sector that reported the read error. So eventually there's an accumulation of these, and in case of a drive failure and another drive that produces a URE, poof, imploded array. Now, you can recover from this, sorta, but it's tedious and requires a sort of skill to do it. So most people give up. Ergo RAID is not a backup. Backup your RAIDs. And make sure drive SCT ERC is shorter than the kernel's SCSI/ATA command timer. Ideally shorten the drive timeout. If that can't be done (consumer drive) then increase the kernel's command timer. Both of these are per device settings.
rasz_pl 12y ago
fun fact: Commodore Amiga floppies used 32bit XOR checksum = 3% of undetected two bit errors!! http://www.techtravels.org/amiga/amigablog/?p=280 http://www.techtravels.org/amiga/amigablog/?p=280 This is probably because Amiga didnt have real hardware floppy-disk controller, just a general IO (CIA) chips, and read raw serial datastream into ram, all the decoding was done in software. Similar to Apple II 2x 8bit XOR checksum.

ryan-c 12y ago

That is based off of the BER (bit error rate) of the drive, as stated by the manufacturer. It's usually spec'd at 1e-14 or 1e-15 - that is to say one in every hundred trillion or one in every quadrillion bits. This works out to 12.5TB or 125TB respectively. http://www.enterprisestorageforum.com/storage-hardware/selecting-a-disk-drive-how-not-to-do-research-1.html http://www.enterprisestorageforum.com/storage-hardware/selec...

cmurf 12y ago

The problem is due to a common misconfiguration. Consumer drives have longer recoveries potentially, upwards of minutes, trying to recover a flaky sector. The OS sees this as an unresponsive drive, and eventually does a link reset. This is 30 seconds by default on Linux, not sure about others. The link reset prevents the drive from reporting an explicit read error along with the affected sector LBA. That information is needed for RAID to know what data to rebuild from parity, send that up to the app layer, and also send a good copy back to the sector that reported the read error. So eventually there's an accumulation of these, and in case of a drive failure and another drive that produces a URE, poof, imploded array. Now, you can recover from this, sorta, but it's tedious and requires a sort of skill to do it. So most people give up. Ergo RAID is not a backup. Backup your RAIDs. And make sure drive SCT ERC is shorter than the kernel's SCSI/ATA command timer. Ideally shorten the drive timeout. If that can't be done (consumer drive) then increase the kernel's command timer. Both of these are per device settings.

rasz_pl 12y ago

fun fact: Commodore Amiga floppies used 32bit XOR checksum = 3% of undetected two bit errors!! http://www.techtravels.org/amiga/amigablog/?p=280 http://www.techtravels.org/amiga/amigablog/?p=280 This is probably because Amiga didnt have real hardware floppy-disk controller, just a general IO (CIA) chips, and read raw serial datastream into ram, all the decoding was done in software. Similar to Apple II 2x 8bit XOR checksum.