4 ms·

Great write-up. Personally I'd have liked a more explicit warning for using raid 5/6 as the bigger they get, the more likely you're to lose the whole array on f

Great write-up. Personally I'd have liked a more explicit warning for using raid 5/6 as the bigger they get, the more likely you're to lose the whole array on failures because the remaining disks will be under heavy load for recovery and are likely about to fail as well. He did mention it as a side note, but it deserves more attention I think.

scandox 7y ago
I remember standing with a few people discussing a RAID failure of this kind and asking each person how it could have happened. All of them could explain the benefits of RAID 5 for the specific application but not one of them could explain its failure modes.
- jldugger 7y ago
  That's just one of many. Another good one is the violent failure mode. We had a RAID 6 go bad because a head crashed hard, hard enough that two other disks in the array also head crashed. Since the enclosure puts them pretty much lines them up, this is kinda inevitable.
  - blackflame7000 7y ago
    Wow, was that an IBM Deathstar? Two additional reasons for raid 6: 1. Fan failure on a hard disk pod causes damage to multiple drives simultaneously. 2. Many adjacent drives are connected via the same power cable which always has the potential to destroy drives in groups via over-voltage from a failing power supply.
bluegreyred 7y ago
Yes, there's also the topic of unrecoverable read errors (URE) and their effect on successful RAID rebuilds. [1][2] Most consumer drives are still sold rated as <10^14 bits read per error. That's 12.5 terabytes, so in the worst case you could end up in situations where — on average — you are unable read a full 16TB drive without an error. Needless to say this is less than optimal for rebuilding a failed RAID array. Anecdotal evidence (i.e. very low error rates during ZFS scrubs) suggests that manufacturers underrate their drives and they are much more reliable than that, but it is something to keep in mind. Fortunately drive capacity completely outpaced my needs for personal data storage in the recent years, so I am happy with JBOD or RAID1, with backups of course. [1]http://www.raidtips.com/raid5-ure.aspx http://www.raidtips.com/raid5-ure.aspx [2]https://magj.github.io/raid-failure/ https://magj.github.io/raid-failure/
- vardump 7y ago
  > Anecdotal evidence (i.e. very low error rates during ZFS scrubs) suggests that manufacturers underrate their drives and they are much more reliable than that, but it is something to keep in mind. I've seen plenty of drives returning invalid data with correct CRC over the years. On reliable server-grade Xeon + ECC hardware. Of course the vast majority of drives never do it, there's just no way to know which ones do until it happens. Firmware bugs in weird corner cases? Cosmic rays? Perhaps, but I think it's more reasonable just consider it one of those weird things that occasionally Just Happen (TM) and just need to be protected against at a higher level. All drives produced in the last 30 years or so are running ever more complicated software stacks. For example they all have features that move data at risk to safer locations without the host system requesting or even knowing about it. Their physical (like bits on spinning rust or NAND flash block) and logical (what the host sees) data representations can be completely different. Plain old CRC errors, though, are way more frequent. I feel much more comfortable about those errors, at least the drive knows the data is corrupted.
cyphar 7y ago
The other issue is that RAID (even RAID 6 with no dead disks) fundamentally cannot deal with data corruption or any of other fun bugs hardware likes to throw at you like phantom reads and writes. This is because it has no checksumming and most implementations will overwrite the parity if a mismatch is found. There's also the RAID "write hole" problem, but most implementations have worked around that since the mid-2000s. I switched to ZFS a few years ago and it's really quite amazing. In fairness, raidz does suffer from the same "resilvering takes a long time and increases the chance of subsequent failures" problem, which is one of the reasons why mirrors are more commonly used. Though unlike RAID, ZFS knows which data on the disk is actually used so can reduce resilver times pretty drastically.

scandox 7y ago

I remember standing with a few people discussing a RAID failure of this kind and asking each person how it could have happened. All of them could explain the benefits of RAID 5 for the specific application but not one of them could explain its failure modes.

jldugger 7y ago
That's just one of many. Another good one is the violent failure mode. We had a RAID 6 go bad because a head crashed hard, hard enough that two other disks in the array also head crashed. Since the enclosure puts them pretty much lines them up, this is kinda inevitable.
- blackflame7000 7y ago
  Wow, was that an IBM Deathstar? Two additional reasons for raid 6: 1. Fan failure on a hard disk pod causes damage to multiple drives simultaneously. 2. Many adjacent drives are connected via the same power cable which always has the potential to destroy drives in groups via over-voltage from a failing power supply.

bluegreyred 7y ago

Yes, there's also the topic of unrecoverable read errors (URE) and their effect on successful RAID rebuilds. [1][2] Most consumer drives are still sold rated as <10^14 bits read per error. That's 12.5 terabytes, so in the worst case you could end up in situations where — on average — you are unable read a full 16TB drive without an error. Needless to say this is less than optimal for rebuilding a failed RAID array. Anecdotal evidence (i.e. very low error rates during ZFS scrubs) suggests that manufacturers underrate their drives and they are much more reliable than that, but it is something to keep in mind. Fortunately drive capacity completely outpaced my needs for personal data storage in the recent years, so I am happy with JBOD or RAID1, with backups of course. [1]http://www.raidtips.com/raid5-ure.aspx http://www.raidtips.com/raid5-ure.aspx [2]https://magj.github.io/raid-failure/ https://magj.github.io/raid-failure/

vardump 7y ago
> Anecdotal evidence (i.e. very low error rates during ZFS scrubs) suggests that manufacturers underrate their drives and they are much more reliable than that, but it is something to keep in mind. I've seen plenty of drives returning invalid data with correct CRC over the years. On reliable server-grade Xeon + ECC hardware. Of course the vast majority of drives never do it, there's just no way to know which ones do until it happens. Firmware bugs in weird corner cases? Cosmic rays? Perhaps, but I think it's more reasonable just consider it one of those weird things that occasionally Just Happen (TM) and just need to be protected against at a higher level. All drives produced in the last 30 years or so are running ever more complicated software stacks. For example they all have features that move data at risk to safer locations without the host system requesting or even knowing about it. Their physical (like bits on spinning rust or NAND flash block) and logical (what the host sees) data representations can be completely different. Plain old CRC errors, though, are way more frequent. I feel much more comfortable about those errors, at least the drive knows the data is corrupted.

cyphar 7y ago

The other issue is that RAID (even RAID 6 with no dead disks) fundamentally cannot deal with data corruption or any of other fun bugs hardware likes to throw at you like phantom reads and writes. This is because it has no checksumming and most implementations will overwrite the parity if a mismatch is found. There's also the RAID "write hole" problem, but most implementations have worked around that since the mid-2000s. I switched to ZFS a few years ago and it's really quite amazing. In fairness, raidz does suffer from the same "resilvering takes a long time and increases the chance of subsequent failures" problem, which is one of the reasons why mirrors are more commonly used. Though unlike RAID, ZFS knows which data on the disk is actually used so can reduce resilver times pretty drastically.