3 ms·

> Why would the size of the array make any difference as to whether or not the extra drives needed by RAID6 can be characterized as waste? RAID risk is about y

by zaarn 8y ago

> Why would the size of the array make any difference as to whether or not the extra drives needed by RAID6 can be characterized as waste? RAID risk is about your rebuild failure rate, in this case URE (Unrecoverable Read Error). If one such is likely to occur during your rebuild, that can lead to a lot of problems. In RAID5 until you hit about 2 TB disks and up to about 4 or 5 disks total the risk of a URE during the full read of the array is fairly low. Once you go above that you risk loosing data to URE errors. In RAID6 you essentially multiply the URE rate together, which means you have a much much lower error floor and you can repair bigger arrays. In a cost-benefit analysis this means that if your array is small enough, a RAID5 gives you more effective disk space with little additional risk. >If you have node-level redundancy, why bother with the intra-node redundancy? A failover is still a failover and can reduce performance and it reduces your remaining failover margin. You want to keep your failover rate low, though if you doN't particularly care you can use RAID 0 too. >It's not clear to me what limit you're referring to, but, considering how long RAID6 has been implemented, especially in hardware, and how far computing power has increased since then, such a claim is dubious, if not extraordinary. This is not a CPU power limit, rather RAID6 will in the next few years hit the spot where the risk of loosing 2 drives during a rebuild, due to the large drive sizes, becomes large (URE is still low IIRC), so at that point you want more redundancy to keep the rebuild risk low.

mmt 8y ago
> RAID risk is about your rebuild failure rate, in this case URE (Unrecoverable Read Error). This seems to be the keystone to your reasoning, and I'm not sure it's true, in practice. IME and in the literature I've read, modern RAIDs operate assuming a whole-disk failure (both on write and on read). Sub-whole-device errors seem to be an issue of concern limited to ECC in RAM. Do you have pointers to literature about this in the context of RAID? > A failover is still a failover and can reduce performance and it reduces your remaining failover margin. I think that means that the node-level redundancy isn't actually equivalent to RAID1-style redundancy (where there's no explicit failover), which is important to mention. Of course, my question wasn't really "why bother" so much as asking if that architecture isn't far more wasteful than, say, RAID1 (or even just RAID6/RAIDZ3) intra-node and less node-level redundancy. > You want to keep your failover rate low I'm pretty sure best practices are the opposite of this. One can only know if the redundant copy actually works by using it, so a system that can failover more often is better. Ideally, the system is, like RAID1, requires no explicit failover and merely load balances both copies. > RAID6 will in the next few years hit the spot where the risk of loosing 2 drives during a rebuild, due to the large drive sizes, becomes large You'll need to quantify these, though, since, though drive capacities have continued to grow faster than transfer speeds (for mechanical drives), it's still in the OOM of 30 hours for minimum rebuild time. What's the risk of a second and third drive failure in 300 hours (1/10th the maximum rebuild rate)?
- zaarn 8y ago
  >Do you have pointers to literature about this in the context of RAID? URE Rate is documented in most handbooks or manuals on harddisks, it's usually about 10^-12, though this is a worst case error rate and -13 or -14 is more realistic. At that rate you means your harddrive will return a read error every terabyte to every hundred terabytes. A URE can cause the raid controller to think the HDD is defective and crash the array if no parity is left to compensate. >Of course, my question wasn't really "why bother" so much as asking if that architecture isn't far more wasteful than, say, RAID1 (or even just RAID6/RAIDZ3) intra-node and less node-level redundancy. Depends on your use case. >I'm pretty sure best practices are the opposite of this. Best practise is to test your failover regularly, correct, but this doesn't mean you shouldn't minimize your actual failover rate. >What's the risk of a second and third drive failure in 300 hours (1/10th the maximum rebuild rate)? It's not that low, especially since a lot of arrays contain similar enough harddrives that concurrent failure is possible (although you can make it less likely by avoiding similar batches). And again, that URE rate has a risk of marking of your array as dead if you get unlucky.
  - mmt 8y ago
    > URE Rate is documented in most handbooks or manuals on harddisks, it's usually about 10^-12, though this is a worst case error rate and -13 or -14 is more realistic. The kind of literature I'm looking for is based on real world data, since spec sheets are generally marketing documents and therefore, at best, uninformative. This is particularly noticeable when a single number is used for a statistic, as in this situation. Let's take a current datasheet [1] which actually says 1 sector (unclear if that's 512 or 4k, but let's assume 512) per 10E15, max, a full 3 OOMs higher than what you mentioned. That's 512PB, which, at the highest listed transfer rate, would take almost 545k hours, corresponding to slightly above a 1.6% AFR for worst-case URE-caused failures. Since I don't believe the spec sheet's 0.35% (average) AFR, but since real-world AFRs are in the low single-percent, I'm staying with the conclusion that read errors need not be considered separately from any other underlying cause of drive failure. > Depends on your use case. I'm not convinced that "waste" can ever depend on use case, when examined under the narrow lense of redundancy. Regardless, it was a use case you, specifically brought up, so the question remains open. > Best practise is to test your failover regularly, correct You're misunderstanding my assertion, which wasn't about regularity but frequency. I made no statement as to regularity (and it may even be better, with discrete failovers, to do so irregularly, rather than regularly). My assertion is that best practice dictates frequent failover. > It's not that low > risk of marking of your array as dead if you get unlucky. You still haven't actually quantified the risk. "Unlucky" and "possible" are too hand-wavy to engineer around. Quantifying the risk, even if it's a very coarse approximation, is a prerequisite to avoiding FUD-based decisions/actions. Usually, the most visible consequence of the latter is the waste alluded to earlier, manifesting as higher cost and/or lower performance. The less visible consequence is misallocated resources/attention from something relatively likely (e.g. human error causing catastrophic data loss) to something much less likely (e.g. single sector read error causing catastrophic data loss). [1] https://www.seagate.com/www-content/datasheets/pdfs/exos-x-12-DS1946-2-1712US-en_US.pdf https://www.seagate.com/www-content/datasheets/pdfs/exos-x-1...

by zaarn 8y ago

mmt 8y ago

> RAID risk is about your rebuild failure rate, in this case URE (Unrecoverable Read Error). This seems to be the keystone to your reasoning, and I'm not sure it's true, in practice. IME and in the literature I've read, modern RAIDs operate assuming a whole-disk failure (both on write and on read). Sub-whole-device errors seem to be an issue of concern limited to ECC in RAM. Do you have pointers to literature about this in the context of RAID? > A failover is still a failover and can reduce performance and it reduces your remaining failover margin. I think that means that the node-level redundancy isn't actually equivalent to RAID1-style redundancy (where there's no explicit failover), which is important to mention. Of course, my question wasn't really "why bother" so much as asking if that architecture isn't far more wasteful than, say, RAID1 (or even just RAID6/RAIDZ3) intra-node and less node-level redundancy. > You want to keep your failover rate low I'm pretty sure best practices are the opposite of this. One can only know if the redundant copy actually works by using it, so a system that can failover more often is better. Ideally, the system is, like RAID1, requires no explicit failover and merely load balances both copies. > RAID6 will in the next few years hit the spot where the risk of loosing 2 drives during a rebuild, due to the large drive sizes, becomes large You'll need to quantify these, though, since, though drive capacities have continued to grow faster than transfer speeds (for mechanical drives), it's still in the OOM of 30 hours for minimum rebuild time. What's the risk of a second and third drive failure in 300 hours (1/10th the maximum rebuild rate)?

zaarn 8y ago
>Do you have pointers to literature about this in the context of RAID? URE Rate is documented in most handbooks or manuals on harddisks, it's usually about 10^-12, though this is a worst case error rate and -13 or -14 is more realistic. At that rate you means your harddrive will return a read error every terabyte to every hundred terabytes. A URE can cause the raid controller to think the HDD is defective and crash the array if no parity is left to compensate. >Of course, my question wasn't really "why bother" so much as asking if that architecture isn't far more wasteful than, say, RAID1 (or even just RAID6/RAIDZ3) intra-node and less node-level redundancy. Depends on your use case. >I'm pretty sure best practices are the opposite of this. Best practise is to test your failover regularly, correct, but this doesn't mean you shouldn't minimize your actual failover rate. >What's the risk of a second and third drive failure in 300 hours (1/10th the maximum rebuild rate)? It's not that low, especially since a lot of arrays contain similar enough harddrives that concurrent failure is possible (although you can make it less likely by avoiding similar batches). And again, that URE rate has a risk of marking of your array as dead if you get unlucky.
- mmt 8y ago
  > URE Rate is documented in most handbooks or manuals on harddisks, it's usually about 10^-12, though this is a worst case error rate and -13 or -14 is more realistic. The kind of literature I'm looking for is based on real world data, since spec sheets are generally marketing documents and therefore, at best, uninformative. This is particularly noticeable when a single number is used for a statistic, as in this situation. Let's take a current datasheet [1] which actually says 1 sector (unclear if that's 512 or 4k, but let's assume 512) per 10E15, max, a full 3 OOMs higher than what you mentioned. That's 512PB, which, at the highest listed transfer rate, would take almost 545k hours, corresponding to slightly above a 1.6% AFR for worst-case URE-caused failures. Since I don't believe the spec sheet's 0.35% (average) AFR, but since real-world AFRs are in the low single-percent, I'm staying with the conclusion that read errors need not be considered separately from any other underlying cause of drive failure. > Depends on your use case. I'm not convinced that "waste" can ever depend on use case, when examined under the narrow lense of redundancy. Regardless, it was a use case you, specifically brought up, so the question remains open. > Best practise is to test your failover regularly, correct You're misunderstanding my assertion, which wasn't about regularity but frequency. I made no statement as to regularity (and it may even be better, with discrete failovers, to do so irregularly, rather than regularly). My assertion is that best practice dictates frequent failover. > It's not that low > risk of marking of your array as dead if you get unlucky. You still haven't actually quantified the risk. "Unlucky" and "possible" are too hand-wavy to engineer around. Quantifying the risk, even if it's a very coarse approximation, is a prerequisite to avoiding FUD-based decisions/actions. Usually, the most visible consequence of the latter is the waste alluded to earlier, manifesting as higher cost and/or lower performance. The less visible consequence is misallocated resources/attention from something relatively likely (e.g. human error causing catastrophic data loss) to something much less likely (e.g. single sector read error causing catastrophic data loss). [1] https://www.seagate.com/www-content/datasheets/pdfs/exos-x-12-DS1946-2-1712US-en_US.pdf https://www.seagate.com/www-content/datasheets/pdfs/exos-x-1...