3 ms·

the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Two disk failures in the same array within 15 hours is very rare.

by moe 12y ago

ghshephard 12y ago
Unless the root cause is something to do with the array - flakey controller, or power supply.
- craigyk 12y ago
  these kinds of problems may cause the array to fail, but won't necessarily lose data, right? If you have a ZFS pool and a faulty controller causes the pool to fail, ie. "losing" three disks in a raidz2 vdev, my guess is that you would just replace the faulty card and reimport the pool and chances are things would be OK... no?
  - DiabloD3 12y ago
    ZFS can mostly handle that. It will try to recover any data it can, and rebuild whatever it can. Due to how paranoid ZFS is designed, it will survive edge cases like this much better, which is also why I like deploying it here even at home. And if it can't recover it? You can prove it by failed checksum check. This is why, in my opinion, ZFS is so much better than all other file systems: if it fails, you can prove it instead of just sorta vaguely questioning everything because you can't tell but you suspect it, and it slowly drives you insane. Once you try ZFS, you never go back.
vidarh 12y ago
Had it happen 3 times within a one month window (as in 3x double drive failures in the same array too close together for a rebuild to complete). It may be rare, but manage enough servers and sooner or later these things happens, especially if the cause of the failure is a power supply or the controller. (We didn't lose any data - everything is replicated elsewhere in addition to the raid)
letstryagain 12y ago
At scale everything happens, even very rare events.
DiabloD3 12y ago
I have to say this is false. I work in that industry, and even when you have, say, a 4 drive RAID, and all 4 drives are the same model but intentionally different batches, you will still, more often than not, have 2 drive failures within 24 hours. Now, what does this mean in, exactly? In that four drive scenario, RAID10 has a 1/3rd chance of failure (both failed drives being both the A or B parts of the inner RAID 0s is two out of six possible scenarios), RAID5 will fail no matter which two drives, RAID6 will survive no matter which two drives, ZFS's RAID-Z3 (three parity RAID5/6) will survive no matter which three drives, and RAID1 will survive no matter which three drives. RAID10 is the most performant of the possible outcomes, and the usual makeup of a 4 drive array unless you absolutely need the storage, then its usually RAID5; unless you seriously do not care about sanity at all, then its RAID0 or just independent disks. Now, my suggestion for 15 hour rebuild times? Whatever you do, have a hotspare so it can begin immediately rebuilding and do not use any RAID variant that can't handle more than 2 failures in 15 hours. This means no RAID10 or RAID5, only RAID6 or RAID-Z3. Ceph mitigates this 15 hour time because it can rebuild a lost drive by just allocating new blocks on every other drive simultaneously and maxing out your network and/or storage IO (whichever is slowest) so the window of potential doom is much smaller (depending on Ceph cluster size, obviously, bigger the better in this case).
jacquesm 12y ago
Frequent enough that I've seen it twice on different arrays. The reason for this is that a drive failure can be caused by an external factor (usually multiple drives fed by the same broken power supply, but not yet so broken that it goes off-line).
- moe 12y ago
  I didn't say they don't happen. They're just so rare that the difference between ~6h rebuilds with 4T spindles versus 15h rebuilds with 10T spindles simply doesn't matter. Any homeopathic advantage of slightly faster rebuilds is dwarfed by the benefits of higher density (less hardware per capacity).
TheCondor 12y ago
This is true of natural age type failures. When in a RAID every drive is effectively experiencing the same experiences, down to the writes and reads, if the failure is due to the drive's interaction with its environment then one failure almost always predicts more failures. Natural age failures are pretty rare.

the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Two disk failures in the same array within 15 hours is very rare.

by moe 12y ago

the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Two disk failures in the same array within 15 hours is very rare.

ghshephard 12y ago

Unless the root cause is something to do with the array - flakey controller, or power supply.

craigyk 12y ago
these kinds of problems may cause the array to fail, but won't necessarily lose data, right? If you have a ZFS pool and a faulty controller causes the pool to fail, ie. "losing" three disks in a raidz2 vdev, my guess is that you would just replace the faulty card and reimport the pool and chances are things would be OK... no?
- DiabloD3 12y ago
  ZFS can mostly handle that. It will try to recover any data it can, and rebuild whatever it can. Due to how paranoid ZFS is designed, it will survive edge cases like this much better, which is also why I like deploying it here even at home. And if it can't recover it? You can prove it by failed checksum check. This is why, in my opinion, ZFS is so much better than all other file systems: if it fails, you can prove it instead of just sorta vaguely questioning everything because you can't tell but you suspect it, and it slowly drives you insane. Once you try ZFS, you never go back.

vidarh 12y ago

Had it happen 3 times within a one month window (as in 3x double drive failures in the same array too close together for a rebuild to complete). It may be rare, but manage enough servers and sooner or later these things happens, especially if the cause of the failure is a power supply or the controller. (We didn't lose any data - everything is replicated elsewhere in addition to the raid)

letstryagain 12y ago

At scale everything happens, even very rare events.

DiabloD3 12y ago

I have to say this is false. I work in that industry, and even when you have, say, a 4 drive RAID, and all 4 drives are the same model but intentionally different batches, you will still, more often than not, have 2 drive failures within 24 hours. Now, what does this mean in, exactly? In that four drive scenario, RAID10 has a 1/3rd chance of failure (both failed drives being both the A or B parts of the inner RAID 0s is two out of six possible scenarios), RAID5 will fail no matter which two drives, RAID6 will survive no matter which two drives, ZFS's RAID-Z3 (three parity RAID5/6) will survive no matter which three drives, and RAID1 will survive no matter which three drives. RAID10 is the most performant of the possible outcomes, and the usual makeup of a 4 drive array unless you absolutely need the storage, then its usually RAID5; unless you seriously do not care about sanity at all, then its RAID0 or just independent disks. Now, my suggestion for 15 hour rebuild times? Whatever you do, have a hotspare so it can begin immediately rebuilding and do not use any RAID variant that can't handle more than 2 failures in 15 hours. This means no RAID10 or RAID5, only RAID6 or RAID-Z3. Ceph mitigates this 15 hour time because it can rebuild a lost drive by just allocating new blocks on every other drive simultaneously and maxing out your network and/or storage IO (whichever is slowest) so the window of potential doom is much smaller (depending on Ceph cluster size, obviously, bigger the better in this case).

jacquesm 12y ago

Frequent enough that I've seen it twice on different arrays. The reason for this is that a drive failure can be caused by an external factor (usually multiple drives fed by the same broken power supply, but not yet so broken that it goes off-line).

moe 12y ago
I didn't say they don't happen. They're just so rare that the difference between ~6h rebuilds with 4T spindles versus 15h rebuilds with 10T spindles simply doesn't matter. Any homeopathic advantage of slightly faster rebuilds is dwarfed by the benefits of higher density (less hardware per capacity).

TheCondor 12y ago

This is true of natural age type failures. When in a RAID every drive is effectively experiencing the same experiences, down to the writes and reads, if the failure is due to the drive's interaction with its environment then one failure almost always predicts more failures. Natural age failures are pretty rare.