39 ms·

....vs stressing all of them for parity rebuild.

by ilyt 3y ago

edmundsauto 3y ago
Wouldn't it be better to apply stress over more drives to minimize the chances that you lose a second drive during a rebuild?
- ErneX 3y ago
  I believe this is the article I read when I started: https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...
  - ilyt 3y ago
    That talks more about RAID5/Z1 vs mirroring. You still need 6/Z2 if you want to have reasonable fault tolerance, unless you want to waste ungodly amount of space.
    - ErneX 3y ago
      My reasoning for using mirrored vdevs was more about the ease of expanding the storage than the redundancy.
      - ilyt 3y ago
        We just used RAID6... expanding by more drives is easy while keeping data waste low and the "we want to put bigger drives now" case isn't all that problematic considering that what you save you waste on low efficiency of RAID1 setup. We did had a big case so we just ran 2xRAID6 setup and that one time where it was needed we just replaced to bigger drives one by one, while using the removed ones as spares for other machines. But that's benefit of scale bigger than "a NAS server".
        ErneX 3y ago
        Yeah mine is a home NAS, not a professional deployment. At first I was a bit disappointed at losing half the storage due to using mirrors but honestly it’s not a big deal considering the prices I managed to find on the drives. And I also wanted to expand at my own pace. The drive I recently used to replace the dead 4TB drive cost me 110€ and it’s a 10TB drive. I got 2 so I’ll also replace the sibling of the dead 4TB one.
  - labawi 3y ago
    Note that the listed probabilities of surviving N-disk failures are not correct, though after calculation, though the differences may not be that important. Listed survival probability of N-disk failure in 8-drive/4-vdev mirror. 1: 1; 2: 0.857; 3: 0.667; 4: 0.400; 5: 0; 6: N/A; 7: N/A; 8: N/A Proper survival probability: 1: 1; 2: 0.857; 3: 0.571; 4: 0.229; 5: 0; 6: 0; 7: 0; 8: 0 Comparatively, survival probability for 8-drive RAIDz4 with equal amount of usable space: 1: 1; 2: 1; 3: 1; 4: 1; 5: 0; 6: 0; 7: 0; 8: 0 Personally, I'd use multi-vdev mirror pools only for data that is either backed-up or data I can afford to lose completely.
- ilyt 3y ago
  To replace drive in RAID5/6 array, you need to read the entirety of every drive To replace drive in RAID1 you need to read the entirety of one drive. Instead of reading one whole drive worth of stress, you're reading N drives worth of stress But it is funny that the people making arguments about "stressing drives" don't even fucking know how RAID works...
  - edmundsauto 3y ago
    However, RAID1, you stress the drive that has your only copy of the data. With 5/6 (RAIDz w/ 2 spares as well) you can have multiple copies of the data, so although you are overall increasing stress on your fleet, wouldn't it have a lower probability of data loss? Drive stress (for me) is mostly a concern about data loss.
    - ilyt 3y ago
      With RAID5 you'd have higher probability of data loss. Remember, you just loss the only redundancy you had, brining your redundancy to same level as RAID1 with failed drive. ... except now you have more drives that can fail. With RAID6 yes, you can still fail one more time and still keep your data, which is why it is recommended. Data safety wise I'd go RAID6 -> RAID1/10 (particularly linux implementation can do raid10 on odd number of drives which is nice) -> RAID5
didntcheck 3y ago
However RAIDZ2 can survive up to one of those stressed disks failing, while a 2-mirror has everything riding on that single survivor. It seems intuitive that mirrors would be safer, yeah, but if you run the binomial distribution numbers there are some realistic combinations of array size and individual drive failure probabilities where RAIDZ2 really does appear to be safer Imagine a gambler telling you "you can either throw one k-faced die and you lose if it comes up on 1, or throw n k-faced dice and you lose if two come up on 1". Depending on k and n, the second really can be the better choice Often people cite the faster rebuild of mirrors as a safety advantage, but the same amount of rebuild IO will occur regardless of how long it takes. Yes, there will be more non-rebuild IO in a larger time window, but unless that routine load was causing disks to fail weekly then I doubt it will change the numbers non-negligibly. It will of course affect array performance though, so mirrors for performance is a good argument
- labawi 3y ago
  If we were to only compare survival after a fixed number of drives failed vs. storage efficiency, RAIDzN should always come out ahead in any configuration - with mirrors you can get unlucky drives choice fail, with RAIDzN any choice is a good choice. Only way to have RAIDz fail sooner than mirror is to have a comparatively less redundant setup (your choice of N and K). Realistically though, RAIDz recovery is longer and more stressful, so more of your drives can fail in the critical period, and, assuming you have backups, your storage is there for for usability - mirroring gives you a performant usable system during a fast recovery for the price of a small chance of complete data loss (but you have backups?) vs RAIDz that gives you long recovery pains on a degraded system, but I expect a smaller chance of data loss on a lightly loaded system.
- ilyt 3y ago
  Well, from my probe of 500 servers I have at work I have seen RAID1 rebuild failing zero times and RAID6 failing once (but we did recover it). Granted, that was a bit of uncommon case where: * someone forgot to order spares after taking last one * we still had our consumable buying pipeline going thru helpdesk * helpdesk didn't had any importance communicated about it and because of some accounting bullshit the purchase got delayed long enough * the drives in question were all from some segate's fuckup of a model with much higher failure rates. One disk failed with some media errors, remaining 2 got kicked out of array for same reason during resilvering We ddrescue'd the 2 on the pair of fresh ones and the bad blocks didn't land in the same place on both drives so it made full recovery. But we did learn many lessons from that..

....vs stressing all of them for parity rebuild.

by ilyt 3y ago

....vs stressing all of them for parity rebuild.

Wouldn't it be better to apply stress over more drives to minimize the chances that you lose a second drive during a rebuild?

ErneX 3y ago
I believe this is the article I read when I started: https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...
- ilyt 3y ago
  That talks more about RAID5/Z1 vs mirroring. You still need 6/Z2 if you want to have reasonable fault tolerance, unless you want to waste ungodly amount of space.
  - ErneX 3y ago
    My reasoning for using mirrored vdevs was more about the ease of expanding the storage than the redundancy.
    - ilyt 3y ago
      We just used RAID6... expanding by more drives is easy while keeping data waste low and the "we want to put bigger drives now" case isn't all that problematic considering that what you save you waste on low efficiency of RAID1 setup. We did had a big case so we just ran 2xRAID6 setup and that one time where it was needed we just replaced to bigger drives one by one, while using the removed ones as spares for other machines. But that's benefit of scale bigger than "a NAS server".
      - ErneX 3y ago
        Yeah mine is a home NAS, not a professional deployment. At first I was a bit disappointed at losing half the storage due to using mirrors but honestly it’s not a big deal considering the prices I managed to find on the drives. And I also wanted to expand at my own pace. The drive I recently used to replace the dead 4TB drive cost me 110€ and it’s a 10TB drive. I got 2 so I’ll also replace the sibling of the dead 4TB one.
- labawi 3y ago
  Note that the listed probabilities of surviving N-disk failures are not correct, though after calculation, though the differences may not be that important. Listed survival probability of N-disk failure in 8-drive/4-vdev mirror. 1: 1; 2: 0.857; 3: 0.667; 4: 0.400; 5: 0; 6: N/A; 7: N/A; 8: N/A Proper survival probability: 1: 1; 2: 0.857; 3: 0.571; 4: 0.229; 5: 0; 6: 0; 7: 0; 8: 0 Comparatively, survival probability for 8-drive RAIDz4 with equal amount of usable space: 1: 1; 2: 1; 3: 1; 4: 1; 5: 0; 6: 0; 7: 0; 8: 0 Personally, I'd use multi-vdev mirror pools only for data that is either backed-up or data I can afford to lose completely.
ilyt 3y ago
To replace drive in RAID5/6 array, you need to read the entirety of every drive To replace drive in RAID1 you need to read the entirety of one drive. Instead of reading one whole drive worth of stress, you're reading N drives worth of stress But it is funny that the people making arguments about "stressing drives" don't even fucking know how RAID works...
- edmundsauto 3y ago
  However, RAID1, you stress the drive that has your only copy of the data. With 5/6 (RAIDz w/ 2 spares as well) you can have multiple copies of the data, so although you are overall increasing stress on your fleet, wouldn't it have a lower probability of data loss? Drive stress (for me) is mostly a concern about data loss.
  - ilyt 3y ago
    With RAID5 you'd have higher probability of data loss. Remember, you just loss the only redundancy you had, brining your redundancy to same level as RAID1 with failed drive. ... except now you have more drives that can fail. With RAID6 yes, you can still fail one more time and still keep your data, which is why it is recommended. Data safety wise I'd go RAID6 -> RAID1/10 (particularly linux implementation can do raid10 on odd number of drives which is nice) -> RAID5

didntcheck 3y ago

However RAIDZ2 can survive up to one of those stressed disks failing, while a 2-mirror has everything riding on that single survivor. It seems intuitive that mirrors would be safer, yeah, but if you run the binomial distribution numbers there are some realistic combinations of array size and individual drive failure probabilities where RAIDZ2 really does appear to be safer Imagine a gambler telling you "you can either throw one k-faced die and you lose if it comes up on 1, or throw n k-faced dice and you lose if two come up on 1". Depending on k and n, the second really can be the better choice Often people cite the faster rebuild of mirrors as a safety advantage, but the same amount of rebuild IO will occur regardless of how long it takes. Yes, there will be more non-rebuild IO in a larger time window, but unless that routine load was causing disks to fail weekly then I doubt it will change the numbers non-negligibly. It will of course affect array performance though, so mirrors for performance is a good argument

labawi 3y ago
If we were to only compare survival after a fixed number of drives failed vs. storage efficiency, RAIDzN should always come out ahead in any configuration - with mirrors you can get unlucky drives choice fail, with RAIDzN any choice is a good choice. Only way to have RAIDz fail sooner than mirror is to have a comparatively less redundant setup (your choice of N and K). Realistically though, RAIDz recovery is longer and more stressful, so more of your drives can fail in the critical period, and, assuming you have backups, your storage is there for for usability - mirroring gives you a performant usable system during a fast recovery for the price of a small chance of complete data loss (but you have backups?) vs RAIDz that gives you long recovery pains on a degraded system, but I expect a smaller chance of data loss on a lightly loaded system.
ilyt 3y ago
Well, from my probe of 500 servers I have at work I have seen RAID1 rebuild failing zero times and RAID6 failing once (but we did recover it). Granted, that was a bit of uncommon case where: * someone forgot to order spares after taking last one * we still had our consumable buying pipeline going thru helpdesk * helpdesk didn't had any importance communicated about it and because of some accounting bullshit the purchase got delayed long enough * the drives in question were all from some segate's fuckup of a model with much higher failure rates. One disk failed with some media errors, remaining 2 got kicked out of array for same reason during resilvering We ddrescue'd the 2 on the pair of fresh ones and the bad blocks didn't land in the same place on both drives so it made full recovery. But we did learn many lessons from that..