3 ms·

> This will saturate the replacement disk with writes while scattering seeks over the rest of the vdev. For 14 wide RAID-Z2 vdevs using 12TB spindles, rebuilds

> This will saturate the replacement disk with writes while scattering seeks over the rest of the vdev. For 14 wide RAID-Z2 vdevs using 12TB spindles, rebuilds can take weeks. Didn't sequential resilver already solve the read half of the problem? Because that would already cut it down from weeks.

magicalhippo 6y ago
When replacing a disk with a hotspare in a regular RAID-Z vdev, the process is limited to the sequential write speed of that single new disk. The point with DRAID is that the spare capacity is distributed across all the disks in the vdev, hence the name. Thus all disks are read from and written to when "replacing" a failed disk with the spare capacity. The combined throughput of the multiple disks is typically much higher than that of the single disk. In addition they have an additional constraint which allows DRAID to also support sequential resilver. At least that's my understanding. There's been several presentations of DRAID on the OpenZFS channel, but the latest one is here[1]. There's also a writeup here[2], though not sure if it's 100% up to date, though should be close enough for getting the concept. [1]: https://www.youtube.com/watch?v=jdXOtEF6Fh0 https://www.youtube.com/watch?v=jdXOtEF6Fh0 [2]: https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAI...
- the8472 6y ago
  What I was questioning was that it shouldn't be taking weeks because the write speed of a single disk is enough to fill it in a day or two with sequential IO. I.e. the sequential resilver alone should be enough to fix the "can take weeks" problem. Additional speed on top of that is nice of course, if you're using such wide pools.
  - magicalhippo 6y ago
    With DRAID you'd cut down the time to an hour or so though. Not sure how much that matters in practice, I've just got a home NAS with 6 disks to play with.
  - e12e 6y ago
    I get that using hot virtual spares is much better, I also wonder at a week to write 12tb. 12tb at 200 mb/s is roughly 20 hours. Is it the fact that you get 12tb of essentially non sequencial io mixed with random reads during the rebuild? Still 20 mb/s seems 1990s slow writing tio a mechanical HD? Ed: ok, ok, from tfa: > For 14 wide RAID-Z2 vdevs using 12TB spindles, rebuilds can take weeks. Resilver I/O activity is deprioritized when the system has not been idle for a minimum period. Full zpools get fragmented and require additional I/O’s to recalculate data during reslivering. Obviously writing 12 tb at 0 mb/s will take time..
- FeepingCreature 6y ago
  That kinda makes no sense though, because the whole replacement disk needs to be written anyway. Presumably before the failure you were using some part of the old disk, so you need to fill up the new disk to the same level before you're in the same state.
  - mbreese 6y ago
    I’m thinking of this in terms of hot spares, not cold spares. If you insert a cold spare to replace a disk — yes, you have to write the full disk. If you migrate a disk from a hot spare to an active disk, it’s already largely populated, so you only have to write the missing chunk. At least, that’s my limited understanding/reading. I’m not sure I fully grok what is happening as there aren’t necessarily dedicated spares in this setup. But I think the difference in your scenario is hot vs cold spares.
  - rincebrain 6y ago
    The primary benefit of dRAID is when you're doing a replace with a virtual "distributed hot spare", where it uses spare capacity distributed across all the disks in the vdev to replace a failed disk, thereby letting the rebuild scale with the total number of disks in the vdev. When you do a rebuild with a real, distinct physical disk, there's no special benefit, you're still almost certainly going to bottleneck on the write speed of that single disk.
    - jsmith45 6y ago
      While I can see the hot-spare advantage here, I do have to wonder about the impact of replacing the broken drive. I suppose it works out as follows. With traditional raid, you are rebuilding while degraded from a safety perspective. So obviously, if your configuration guarantees losing `n` drives while healthy does not cause data loss, you could lose data during rebuild if `n` drives failed. With the distributed hot spare approach, you are no longer degraded from a safety perspective as soon as the distributed spare's data is written. This is faster than traditional raid, as the writes are distributed among many disks. So time-wise you get out of data safety degraded state faster. (It does seem like it might be riskier though, since you are doing heavy writes to many drives while safety degraded, instead of only writes to the spare). So now you replace the failed drive, and the array rebuilds to make this new spare's capacity distributed. This will take a long time, since you will be writing to basically this whole drive as part of distributing its spare space. (Plus probably shuffling chunks around between many other drives). This could possibly take even longer than traditional raid rebuild. However during this whole process, your array is not in degraded state from a data safety point of view. Or to summarize: during the long single-drive-write-limited part of replacing the failed drive with a new hot spare, you are not degraded from a safety of data point of view.

magicalhippo 6y ago

When replacing a disk with a hotspare in a regular RAID-Z vdev, the process is limited to the sequential write speed of that single new disk. The point with DRAID is that the spare capacity is distributed across all the disks in the vdev, hence the name. Thus all disks are read from and written to when "replacing" a failed disk with the spare capacity. The combined throughput of the multiple disks is typically much higher than that of the single disk. In addition they have an additional constraint which allows DRAID to also support sequential resilver. At least that's my understanding. There's been several presentations of DRAID on the OpenZFS channel, but the latest one is here[1]. There's also a writeup here[2], though not sure if it's 100% up to date, though should be close enough for getting the concept. [1]: https://www.youtube.com/watch?v=jdXOtEF6Fh0 https://www.youtube.com/watch?v=jdXOtEF6Fh0 [2]: https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAI...

the8472 6y ago
What I was questioning was that it shouldn't be taking weeks because the write speed of a single disk is enough to fill it in a day or two with sequential IO. I.e. the sequential resilver alone should be enough to fix the "can take weeks" problem. Additional speed on top of that is nice of course, if you're using such wide pools.
- magicalhippo 6y ago
  With DRAID you'd cut down the time to an hour or so though. Not sure how much that matters in practice, I've just got a home NAS with 6 disks to play with.
- e12e 6y ago
  I get that using hot virtual spares is much better, I also wonder at a week to write 12tb. 12tb at 200 mb/s is roughly 20 hours. Is it the fact that you get 12tb of essentially non sequencial io mixed with random reads during the rebuild? Still 20 mb/s seems 1990s slow writing tio a mechanical HD? Ed: ok, ok, from tfa: > For 14 wide RAID-Z2 vdevs using 12TB spindles, rebuilds can take weeks. Resilver I/O activity is deprioritized when the system has not been idle for a minimum period. Full zpools get fragmented and require additional I/O’s to recalculate data during reslivering. Obviously writing 12 tb at 0 mb/s will take time..
FeepingCreature 6y ago
That kinda makes no sense though, because the whole replacement disk needs to be written anyway. Presumably before the failure you were using some part of the old disk, so you need to fill up the new disk to the same level before you're in the same state.
- mbreese 6y ago
  I’m thinking of this in terms of hot spares, not cold spares. If you insert a cold spare to replace a disk — yes, you have to write the full disk. If you migrate a disk from a hot spare to an active disk, it’s already largely populated, so you only have to write the missing chunk. At least, that’s my limited understanding/reading. I’m not sure I fully grok what is happening as there aren’t necessarily dedicated spares in this setup. But I think the difference in your scenario is hot vs cold spares.
- rincebrain 6y ago
  The primary benefit of dRAID is when you're doing a replace with a virtual "distributed hot spare", where it uses spare capacity distributed across all the disks in the vdev to replace a failed disk, thereby letting the rebuild scale with the total number of disks in the vdev. When you do a rebuild with a real, distinct physical disk, there's no special benefit, you're still almost certainly going to bottleneck on the write speed of that single disk.
  - jsmith45 6y ago
    While I can see the hot-spare advantage here, I do have to wonder about the impact of replacing the broken drive. I suppose it works out as follows. With traditional raid, you are rebuilding while degraded from a safety perspective. So obviously, if your configuration guarantees losing `n` drives while healthy does not cause data loss, you could lose data during rebuild if `n` drives failed. With the distributed hot spare approach, you are no longer degraded from a safety perspective as soon as the distributed spare's data is written. This is faster than traditional raid, as the writes are distributed among many disks. So time-wise you get out of data safety degraded state faster. (It does seem like it might be riskier though, since you are doing heavy writes to many drives while safety degraded, instead of only writes to the spare). So now you replace the failed drive, and the array rebuilds to make this new spare's capacity distributed. This will take a long time, since you will be writing to basically this whole drive as part of distributing its spare space. (Plus probably shuffling chunks around between many other drives). This could possibly take even longer than traditional raid rebuild. However during this whole process, your array is not in degraded state from a data safety point of view. Or to summarize: during the long single-drive-write-limited part of replacing the failed drive with a new hot spare, you are not degraded from a safety of data point of view.