3 ms·

While I can see the hot-spare advantage here, I do have to wonder about the impact of replacing the broken drive. I suppose it works out as follows. With tradi

While I can see the hot-spare advantage here, I do have to wonder about the impact of replacing the broken drive. I suppose it works out as follows. With traditional raid, you are rebuilding while degraded from a safety perspective. So obviously, if your configuration guarantees losing `n` drives while healthy does not cause data loss, you could lose data during rebuild if `n` drives failed. With the distributed hot spare approach, you are no longer degraded from a safety perspective as soon as the distributed spare's data is written. This is faster than traditional raid, as the writes are distributed among many disks. So time-wise you get out of data safety degraded state faster. (It does seem like it might be riskier though, since you are doing heavy writes to many drives while safety degraded, instead of only writes to the spare). So now you replace the failed drive, and the array rebuilds to make this new spare's capacity distributed. This will take a long time, since you will be writing to basically this whole drive as part of distributing its spare space. (Plus probably shuffling chunks around between many other drives). This could possibly take even longer than traditional raid rebuild. However during this whole process, your array is not in degraded state from a data safety point of view. Or to summarize: during the long single-drive-write-limited part of replacing the failed drive with a new hot spare, you are not degraded from a safety of data point of view.

mbreese 6y ago
> This will take a long time, since you will be writing to basically this whole drive as part of distributing its spare space. (Plus probably shuffling chunks around between many other drives) Yes. With the caveat that you'll be *reading* from all of the disks to satisfy those writes. In a traditional setup, you'd have multiple striped raidz{,2,3} vdevs. When you swap in a spare, you're only pulling reads from the disks in the same vdev. This will drive the performance from this vdev to the ground. In this draidz setup, you'll be pulling from all of the disks, so the entire pool is stressed at the same rate. I think the way I'm starting to think about it is that each drive has X amount of spare capacity or buckets. But they are otherwise used. When a drive goes out, the primary data from that drive is distributed across the array to the already-in-place spare capacity. So, yes, after the primary data is mirrored out, the pool is no longer degraded. But, I think the point being made here is that you'll still want to replace the drive, which will still cause *that* drive to be fully written.
- jsmith45 6y ago
  Right, the free space buckets are a fine analogy. They are conceptually similar in some ways to the free space in the hot standby drive. I don't think from a data safety or performance perspective pre dRAID vdev really differ too terribly much from raid 50 setups. (They do differ greatly in administrative flexibility, versus the insanely rigid approachs of traditional hardware raid). But yeah, after replacing the failed drive, ideally dRAID would basically do just what it did after drive failure in reverse. (End result: No longer having any drives holding data for a different vdev than they belong to). Obviously this means writing basically that whole disk, and reading from potentially every disk. But as you mention you are spreading the load out across the whole array, hopefully reducing the overall impact, and being less likely to stress any drive into dying.

mbreese 6y ago

> This will take a long time, since you will be writing to basically this whole drive as part of distributing its spare space. (Plus probably shuffling chunks around between many other drives) Yes. With the caveat that you'll be *reading* from all of the disks to satisfy those writes. In a traditional setup, you'd have multiple striped raidz{,2,3} vdevs. When you swap in a spare, you're only pulling reads from the disks in the same vdev. This will drive the performance from this vdev to the ground. In this draidz setup, you'll be pulling from all of the disks, so the entire pool is stressed at the same rate. I think the way I'm starting to think about it is that each drive has X amount of spare capacity or buckets. But they are otherwise used. When a drive goes out, the primary data from that drive is distributed across the array to the already-in-place spare capacity. So, yes, after the primary data is mirrored out, the pool is no longer degraded. But, I think the point being made here is that you'll still want to replace the drive, which will still cause *that* drive to be fully written.

jsmith45 6y ago
Right, the free space buckets are a fine analogy. They are conceptually similar in some ways to the free space in the hot standby drive. I don't think from a data safety or performance perspective pre dRAID vdev really differ too terribly much from raid 50 setups. (They do differ greatly in administrative flexibility, versus the insanely rigid approachs of traditional hardware raid). But yeah, after replacing the failed drive, ideally dRAID would basically do just what it did after drive failure in reverse. (End result: No longer having any drives holding data for a different vdev than they belong to). Obviously this means writing basically that whole disk, and reading from potentially every disk. But as you mention you are spreading the load out across the whole array, hopefully reducing the overall impact, and being less likely to stress any drive into dying.