4 ms·

The primary benefit of dRAID is when you're doing a replace with a virtual "distributed hot spare", where it uses spare capacity distributed across all the disk

The primary benefit of dRAID is when you're doing a replace with a virtual "distributed hot spare", where it uses spare capacity distributed across all the disks in the vdev to replace a failed disk, thereby letting the rebuild scale with the total number of disks in the vdev. When you do a rebuild with a real, distinct physical disk, there's no special benefit, you're still almost certainly going to bottleneck on the write speed of that single disk.

jsmith45 6y ago
While I can see the hot-spare advantage here, I do have to wonder about the impact of replacing the broken drive. I suppose it works out as follows. With traditional raid, you are rebuilding while degraded from a safety perspective. So obviously, if your configuration guarantees losing `n` drives while healthy does not cause data loss, you could lose data during rebuild if `n` drives failed. With the distributed hot spare approach, you are no longer degraded from a safety perspective as soon as the distributed spare's data is written. This is faster than traditional raid, as the writes are distributed among many disks. So time-wise you get out of data safety degraded state faster. (It does seem like it might be riskier though, since you are doing heavy writes to many drives while safety degraded, instead of only writes to the spare). So now you replace the failed drive, and the array rebuilds to make this new spare's capacity distributed. This will take a long time, since you will be writing to basically this whole drive as part of distributing its spare space. (Plus probably shuffling chunks around between many other drives). This could possibly take even longer than traditional raid rebuild. However during this whole process, your array is not in degraded state from a data safety point of view. Or to summarize: during the long single-drive-write-limited part of replacing the failed drive with a new hot spare, you are not degraded from a safety of data point of view.
- mbreese 6y ago
  > This will take a long time, since you will be writing to basically this whole drive as part of distributing its spare space. (Plus probably shuffling chunks around between many other drives) Yes. With the caveat that you'll be *reading* from all of the disks to satisfy those writes. In a traditional setup, you'd have multiple striped raidz{,2,3} vdevs. When you swap in a spare, you're only pulling reads from the disks in the same vdev. This will drive the performance from this vdev to the ground. In this draidz setup, you'll be pulling from all of the disks, so the entire pool is stressed at the same rate. I think the way I'm starting to think about it is that each drive has X amount of spare capacity or buckets. But they are otherwise used. When a drive goes out, the primary data from that drive is distributed across the array to the already-in-place spare capacity. So, yes, after the primary data is mirrored out, the pool is no longer degraded. But, I think the point being made here is that you'll still want to replace the drive, which will still cause *that* drive to be fully written.
  - jsmith45 6y ago
    Right, the free space buckets are a fine analogy. They are conceptually similar in some ways to the free space in the hot standby drive. I don't think from a data safety or performance perspective pre dRAID vdev really differ too terribly much from raid 50 setups. (They do differ greatly in administrative flexibility, versus the insanely rigid approachs of traditional hardware raid). But yeah, after replacing the failed drive, ideally dRAID would basically do just what it did after drive failure in reverse. (End result: No longer having any drives holding data for a different vdev than they belong to). Obviously this means writing basically that whole disk, and reading from potentially every disk. But as you mention you are spreading the load out across the whole array, hopefully reducing the overall impact, and being less likely to stress any drive into dying.
bbarnett 6y ago
Not sure I like this, for many spinning disk raids of that scale I have are mostly read, rarely write. With raid 6 for example, I pop in a new drive, and all the write/repair hits only that drive. With a virtual spare, all drives end up with massive writes, meaning I might hit write errors, thus borking a raid which would have been ok for reads. I realise there are 100 'what ifs' and scenarios here, but when my raids tend to drop a drive, it's on heavy write ops, not reads. I'd say 90% of the time. I imagine ssd cascading/eol write failures might make this virtual spare failure scenario even more likely. (yes I am aware of sector reallocation delays on spinning disks, I am referring to real EOL scenarios on drives during writes, being far more common for me than reads)

jsmith45 6y ago

While I can see the hot-spare advantage here, I do have to wonder about the impact of replacing the broken drive. I suppose it works out as follows. With traditional raid, you are rebuilding while degraded from a safety perspective. So obviously, if your configuration guarantees losing `n` drives while healthy does not cause data loss, you could lose data during rebuild if `n` drives failed. With the distributed hot spare approach, you are no longer degraded from a safety perspective as soon as the distributed spare's data is written. This is faster than traditional raid, as the writes are distributed among many disks. So time-wise you get out of data safety degraded state faster. (It does seem like it might be riskier though, since you are doing heavy writes to many drives while safety degraded, instead of only writes to the spare). So now you replace the failed drive, and the array rebuilds to make this new spare's capacity distributed. This will take a long time, since you will be writing to basically this whole drive as part of distributing its spare space. (Plus probably shuffling chunks around between many other drives). This could possibly take even longer than traditional raid rebuild. However during this whole process, your array is not in degraded state from a data safety point of view. Or to summarize: during the long single-drive-write-limited part of replacing the failed drive with a new hot spare, you are not degraded from a safety of data point of view.

mbreese 6y ago
> This will take a long time, since you will be writing to basically this whole drive as part of distributing its spare space. (Plus probably shuffling chunks around between many other drives) Yes. With the caveat that you'll be *reading* from all of the disks to satisfy those writes. In a traditional setup, you'd have multiple striped raidz{,2,3} vdevs. When you swap in a spare, you're only pulling reads from the disks in the same vdev. This will drive the performance from this vdev to the ground. In this draidz setup, you'll be pulling from all of the disks, so the entire pool is stressed at the same rate. I think the way I'm starting to think about it is that each drive has X amount of spare capacity or buckets. But they are otherwise used. When a drive goes out, the primary data from that drive is distributed across the array to the already-in-place spare capacity. So, yes, after the primary data is mirrored out, the pool is no longer degraded. But, I think the point being made here is that you'll still want to replace the drive, which will still cause *that* drive to be fully written.
- jsmith45 6y ago
  Right, the free space buckets are a fine analogy. They are conceptually similar in some ways to the free space in the hot standby drive. I don't think from a data safety or performance perspective pre dRAID vdev really differ too terribly much from raid 50 setups. (They do differ greatly in administrative flexibility, versus the insanely rigid approachs of traditional hardware raid). But yeah, after replacing the failed drive, ideally dRAID would basically do just what it did after drive failure in reverse. (End result: No longer having any drives holding data for a different vdev than they belong to). Obviously this means writing basically that whole disk, and reading from potentially every disk. But as you mention you are spreading the load out across the whole array, hopefully reducing the overall impact, and being less likely to stress any drive into dying.

bbarnett 6y ago

Not sure I like this, for many spinning disk raids of that scale I have are mostly read, rarely write. With raid 6 for example, I pop in a new drive, and all the write/repair hits only that drive. With a virtual spare, all drives end up with massive writes, meaning I might hit write errors, thus borking a raid which would have been ok for reads. I realise there are 100 'what ifs' and scenarios here, but when my raids tend to drop a drive, it's on heavy write ops, not reads. I'd say 90% of the time. I imagine ssd cascading/eol write failures might make this virtual spare failure scenario even more likely. (yes I am aware of sector reallocation delays on spinning disks, I am referring to real EOL scenarios on drives during writes, being far more common for me than reads)