4 ms·

> That's the easy thing with ZFS, just add the disks as new VDEVs What are the drawbacks of this approach and why can't I just add additional drives to an exis

> That's the easy thing with ZFS, just add the disks as new VDEVs What are the drawbacks of this approach and why can't I just add additional drives to an existing VDEV?

liuliu 6y ago
For RaidZ cases, you need to rebalance all the strips, which is non-trivial to do online (you have to make sure if the power cut off during the rebalancing, you don't lose the data). Without rebalancing, I think it can end up with weird corner cases (for example, in RaidZ2 case, if you have the rest 5 disks full and added 1 more disk, you cannot write data into the pool because there is no place for parity. It is not a problem if the disks are balanced).
magicalhippo 6y ago
Well, you can add drives to _some_ VDEV types. For a single drive VDEV you can add a drive and it will become a mirror. With a 2-way mirror you can add a drive and it will become a 3-way mirror. However for the RAID-Z variants it's more difficult, because of how the parity calculations are done and how the parity is stored. If you have a 5 drive RAID-Z, you can store up to 4 blocks of data for each parity block. With a 6 drive RAID-Z you can store up to 5 blocks of data for each parity block. So if you add a drive to a RAID-Z vdev, the system suddenly needs to keep track of the stripe width that was used when the data was written. In addition it's the issue of _where_ the data is stored. If your RAID-Z VDEV is mostly full and you were to add a new drive to it (not that you currently can), then you might not be able to use all of the capacity without further ado. This is because you can only store one block per stripe on the new drive for redundancy to work, a RAID-Z1 can only recover from the loss of one block per stripe. The rest of the blocks in the stripe has to be distributed on the other drives. However, there is work in progress[1] to support RAID-Z expansion. That said, RAID-Z expansion is for expanding one drive at a time. If you're adding 20 disks in one go, do it as a new VDEV. [1]: https://github.com/openzfs/zfs/pull/8853 https://github.com/openzfs/zfs/pull/8853
- tutfbhuf 6y ago
  Is there any official statement from the ZFS maintainers on when we can expect the Pull Request (RAID-Z expansion) to be merged?
  - magicalhippo 6y ago
    Not really, but as you can see from the PR[1] there is work being done. The ZFS devs don't like to push unstable features, so I expect we're still quite a way off. [1]: https://github.com/openzfs/zfs/pull/8853 https://github.com/openzfs/zfs/pull/8853
  - secabeen 6y ago
    Years. It'll be years.

> That's the easy thing with ZFS, just add the disks as new VDEVs What are the drawbacks of this approach and why can't I just add additional drives to an existing VDEV?

liuliu 6y ago

For RaidZ cases, you need to rebalance all the strips, which is non-trivial to do online (you have to make sure if the power cut off during the rebalancing, you don't lose the data). Without rebalancing, I think it can end up with weird corner cases (for example, in RaidZ2 case, if you have the rest 5 disks full and added 1 more disk, you cannot write data into the pool because there is no place for parity. It is not a problem if the disks are balanced).

magicalhippo 6y ago

Well, you can add drives to _some_ VDEV types. For a single drive VDEV you can add a drive and it will become a mirror. With a 2-way mirror you can add a drive and it will become a 3-way mirror. However for the RAID-Z variants it's more difficult, because of how the parity calculations are done and how the parity is stored. If you have a 5 drive RAID-Z, you can store up to 4 blocks of data for each parity block. With a 6 drive RAID-Z you can store up to 5 blocks of data for each parity block. So if you add a drive to a RAID-Z vdev, the system suddenly needs to keep track of the stripe width that was used when the data was written. In addition it's the issue of _where_ the data is stored. If your RAID-Z VDEV is mostly full and you were to add a new drive to it (not that you currently can), then you might not be able to use all of the capacity without further ado. This is because you can only store one block per stripe on the new drive for redundancy to work, a RAID-Z1 can only recover from the loss of one block per stripe. The rest of the blocks in the stripe has to be distributed on the other drives. However, there is work in progress[1] to support RAID-Z expansion. That said, RAID-Z expansion is for expanding one drive at a time. If you're adding 20 disks in one go, do it as a new VDEV. [1]: https://github.com/openzfs/zfs/pull/8853 https://github.com/openzfs/zfs/pull/8853

tutfbhuf 6y ago
Is there any official statement from the ZFS maintainers on when we can expect the Pull Request (RAID-Z expansion) to be merged?
- magicalhippo 6y ago
  Not really, but as you can see from the PR[1] there is work being done. The ZFS devs don't like to push unstable features, so I expect we're still quite a way off. [1]: https://github.com/openzfs/zfs/pull/8853 https://github.com/openzfs/zfs/pull/8853
- secabeen 6y ago
  Years. It'll be years.