8 ms·

This is great, but an important and little known caveat is that raidz is limited to the iops of one disk. So growing a raidz will at some point have lots of thr

This is great, but an important and little known caveat is that raidz is limited to the iops of one disk. So growing a raidz will at some point have lots of throughput but suffer in small and random reads and writes. At that point, it will be better to grow the pool with additional, separate raidz.

Osiris 5y ago
So you setup two raid-z then stripe them for increased performance?
- unethical_ban 5y ago
  This is how I understand it. The hierarchy is disk < vdev < zpool. Disk is physical. vdev is logical. Purpose: Disk grouping and redundancy. Composition: One or more disks. zpool is logical. Purpose: Higher-level management of one or more vdevs. Composition: It acts like a JBOD. --- zpools can be thought of as "stripes of vdevs". This, in the narrow sense that the failure of any vdev in a zpool is a permanent loss of the entire zpool. All your redundancy in the ZFS ecosystem is via mirrored or RAID'ed vdevs. --- The setup I have heard of that balances performance, redundancy and space is to do what you say: Have a zpool of multiple mirror-type vdevs. You can also stripe at the vdev level, which I would assume has higher performance than having multiple single-disk vdevs in a pool - I'm unaware of the differences at a low level.
- oakwhiz 5y ago
  That's a common configuration for larger arrays where you probably don't want to make individual RAIDZ arrays too wide on their own. It's functionally similar to RAID 50 or 60. However when the budget is tight you might be incentivized to compromise on performance or reliability, so this new expansion feature really does help in hobbyist or shoestring budget situations where you just want one huge array to maximize usable space while still being able to tolerate some failures. Typically you would buy disks as you go along to try to stretch the budget further as utilization increases.
Youden 5y ago
To be fair, I don't think this is little-known. When a write must be acknowledged by multiple devices (as is the case with RAID1 or RAIDZ), a write requires more IOPS.
- Dylan16807 5y ago
  Let's say I have a 4+1 disk layout, and I do 100 data writes followed by an fsync. Naively I would expect that to become 25 data writes per disk, and then the fsync would go to all disks. That might be about 30 writes total, so I'd expect it to be at least 3x as fast as a single disk. Where does that expectation break down?
  - barrkel 5y ago
    100 data writes, assuming they're to different files and/or large enough to fill a slice, means 500 disk writes (5 disk writes per slice). The 500 disk writes are parallelized over 5 disks so they only take the time taken for 100 writes (500 / 5). So the IOPs is the same as a single disk. HOWEVER, bandwidth is 4x. The above assumes that seek time dominates. If your writes are multiple slices in length, they will be written 4x faster because the amount of data per disk is divided across the disks. If you're reading and writing large contiguous files, then you do get a big I/O boost from raidz.
    - Dylan16807 5y ago
      > 100 data writes, assuming they're to different files and/or large enough to fill a slice, means 500 disk writes (5 disk writes per slice). And if I wrote 100 slices to a single drive, is that 100 writes or 400 writes? If it's 100, then it sounds like the slices are sized wrong: they should get bigger when I add more disks. If it's 400 writes, then 100 writes per disk should be much faster.
      - barrkel 5y ago
        The slices depend on record size and sector size. For 4 data disks and default 128k record you have 32k per disk per record. With more disks, the proportion of a slice per disk decreases, not increases, and it's rounded up to the sector size, so there's usually some loss of space on larger parity schemas.
rodgerd 5y ago
> little known caveat I'm surprised that this would be a little-known caveat; it's been a problem with all RAID-parity systems 3,4,5 and Z) since forever.
- wmf 5y ago
  No, RAID-4/5/6 give O(N) IOPS while RAID-Z gives O(1) IOPS.
wyager 5y ago
To clarify, the sequential read bandwidth of a RAIDZN with M total disks will be M times the read bandwidth of a single disk. The sequential write bandwidth will be (M-N) times the write bandwidth of a single disk. These are optimal values. https://calomel.org/zfs_raid_speed_capacity.html https://calomel.org/zfs_raid_speed_capacity.html It's a constraint on iops in particular, not bandwidth, because writing a record to disk involves writing something to every disk in the pool. So if you have a disk that's taking a long time to write, the other disks need to pause and let it catch up.
- uniqueuid 5y ago
  Precisely. The split between bandwidth and iops is what remains largely absent from discussions on ZFS and also from benchmarks.

Osiris 5y ago

So you setup two raid-z then stripe them for increased performance?

unethical_ban 5y ago
This is how I understand it. The hierarchy is disk < vdev < zpool. Disk is physical. vdev is logical. Purpose: Disk grouping and redundancy. Composition: One or more disks. zpool is logical. Purpose: Higher-level management of one or more vdevs. Composition: It acts like a JBOD. --- zpools can be thought of as "stripes of vdevs". This, in the narrow sense that the failure of any vdev in a zpool is a permanent loss of the entire zpool. All your redundancy in the ZFS ecosystem is via mirrored or RAID'ed vdevs. --- The setup I have heard of that balances performance, redundancy and space is to do what you say: Have a zpool of multiple mirror-type vdevs. You can also stripe at the vdev level, which I would assume has higher performance than having multiple single-disk vdevs in a pool - I'm unaware of the differences at a low level.
oakwhiz 5y ago
That's a common configuration for larger arrays where you probably don't want to make individual RAIDZ arrays too wide on their own. It's functionally similar to RAID 50 or 60. However when the budget is tight you might be incentivized to compromise on performance or reliability, so this new expansion feature really does help in hobbyist or shoestring budget situations where you just want one huge array to maximize usable space while still being able to tolerate some failures. Typically you would buy disks as you go along to try to stretch the budget further as utilization increases.

Youden 5y ago

To be fair, I don't think this is little-known. When a write must be acknowledged by multiple devices (as is the case with RAID1 or RAIDZ), a write requires more IOPS.

Dylan16807 5y ago
Let's say I have a 4+1 disk layout, and I do 100 data writes followed by an fsync. Naively I would expect that to become 25 data writes per disk, and then the fsync would go to all disks. That might be about 30 writes total, so I'd expect it to be at least 3x as fast as a single disk. Where does that expectation break down?
- barrkel 5y ago
  100 data writes, assuming they're to different files and/or large enough to fill a slice, means 500 disk writes (5 disk writes per slice). The 500 disk writes are parallelized over 5 disks so they only take the time taken for 100 writes (500 / 5). So the IOPs is the same as a single disk. HOWEVER, bandwidth is 4x. The above assumes that seek time dominates. If your writes are multiple slices in length, they will be written 4x faster because the amount of data per disk is divided across the disks. If you're reading and writing large contiguous files, then you do get a big I/O boost from raidz.
  - Dylan16807 5y ago
    > 100 data writes, assuming they're to different files and/or large enough to fill a slice, means 500 disk writes (5 disk writes per slice). And if I wrote 100 slices to a single drive, is that 100 writes or 400 writes? If it's 100, then it sounds like the slices are sized wrong: they should get bigger when I add more disks. If it's 400 writes, then 100 writes per disk should be much faster.
    - barrkel 5y ago
      The slices depend on record size and sector size. For 4 data disks and default 128k record you have 32k per disk per record. With more disks, the proportion of a slice per disk decreases, not increases, and it's rounded up to the sector size, so there's usually some loss of space on larger parity schemas.

rodgerd 5y ago

> little known caveat I'm surprised that this would be a little-known caveat; it's been a problem with all RAID-parity systems 3,4,5 and Z) since forever.

wmf 5y ago
No, RAID-4/5/6 give O(N) IOPS while RAID-Z gives O(1) IOPS.

wyager 5y ago

To clarify, the sequential read bandwidth of a RAIDZN with M total disks will be M times the read bandwidth of a single disk. The sequential write bandwidth will be (M-N) times the write bandwidth of a single disk. These are optimal values. https://calomel.org/zfs_raid_speed_capacity.html https://calomel.org/zfs_raid_speed_capacity.html It's a constraint on iops in particular, not bandwidth, because writing a record to disk involves writing something to every disk in the pool. So if you have a disk that's taking a long time to write, the other disks need to pause and let it catch up.

uniqueuid 5y ago
Precisely. The split between bandwidth and iops is what remains largely absent from discussions on ZFS and also from benchmarks.