5 ms·

To be fair, I don't think this is little-known. When a write must be acknowledged by multiple devices (as is the case with RAID1 or RAIDZ), a write requires mor

by Youden 5y ago

To be fair, I don't think this is little-known. When a write must be acknowledged by multiple devices (as is the case with RAID1 or RAIDZ), a write requires more IOPS.

Dylan16807 5y ago
Let's say I have a 4+1 disk layout, and I do 100 data writes followed by an fsync. Naively I would expect that to become 25 data writes per disk, and then the fsync would go to all disks. That might be about 30 writes total, so I'd expect it to be at least 3x as fast as a single disk. Where does that expectation break down?
- barrkel 5y ago
  100 data writes, assuming they're to different files and/or large enough to fill a slice, means 500 disk writes (5 disk writes per slice). The 500 disk writes are parallelized over 5 disks so they only take the time taken for 100 writes (500 / 5). So the IOPs is the same as a single disk. HOWEVER, bandwidth is 4x. The above assumes that seek time dominates. If your writes are multiple slices in length, they will be written 4x faster because the amount of data per disk is divided across the disks. If you're reading and writing large contiguous files, then you do get a big I/O boost from raidz.
  - Dylan16807 5y ago
    > 100 data writes, assuming they're to different files and/or large enough to fill a slice, means 500 disk writes (5 disk writes per slice). And if I wrote 100 slices to a single drive, is that 100 writes or 400 writes? If it's 100, then it sounds like the slices are sized wrong: they should get bigger when I add more disks. If it's 400 writes, then 100 writes per disk should be much faster.
    - barrkel 5y ago
      The slices depend on record size and sector size. For 4 data disks and default 128k record you have 32k per disk per record. With more disks, the proportion of a slice per disk decreases, not increases, and it's rounded up to the sector size, so there's usually some loss of space on larger parity schemas.
- secabeen 5y ago
  This summarizes it well for reads: https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSRaidzReadPerformance https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSRaidzRea... For writes, the system is copy-on-write, so for each write, there is a full read of the containing record, often across all disks, so the updated record can be written out in the new location on the disks without disrupting the old record, which may still be referenced by snapshots, etc. Adding additional disks gets you more throughput (you can split a single large write across more disks, but not more OPS.)
  - Dylan16807 5y ago
    Okay, the follow-up post really explains it. It's yet another consequence of ZFS's inability to move blocks. So you more or less can't have multiple writes get lumped together because that would break garbage collection. There are ways to work around elements of this and get much better performance, but not wanting more complexity makes enough sense. Definitely a flaw in the ZFS data model though, rather than something inherent to the use of multiple disks.
    - kaba0 5y ago
      I am a beginner when it comes to ZFS, but isn’t “no moving data” as an axiom a good choice? Any error that may happen during would destroy that data - while without moving it will likely be recoverable even with a dead harddrive.
      - Dylan16807 5y ago
        > I am a beginner when it comes to ZFS, but isn’t “no moving data” as an axiom a good choice? It's a reasonable choice, but only because it makes certain kinds of bugs harder, not because it's safer when the code is correct. > Any error that may happen during would destroy that data - while without moving it will likely be recoverable even with a dead harddrive. That's not true. You make the new copy, then update every reference to the new copy, and only then remove the old one. If there's an error halfway through then there's two copies of the data.
    - 5y ago
- maccam94 5y ago
  Each write is striped across all of the disks. For a stripe to be completely written, all disks must finish writing the data. Thus your max write IOPS equals that of the slowest drive in the vdev.
  - wtallis 5y ago
    > Each write is striped across all of the disks. It's reasonable to expect that issuing a batch of 100 writes at the application layer followed by a fsync would not always require doing 100 writes to each underlying block device. The OS/FS should be able to combine writes when the IO pattern allows for it, and should be doing some buffering prior to the fsync in hopes of assembling full-stripe writes out of smaller application-layer writes.
  - Dylan16807 5y ago
    > Each write is striped across all of the disks. It sure shouldn't be. This isn't dumb RAID.

by Youden 5y ago

To be fair, I don't think this is little-known. When a write must be acknowledged by multiple devices (as is the case with RAID1 or RAIDZ), a write requires more IOPS.

Dylan16807 5y ago

Let's say I have a 4+1 disk layout, and I do 100 data writes followed by an fsync. Naively I would expect that to become 25 data writes per disk, and then the fsync would go to all disks. That might be about 30 writes total, so I'd expect it to be at least 3x as fast as a single disk. Where does that expectation break down?

barrkel 5y ago
100 data writes, assuming they're to different files and/or large enough to fill a slice, means 500 disk writes (5 disk writes per slice). The 500 disk writes are parallelized over 5 disks so they only take the time taken for 100 writes (500 / 5). So the IOPs is the same as a single disk. HOWEVER, bandwidth is 4x. The above assumes that seek time dominates. If your writes are multiple slices in length, they will be written 4x faster because the amount of data per disk is divided across the disks. If you're reading and writing large contiguous files, then you do get a big I/O boost from raidz.
- Dylan16807 5y ago
  > 100 data writes, assuming they're to different files and/or large enough to fill a slice, means 500 disk writes (5 disk writes per slice). And if I wrote 100 slices to a single drive, is that 100 writes or 400 writes? If it's 100, then it sounds like the slices are sized wrong: they should get bigger when I add more disks. If it's 400 writes, then 100 writes per disk should be much faster.
  - barrkel 5y ago
    The slices depend on record size and sector size. For 4 data disks and default 128k record you have 32k per disk per record. With more disks, the proportion of a slice per disk decreases, not increases, and it's rounded up to the sector size, so there's usually some loss of space on larger parity schemas.
secabeen 5y ago
This summarizes it well for reads: https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSRaidzReadPerformance https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSRaidzRea... For writes, the system is copy-on-write, so for each write, there is a full read of the containing record, often across all disks, so the updated record can be written out in the new location on the disks without disrupting the old record, which may still be referenced by snapshots, etc. Adding additional disks gets you more throughput (you can split a single large write across more disks, but not more OPS.)
- Dylan16807 5y ago
  Okay, the follow-up post really explains it. It's yet another consequence of ZFS's inability to move blocks. So you more or less can't have multiple writes get lumped together because that would break garbage collection. There are ways to work around elements of this and get much better performance, but not wanting more complexity makes enough sense. Definitely a flaw in the ZFS data model though, rather than something inherent to the use of multiple disks.
  - kaba0 5y ago
    I am a beginner when it comes to ZFS, but isn’t “no moving data” as an axiom a good choice? Any error that may happen during would destroy that data - while without moving it will likely be recoverable even with a dead harddrive.
    - Dylan16807 5y ago
      > I am a beginner when it comes to ZFS, but isn’t “no moving data” as an axiom a good choice? It's a reasonable choice, but only because it makes certain kinds of bugs harder, not because it's safer when the code is correct. > Any error that may happen during would destroy that data - while without moving it will likely be recoverable even with a dead harddrive. That's not true. You make the new copy, then update every reference to the new copy, and only then remove the old one. If there's an error halfway through then there's two copies of the data.
  - 5y ago
maccam94 5y ago
Each write is striped across all of the disks. For a stripe to be completely written, all disks must finish writing the data. Thus your max write IOPS equals that of the slowest drive in the vdev.
- wtallis 5y ago
  > Each write is striped across all of the disks. It's reasonable to expect that issuing a batch of 100 writes at the application layer followed by a fsync would not always require doing 100 writes to each underlying block device. The OS/FS should be able to combine writes when the IO pattern allows for it, and should be doing some buffering prior to the fsync in hopes of assembling full-stripe writes out of smaller application-layer writes.
- Dylan16807 5y ago
  > Each write is striped across all of the disks. It sure shouldn't be. This isn't dumb RAID.