4 ms·

I’m not sure how much I buy their claim that ZFS being COW is worse for SSDs given SSDs themselves are COW to allow for wear-leveling because they have to erase

by kevvok 5y ago

I’m not sure how much I buy their claim that ZFS being COW is worse for SSDs given SSDs themselves are COW to allow for wear-leveling because they have to erase whole blocks and rewrite them if even one byte is changed

b112 5y ago
They say their fs block size is 64kb. How large are SSD blocks?
- jeffbee 5y ago
  SSD erase blocks vary, could be anywhere from 64KiB to several MiB. Their logical blocks (visible to the host operating system) are usually either 512 or 4096 bytes.
  - wtallis 5y ago
    NAND flash erase blocks passed 1MB a long time ago. For mainstream TLC NAND, 16 to 24 MB is currently typical, and QLC NAND has gone as high as 48 and 96 MB. NAND page sizes are usually 16kB, but often with support for faster partial page programming for the sake of 4kB operations. Logical block sizes of 512 bytes are purely a fiction for the sake of compatibility and SSDs don't actually track allocations at that granularity. They do track things at 4kB granularity even though that's not quite large enough to be a good fit for today's flash.
    - namibj 5y ago
      Where does one find information on erase granularity, and, like, doing the optimizations offered by NVMe ZNS on a "normal" SSD? I assume if you throw a sufficiently uncomplicated ZNS-like workload at them, they don't try to ruin your efforts? I guess arguably this comes from the lack of availability of ZNS drives, especially ones that aren't outcompeted by a Samsung 980 series drive on workloads that don't exceed the 980's warranty's 600 drive writes over 5 years. And depending on what you're doing, you may even use such drives for the IO-heavy load and retire them after 90% of their TBW to something like fileserver duty.
      - wtallis 5y ago
        For recent drives, if you're lucky the vendor has implemented the Preferred Write Alignment, Preferred Write Granularity, and corresponding Deallocate alignment and granularity reports, added in NVMe 1.4 (2019). Those values may not exactly correspond to the underlying NAND page and erase block sizes, but can probably be taken at face value for their intended purpose if the drive actually reports values larger than 4kB. Otherwise, finding that kind of low-level NAND specifications often comes down to trawling through TechInsights teardowns and conference proceedings (especially ISSCC, though what gets presented doesn't always match what goes into mass production). A zoned-like workload that is append-only within large chunks and uses a Deallocate (Trim) or Write Zeros command to clear chunks should perform well even on standard SSDs. I don't think the exact zone size has much impact, as long as it's comfortably above the erase block size.
notacoward 5y ago
The fact that there's two layers both doing this kind of thing (especially oblivious to one another) is a known and studied issue. Most of that study has been in the context of virtual-machine filesystems atop host filesystems but it's also applicable here. I'm pretty sure there was a paper at FAST about this several years ago, but can't find it right now. It's entirely likely that COW-over-COW is a bad choice just like TCP-over-TCP is.
- klodolph 5y ago
  Are you thinking of the classic, "Don't Stack Your Log On My Log"? https://www.usenix.org/conference/inflow14/workshop-program/presentation/yang https://www.usenix.org/conference/inflow14/workshop-program/...
  - notacoward 5y ago
    Yeah, seems about right. Thanks!
- the8472 5y ago
  But the upper-level CoW can coordinate with the lower level via TRIM, that's not quite the same as TCP-in-TCP where the congestion algorithms don't talk to each other.
  - notacoward 5y ago
    Yes, TRIM exists, but it's a very limited form of coordination at best and implementations don't even do all they can with that. Every analogy is imperfect.
toast0 5y ago
Given that they mention snapshots, that's probably the bigger issue. Almost any sort of storage works better when you have more free space, and having a snapshot means you need to keep all that data as well as any data that changed since then, so you have less free space. Using a COW filesystem adds at least some amount of usage, since instead of modifying in place, you'd write a new block and only trim the old block sometime after the new block is committed; but if you don't have snapshots and you have zfs autotrim (and it trims all your old blocks), the commit interval is short (5 seconds by default?), so I wouldn't expect a big difference in effective free space here.
bob1029 5y ago
My experience with SSDs tells me the only way to beat the system is to employ some append-only log storage structure. Potentially with segmentation done at the device level, so that you can have a large fleet of drives in "append" mode while others are reconciling or cleaning all their blocks in anticipation of taking another full sequential fill-up. Throwing mixed workloads at individual devices is just asking for trouble if you are trying to maintain some razor-thin SLA. Dirty pages and all the weird tricks employed to hide this concern results in side effects that break more complex schemes on top. Taking to the next level - Batching your I/O in software is how you can start saying things like "Transactions per disk I/O", not just "Fewer I/O for those transactions which now fit in fewer blocks due to compression". Batching doesn't have to mean "nightly processing". It can mean "all requests which occurred over the last 100uS". From a user's perspective, this can effectively still be a real-time RPC experience. For systems with very heavy load, this sort of micro-batching can add many orders of magnitude improvement in throughput. Also bear in mind that the more transactions you have available to compress each time means you get better odds when dealing with entropy. I have personally developed software that can insert 2-5x the stated write IOPS figure with these sorts of tactics. On modern NVMe devices this can mean you start tickling 8 figure transactions per second if the size of each request is very modest.
- pdimitar 5y ago
  I would love to read a more detailed blog post about the software you wrote (if it's not confidential). Sounds like you solved very interesting problems.
- wtallis 5y ago
  Along these lines, NVMe has standardized an optional Predictable Latency Mode so the host can instruct the SSD to temporarily suspend background work. It's intended to help you offer tight latency guarantees when you have intelligent load balancing across a large pool of drives.
sipos 5y ago
I thought the same thing as I was reading it, but I think they are probably using larger block sizes than the SSD's blocks for better compression. I'm not certain though. Edit: reading other comments, it looks like their block size is only 64KiB, so this isn't the case, so I don't know the answer. I can only think that perhaps it is an issue because ZFS doesn't deallocate the freed blocks quickly enough and they are making changes fast, meaning a significant amount of disk space is used by blocks that are about to be TRIMed but haven't been yet.

by kevvok 5y ago

b112 5y ago

They say their fs block size is 64kb. How large are SSD blocks?

jeffbee 5y ago
SSD erase blocks vary, could be anywhere from 64KiB to several MiB. Their logical blocks (visible to the host operating system) are usually either 512 or 4096 bytes.
- wtallis 5y ago
  NAND flash erase blocks passed 1MB a long time ago. For mainstream TLC NAND, 16 to 24 MB is currently typical, and QLC NAND has gone as high as 48 and 96 MB. NAND page sizes are usually 16kB, but often with support for faster partial page programming for the sake of 4kB operations. Logical block sizes of 512 bytes are purely a fiction for the sake of compatibility and SSDs don't actually track allocations at that granularity. They do track things at 4kB granularity even though that's not quite large enough to be a good fit for today's flash.
  - namibj 5y ago
    Where does one find information on erase granularity, and, like, doing the optimizations offered by NVMe ZNS on a "normal" SSD? I assume if you throw a sufficiently uncomplicated ZNS-like workload at them, they don't try to ruin your efforts? I guess arguably this comes from the lack of availability of ZNS drives, especially ones that aren't outcompeted by a Samsung 980 series drive on workloads that don't exceed the 980's warranty's 600 drive writes over 5 years. And depending on what you're doing, you may even use such drives for the IO-heavy load and retire them after 90% of their TBW to something like fileserver duty.
    - wtallis 5y ago
      For recent drives, if you're lucky the vendor has implemented the Preferred Write Alignment, Preferred Write Granularity, and corresponding Deallocate alignment and granularity reports, added in NVMe 1.4 (2019). Those values may not exactly correspond to the underlying NAND page and erase block sizes, but can probably be taken at face value for their intended purpose if the drive actually reports values larger than 4kB. Otherwise, finding that kind of low-level NAND specifications often comes down to trawling through TechInsights teardowns and conference proceedings (especially ISSCC, though what gets presented doesn't always match what goes into mass production). A zoned-like workload that is append-only within large chunks and uses a Deallocate (Trim) or Write Zeros command to clear chunks should perform well even on standard SSDs. I don't think the exact zone size has much impact, as long as it's comfortably above the erase block size.

notacoward 5y ago

The fact that there's two layers both doing this kind of thing (especially oblivious to one another) is a known and studied issue. Most of that study has been in the context of virtual-machine filesystems atop host filesystems but it's also applicable here. I'm pretty sure there was a paper at FAST about this several years ago, but can't find it right now. It's entirely likely that COW-over-COW is a bad choice just like TCP-over-TCP is.

klodolph 5y ago
Are you thinking of the classic, "Don't Stack Your Log On My Log"? https://www.usenix.org/conference/inflow14/workshop-program/presentation/yang https://www.usenix.org/conference/inflow14/workshop-program/...
- notacoward 5y ago
  Yeah, seems about right. Thanks!
the8472 5y ago
But the upper-level CoW can coordinate with the lower level via TRIM, that's not quite the same as TCP-in-TCP where the congestion algorithms don't talk to each other.
- notacoward 5y ago
  Yes, TRIM exists, but it's a very limited form of coordination at best and implementations don't even do all they can with that. Every analogy is imperfect.

toast0 5y ago

Given that they mention snapshots, that's probably the bigger issue. Almost any sort of storage works better when you have more free space, and having a snapshot means you need to keep all that data as well as any data that changed since then, so you have less free space. Using a COW filesystem adds at least some amount of usage, since instead of modifying in place, you'd write a new block and only trim the old block sometime after the new block is committed; but if you don't have snapshots and you have zfs autotrim (and it trims all your old blocks), the commit interval is short (5 seconds by default?), so I wouldn't expect a big difference in effective free space here.

bob1029 5y ago

My experience with SSDs tells me the only way to beat the system is to employ some append-only log storage structure. Potentially with segmentation done at the device level, so that you can have a large fleet of drives in "append" mode while others are reconciling or cleaning all their blocks in anticipation of taking another full sequential fill-up. Throwing mixed workloads at individual devices is just asking for trouble if you are trying to maintain some razor-thin SLA. Dirty pages and all the weird tricks employed to hide this concern results in side effects that break more complex schemes on top. Taking to the next level - Batching your I/O in software is how you can start saying things like "Transactions per disk I/O", not just "Fewer I/O for those transactions which now fit in fewer blocks due to compression". Batching doesn't have to mean "nightly processing". It can mean "all requests which occurred over the last 100uS". From a user's perspective, this can effectively still be a real-time RPC experience. For systems with very heavy load, this sort of micro-batching can add many orders of magnitude improvement in throughput. Also bear in mind that the more transactions you have available to compress each time means you get better odds when dealing with entropy. I have personally developed software that can insert 2-5x the stated write IOPS figure with these sorts of tactics. On modern NVMe devices this can mean you start tickling 8 figure transactions per second if the size of each request is very modest.

pdimitar 5y ago
I would love to read a more detailed blog post about the software you wrote (if it's not confidential). Sounds like you solved very interesting problems.
wtallis 5y ago
Along these lines, NVMe has standardized an optional Predictable Latency Mode so the host can instruct the SSD to temporarily suspend background work. It's intended to help you offer tight latency guarantees when you have intelligent load balancing across a large pool of drives.

sipos 5y ago

I thought the same thing as I was reading it, but I think they are probably using larger block sizes than the SSD's blocks for better compression. I'm not certain though. Edit: reading other comments, it looks like their block size is only 64KiB, so this isn't the case, so I don't know the answer. I can only think that perhaps it is an issue because ZFS doesn't deallocate the freed blocks quickly enough and they are making changes fast, meaning a significant amount of disk space is used by blocks that are about to be TRIMed but haven't been yet.