9 ms·

Moving from spinning rust to solid-state storage dramatically improved the performance of random reads, but random writes still carry a penalty compared to sequ

Moving from spinning rust to solid-state storage dramatically improved the performance of random reads, but random writes still carry a penalty compared to sequential writes. So sequential I/O is perhaps half-dead; there's no need to optimize layouts for sequential reads any more.

GauntletWizard 3y ago
There's still a ton of performance benefit that can be gained by keeping your read pipeline full, and you can most easily keep your read pipeline full by making it guessable by doing readahead. It doesn't need to be strictly disk-layout sequential, file-layout sequential will usually do.
Proven 3y ago
Far from it, sequential IO optimization isn't dead. Sequential reads need to be optimized to produce optimal performance for selected workload. That usually means applying a lower level of inline compression. In some cases deduplication works better before, in other after, compression. Sometimes post-process dedupe is more suitable than inline. Then there's erasure coding and data protection methods that are still being optimized for NVMe with sequential workloads, including random workloads which are being sequentialized to work better with latest flash media. I would even say developments in sequential IO are becoming more important than random IO.
jasonwatkinspdx 3y ago
Yes. And the Flash Translation Layer inside modern drives are quite sophisticated. This means you could start out with a write pattern that's causing a lot of copying and reorganization of pages in the back ground, typically because it's doing a lot of unaligned random writes smaller than the erase block size, and initially the performance will be fine. But as the drive wears in, and the capacity fills up, the FTL has less extra space to work with, forcing this garbage collection activity to become increasingly common. So you get performance degradation and premature wearing of the drive in a way that's opaque.
- bcrl 3y ago
  Short stroking SSDs is a commonly used trick in the toolbag of wise sysadmins for dealing with this kind of workload.
  - Waterluvian 3y ago
    I’m guessing that means pretending they’re smaller than they really are?
    - colordrops 3y ago
      Looks like that's the case based on a cursory google search. Seems analogous to only charging batteries up to 80%. I wish companies would factor these issues into their products and leave invisible buffers, but I guess making them cheaper is more important than being consistent and reliable.
      - Waterluvian 3y ago
        Sales and marketing will never leave anything on the table that might make their job easier.
      - wtallis 3y ago
        They do leave invisible buffers, commonly referred to as over provisioning. The discrepancy between GB and GiB is ~7% hiding in plain sight, and then more expensive enterprise drives are commonly sold in (usable) capacities like 960GB and 7.68TB rather than 1TB and 8TB.
        andrecarini 3y ago
        1 TB = 930 GiB The value difference is not because of invisible buffers! Marketing material and usable OS space are measured in different units, but the usable amount of bits in each value is exactly the same.
        3y ago
ilyt 3y ago
Sequential reads will still generally be faster at least for small ones, just that NVMe are fast enough that this rarely matters
deleted 3y ago
[deleted]
loeg 3y ago
> So sequential I/O is perhaps half-dead; there's no need to optimize layouts for sequential reads any more. Yes and no. Large contiguous IOs are still faster to read than a bunch of random small sectors. You generally want your frequently read files/blobs to be split into as few operations as possible. https://dl.acm.org/doi/abs/10.1145/3477132.3483593 https://dl.acm.org/doi/abs/10.1145/3477132.3483593
- kevincox 3y ago
  This bears a strong resemblance to RAM and caches where cache lines are relatively large so you want to optimize your algorithms to touch as few "cache lines" as possible. So even in RAM random IO is slower than sequential IO.
  - loeg 3y ago
    Yeah. Causing additional page accesses can also be expensive due to the TLB impact.
almog 3y ago
To add to that, the penalty for random writes is paid not only with performance but also in durability of the drive.
deleted 3y ago
[deleted]
xenadu02 3y ago
Flash controllers provide an abstraction layer that should turn any write workload into a sequential one, freeing software developers from having to care about it.
- wtallis 3y ago
  That abstraction layer cannot always avoid creating write amplification when you send it writes that are too small.
  - dsr_ 3y ago
    Which is why having a nice chunk of RAM on the disk itself is useful. 2x write queue size x block size is nice.
  - xenadu02 3y ago
    It can but it involves additional complexity in the controller. Some definitely do a much better job than others. Most controllers handle the block to page mapping problem well. They just mark the old block in the map as dead and write a new block on a new page. GC will later erase pages, hopefully prioritizing pages with the most dead blocks (subject to wear-leveling concerns). But that same concept can be extended to partial block writes. It complicates the read process but there is no reason the controller can't coalesce multiple partial block writes into special partial update blocks and update the map accordingly. Basically write a record saying byte range (X,Y) was overwritten with just those bytes - no need to read the old block, merge the change, and write the whole block back out. Obviously you need to handle things like read chain thresholds (too many partially overlapping updates). Let the GC handle merging the partial updates into the full block during idle periods. Again prioritize doing this for blocks on pages that are going to be erased - you had to rewrite those blocks anyway, turning what would be write amplification into a "free" write. Like I said a lot of controller firmware doesn't bother but that doesn't mean it is impossible or unprofitable to do.
    - wtallis 3y ago
      I didn't say it was impossible, I said it is not always possible. Even the most advanced SSD firmware will cause write amplification for some pathological workloads.
NovaDudely 3y ago
It was Nautilus file manager in Ubuntu that used to drag if you opened a folder with a few thousand files. It was it was essentially doing at least one request for each file when you opened a folder and this was limited by the spin rate of the HDD. So if you had 5,400 files and a 5400RPM drive, yep that would take a full minute to resolve. Eventually the solution really just become - get an SSD because you could throw thousand of requests and the results was fast enough.
- vistu 3y ago
  While I do not doubt that Nautilus did that, I don't think it's a given that a 5400 RPM drive will take exactly a minute to scan through 5400 files.
  - geraldhh 3y ago
    a sensible conclusion
  - NovaDudely 3y ago
    I should not have said that so literally but it was more an example of how the physical driver can limit these things.
- opan 3y ago
  You'll still find that the gtk file managers choke hard when you have hundreds of thousands of files in one directory (such as screenshots), taking minutes to load or even hanging and crashing. Both pcmanfm-qt and dolphin handle them fine and load within 30 seconds. I last tested this a year or two ago and every gtk file manager was roughly equally bad. Made me switch from pcmanfm to pcmanfm-qt (I didn't like dolphin as much).
sliken 3y ago
Sort of. Random writes are fine as long as they are as large as the physical write block size (not the virtual 512b or 4k) and there's a queue depth high enough to keep the drive busy. Unless you are lucky enough to have Intel's (no longer made) Optanes that give nearly 100% of performance with low queue depths. This avoids the ugly read/modify/write, as well as much of the behind the scenes magic for minimizing write amplification and shuffling blocks around.

GauntletWizard 3y ago

There's still a ton of performance benefit that can be gained by keeping your read pipeline full, and you can most easily keep your read pipeline full by making it guessable by doing readahead. It doesn't need to be strictly disk-layout sequential, file-layout sequential will usually do.

Proven 3y ago

Far from it, sequential IO optimization isn't dead. Sequential reads need to be optimized to produce optimal performance for selected workload. That usually means applying a lower level of inline compression. In some cases deduplication works better before, in other after, compression. Sometimes post-process dedupe is more suitable than inline. Then there's erasure coding and data protection methods that are still being optimized for NVMe with sequential workloads, including random workloads which are being sequentialized to work better with latest flash media. I would even say developments in sequential IO are becoming more important than random IO.

jasonwatkinspdx 3y ago

Yes. And the Flash Translation Layer inside modern drives are quite sophisticated. This means you could start out with a write pattern that's causing a lot of copying and reorganization of pages in the back ground, typically because it's doing a lot of unaligned random writes smaller than the erase block size, and initially the performance will be fine. But as the drive wears in, and the capacity fills up, the FTL has less extra space to work with, forcing this garbage collection activity to become increasingly common. So you get performance degradation and premature wearing of the drive in a way that's opaque.

bcrl 3y ago
Short stroking SSDs is a commonly used trick in the toolbag of wise sysadmins for dealing with this kind of workload.
- Waterluvian 3y ago
  I’m guessing that means pretending they’re smaller than they really are?
  - colordrops 3y ago
    Looks like that's the case based on a cursory google search. Seems analogous to only charging batteries up to 80%. I wish companies would factor these issues into their products and leave invisible buffers, but I guess making them cheaper is more important than being consistent and reliable.
    - Waterluvian 3y ago
      Sales and marketing will never leave anything on the table that might make their job easier.
    - wtallis 3y ago
      They do leave invisible buffers, commonly referred to as over provisioning. The discrepancy between GB and GiB is ~7% hiding in plain sight, and then more expensive enterprise drives are commonly sold in (usable) capacities like 960GB and 7.68TB rather than 1TB and 8TB.
      - andrecarini 3y ago
        1 TB = 930 GiB The value difference is not because of invisible buffers! Marketing material and usable OS space are measured in different units, but the usable amount of bits in each value is exactly the same.
        3y ago

ilyt 3y ago

Sequential reads will still generally be faster at least for small ones, just that NVMe are fast enough that this rarely matters

deleted 3y ago

[deleted]

loeg 3y ago

> So sequential I/O is perhaps half-dead; there's no need to optimize layouts for sequential reads any more. Yes and no. Large contiguous IOs are still faster to read than a bunch of random small sectors. You generally want your frequently read files/blobs to be split into as few operations as possible. https://dl.acm.org/doi/abs/10.1145/3477132.3483593 https://dl.acm.org/doi/abs/10.1145/3477132.3483593

kevincox 3y ago
This bears a strong resemblance to RAM and caches where cache lines are relatively large so you want to optimize your algorithms to touch as few "cache lines" as possible. So even in RAM random IO is slower than sequential IO.
- loeg 3y ago
  Yeah. Causing additional page accesses can also be expensive due to the TLB impact.

almog 3y ago

To add to that, the penalty for random writes is paid not only with performance but also in durability of the drive.

deleted 3y ago

[deleted]

xenadu02 3y ago

Flash controllers provide an abstraction layer that should turn any write workload into a sequential one, freeing software developers from having to care about it.

wtallis 3y ago
That abstraction layer cannot always avoid creating write amplification when you send it writes that are too small.
- dsr_ 3y ago
  Which is why having a nice chunk of RAM on the disk itself is useful. 2x write queue size x block size is nice.
- xenadu02 3y ago
  It can but it involves additional complexity in the controller. Some definitely do a much better job than others. Most controllers handle the block to page mapping problem well. They just mark the old block in the map as dead and write a new block on a new page. GC will later erase pages, hopefully prioritizing pages with the most dead blocks (subject to wear-leveling concerns). But that same concept can be extended to partial block writes. It complicates the read process but there is no reason the controller can't coalesce multiple partial block writes into special partial update blocks and update the map accordingly. Basically write a record saying byte range (X,Y) was overwritten with just those bytes - no need to read the old block, merge the change, and write the whole block back out. Obviously you need to handle things like read chain thresholds (too many partially overlapping updates). Let the GC handle merging the partial updates into the full block during idle periods. Again prioritize doing this for blocks on pages that are going to be erased - you had to rewrite those blocks anyway, turning what would be write amplification into a "free" write. Like I said a lot of controller firmware doesn't bother but that doesn't mean it is impossible or unprofitable to do.
  - wtallis 3y ago
    I didn't say it was impossible, I said it is not always possible. Even the most advanced SSD firmware will cause write amplification for some pathological workloads.

NovaDudely 3y ago

It was Nautilus file manager in Ubuntu that used to drag if you opened a folder with a few thousand files. It was it was essentially doing at least one request for each file when you opened a folder and this was limited by the spin rate of the HDD. So if you had 5,400 files and a 5400RPM drive, yep that would take a full minute to resolve. Eventually the solution really just become - get an SSD because you could throw thousand of requests and the results was fast enough.

vistu 3y ago
While I do not doubt that Nautilus did that, I don't think it's a given that a 5400 RPM drive will take exactly a minute to scan through 5400 files.
- geraldhh 3y ago
  a sensible conclusion
- NovaDudely 3y ago
  I should not have said that so literally but it was more an example of how the physical driver can limit these things.
opan 3y ago
You'll still find that the gtk file managers choke hard when you have hundreds of thousands of files in one directory (such as screenshots), taking minutes to load or even hanging and crashing. Both pcmanfm-qt and dolphin handle them fine and load within 30 seconds. I last tested this a year or two ago and every gtk file manager was roughly equally bad. Made me switch from pcmanfm to pcmanfm-qt (I didn't like dolphin as much).

sliken 3y ago

Sort of. Random writes are fine as long as they are as large as the physical write block size (not the virtual 512b or 4k) and there's a queue depth high enough to keep the drive busy. Unless you are lucky enough to have Intel's (no longer made) Optanes that give nearly 100% of performance with low queue depths. This avoids the ugly read/modify/write, as well as much of the behind the scenes magic for minimizing write amplification and shuffling blocks around.