4 ms·

Io_uring Without Readahead

121 points by porridgeraisin 2mo ago

marginalia_nu 2mo ago
If you're doing contiguous readahead in userspace, why not just use preadv? It'll limit you to doing readahead up until the next resident page, but at least in my experiments in Marginalia's index, preadv beats io_uring in all cases you can use a single preadv call to do the full read.
- vlovich123 2mo ago
  Not sure what you mean. Nothing about preadv lets you indicate you only want to read what's already in the page cache. And io_uring and preadv aren't orthogonal - you can give io_uring a preadv op to do the scattered read instead of issuing separate read OPs although I'm not 100% sure how much of a win that is in practice. Also, I think you misunderstood the blog as it's describing application read-ahead which is what you have to do when using O_DIRECT.
  - marginalia_nu 2mo ago
    So I have a buffer pool with O_DIRECT reads. I implement read-ahead in the application by (optionally) preadv:ing a single read into multiple destination buffers in the pool, leaving them unpinned, since as long as you aren't up against the bandwidth limit of the drive, a larger read is generally as fast as multiple smaller one on modern hardware. I've tried doing this with io_uring as well, but found just eating the preadv syscall cost was faster.
    - lossolo 2mo ago
      > a larger read is generally as fast as multiple smaller one on modern hardware. Not always if by modern you mean NVMe drives. One synchronous preadv() for 256 KiB gives the kernel/device one big request but 16 independent asynchronous 16 KiB reads can be serviced concurrently. So the latter gives the NVMe controller 16 operations it can schedule in parallel. So depending on the workload and hardware, offsets, filesystem and request sizes that can give you lower aggregate latency or higher throughput.
      - marginalia_nu 2mo ago
        If you submit 16 contiguous read requests the system will just merge them into one large read request. Modern SSDs tolerate moderate queue depths very well, but piling on the I/O queue also incurs tail latency jitter unless you're able to ensure the queue depth stays in the moderate range and never goes higher. All else being equal, fewer larger requests is better for I/O latency (though read amplification for the sake of reading more data obviously doesn't help anyone). Though in this scenario, we're mostly comparing the syscall overhead of a single preadv against io_uring bookkeeping for multiple preads, regardless of how you submit the reads they end up being the same operation.
        vlovich123 2mo ago
        Doesn’t the Linux kernel automatically merge requests for contiguous regions into a single request? I’m not sure 16 read I/Os submitted to io_uring for a contiguous 64kib region will behave differently at the disk level than a single 64kib request. There’s a little bit extra work the kernel has to do to merge the requests but that should be negligible. In a poorly written micro benchmark test it’s possible the kernel will fail to coalesce all of them because your submissions aren’t visible all at once (ie it starts submitting requests and doesn’t have an opportunity to merge). Whether that actually is possible to happen requires digging a bit more into the Linux kernel source.
        marginalia_nu 2mo ago
        Yes that is correct. You can look at 'rrqm/s' and '%rrqm' in `iostat -x` to get statistics for how many requests are merged, and `rareq-sz` for the average request size.
      - ComputerGuru 2mo ago
        I think that’s only true on paper; in practice it’ll be true only when you have competing reads (at a thousand-foot view) and it might be possible to algorithmically bundle a portion thereof. It originally let SCSI controllers attached to spinning rust HDDs optimize physical manipulation of the disk heads to optimized queued reads of data in a “traveling salesman” sort of way, but modern nand flash can only internally read a full page at a time (which may be much greater than even the apparent physical sector size) anyway and with a strictly constant cost regardless of the “physical location” of the data on the non-existent platter. Old drives had optimization constraints like higher sequential read speeds at the outside of the platter (more bytes per physical rotation) and extremely pathological cases for data written to the innermost tracks of the platter. Individual requests were much finer-grained and the latency was much more varied, so a request from app/thread X for as little as 512 bytes from one location could be cheaply piggy-backed on an existing request from app/thread Y to read multiple megabytes from a physically proximate source that would otherwise have seriously delayed or starved the queued waiting read while the outstanding request was serviced. In fact, one consistently sees higher bulk IO numbers when using physical media that has been formatted with a large sector size compared to the old 512 byte fixed emulated size. You’d routinely see lower latency and higher IOPs with 4kn (HDDs or SSDs) than you would with 512e disks, even with SCSI or AHCI controllers that featured similar pipelining support to today’s NVME controllers (or even if you place a spinning rust HDD behind NVMe today!).
    - vlovich123 2mo ago
      Do this with io_uring with the preadv syscall. It’ll be the same or faster (faster only if you can do something else while waiting for I/O or you can submit multiple requests simultaneously - a single io_uring will be basically identical)
      - marginalia_nu 2mo ago
        io_uring is significantly faster for certain workloads, but you can't expect that merely running a syscall via io_uring will somehow magically make it faster. Replacing single sycalls with their equivalents in io_uring is generally slower than just making the syscall directly. io_uring still uses syscalls after all. io_uring generally only wins if you can amortize its overhead across multiple simultaneous operations. Implementing readahead would be such a case, except you can accomplish the same amortization with a single preadv instead, which again turns into a single syscall for multiple reads.
        vlovich123 2mo ago
        That’s not the only case. A read call makes your thread unable to do anything for the duration of the read. Io_uring lets that same thread continue to handle other requests which themselves might generate more I/O that gets amortized. The fair comparison isn’t 1 syscall on a single thread processing 1 task against io_uring. That would be insane because you clearly don’t have any performance requirements in such a workload already. The closest realistic equivalent would be using Tokio’s spawn_blocking to do that 1 syscall vs doing that syscall in io_uring. It’s probably still more efficient if your benchmark literally is the cost of 1 syscall at a time but not by as much and io_uring in poll mode doesn’t even enter the kernel so it can actually outperform the syscall offloaded to a background thread (even though yes under the hood it’s the same kernel code).
        marginalia_nu 2mo ago
        DBMS workloads aren't asynchronous like that though. There generally isn't anything you can do while you wait for the buffer pool to execute a read, since what you want to do next depends on the data being read.
  - wahern 2mo ago
    > Nothing about preadv lets you indicate you only want to read what's already in the page cache. preadv2 + RWF_NOWAIT
    - vlovich123 2mo ago
      That’s not going to work for read ahead unless you’re really only requesting 1 page at a time which is terrible for performance across the board. That syscall will return EAGAIN if any data is missing to satisfy the read. Meaning if you issue a request for 128kib, it’ll error even if it has 124kib. There really aren’t optimal user space APIs. What you want ideally is a way to register a callback on a memory region so that the kernel is able to treat it like page cache and then on a fault you regenerate it using said callback. Thats even more efficient because usually you have to do some processing of the on disk representation (eg if it’s compressed).
- Asmod4n 2mo ago
  I found out that using mmap and just telling uring to read form there to beat anything else.
  - marginalia_nu 2mo ago
    Depends a lot on the memory pressure. If you can be fairly certain the data is (or will be) resident in memory, mmap is basically unbeatable. If you can't (because the data is larger than RAM or there's other stuff competing for RAM), mmap can have gnarly system-wide performance implications[1]. [1] Mandatory mmap=poop-emoji link: https://db.cs.cmu.edu/mmap-cidr2022/ https://db.cs.cmu.edu/mmap-cidr2022/
    - Asmod4n 2mo ago
      In my case: its a PROT_READ, MAP_PRIVAT map and i cannot get a SIGBUS since i tell the kernel to handle it all for me, thanks to liburing, instead i get a short send in that case.
  - 0c3ca83 2mo ago
    Why would you use uring to read from an mmap? Couldn't you just memcpy?
    - jeffbee 2mo ago
      Maybe they meant using IORING_OP_MADVISE with MADV_WILLNEED to bring ranges into memory?
      - marginalia_nu 2mo ago
        Though at that point you can just use FADV_WILLNEED.
        jeffbee 2mo ago
        I'm just steelmanning here ¯\_(ツ)_/¯
amluto 2mo ago
I’m curious why the choice is between syscalls and, specifically, io_uring with O_DIRECT. AFAIK Turso is like SQLite and supports multiple processes accessing the same database, and I would expect buffering to be a huge win in some workloads. What’s wrong with io_uring without direct? There’s also the middle ground of RWF_DONTCACHE.
- adrian_b 2mo ago
  You can use io_uring very well with sequential files, where the kernel will do read-ahead for you, possibly helped by you with fadvise. O_DIRECT is to be used with files that will be accessed in a random order (random meaning that the order is not predictable by the kernel) and with buffer sizes per access great enough that you want to avoid their copying between kernel and application (e.g. at least a few kByte per access). Whenever O_DIRECT is used, the programmer takes responsibility to implement an adequate form of read-ahead, based on the access pattern that is predicted for the application, and which cannot be guessed by the kernel. If the programmer did not implement read-ahead, like it was the case before doing the update described in TFA, that was an incompletely written program. One should not enable O_DIRECT without a complete implementation for its requirements, as that can lead only to lower performance than standard I/O.
weatherlite 2mo ago
[flagged]
gdiskcfg 2mo ago
[dead]
ComputerGuru 2mo ago
Interesting article but it gave me a bit of a panic attack. Benchmarking (with TPC or otherwise) is NOT the way to determine the correct approach here; that is strictly only to be used for databases (typically RDBMS) effectively “owning” the complete hardware they are running on. An embedded database might be used in that manner if it’s operating as the backend for a pure crud application that performs ~zero server side rendering, parsing, validation, etc and is essentially just an async http-to-SQLite interface. But more likely than not, an embedded db will be used and deployed on machines (not necessarily even servers) serving many a purpose, and need to perform best both within the confines of the resources available to the machine and in relative terms, necessarily making tradeoffs that might sacrifice performance for “value” in terms of CPU or memory usage. This isn’t just with regards to benchmarking, it’s an essential consideration *any* time you are taking ownership of the cache away from the kernel, which is the only piece in the stack that has viability into the global state and can be trusted to give back memory under pressure to ensure everything plays nice together. It’s not limited to just databases or even just memory, for example FreeBSD has had greater than its fair share of issues that trace back to the ZFS having a separate cache from the kernel (despite the much tighter integration between the two and presence of various mechanisms to address pathological cases). You can also refer to any comparison or benchmark between the use of spinlocks vs mutexes: spin locks consistently perform better in/on (micro)benchmarks but are almost always actually the worse choice in the grand scheme of things because the benchmarks falsely assume complete and uncontended ownership over system resources. This isn’t even io_uring specific and I’m hardly the first to bring this up in the context of O_DIRECT.
- pbowyer 2mo ago
  What would be the correct way to do benchmarking/profiling here? I ask as I've started to run some for Vinyl cache to see what performance issues I can find, and I've quickly learned just how hard it is to get good, repeatable benchmarks that isolate the right thing.
  - ComputerGuru 2mo ago
    No easy answer! You’d have to have a myriad of different test suites that as closely resemble your real world usage as possible. Or ship it behind a gate and let your users give you real world feedback if you’re at SQLite (not turso) scale - they’ve done that with a lot of ambiguous optimizations.
andddd 2mo ago
[dead]
ErroneousBosh 2mo ago
This has scrolled past in my feed and every time I've read it as "Io_uring without Radiohead", and I mean you could but what would be the point?
brianhliou 2mo ago
[flagged]

121 points by porridgeraisin 2mo ago

If you're doing contiguous readahead in userspace, why not just use preadv? It'll limit you to doing readahead up until the next resident page, but at least in my experiments in Marginalia's index, preadv beats io_uring in all cases you can use a single preadv call to do the full read.

vlovich123 2mo ago
Not sure what you mean. Nothing about preadv lets you indicate you only want to read what's already in the page cache. And io_uring and preadv aren't orthogonal - you can give io_uring a preadv op to do the scattered read instead of issuing separate read OPs although I'm not 100% sure how much of a win that is in practice. Also, I think you misunderstood the blog as it's describing application read-ahead which is what you have to do when using O_DIRECT.
- marginalia_nu 2mo ago
  So I have a buffer pool with O_DIRECT reads. I implement read-ahead in the application by (optionally) preadv:ing a single read into multiple destination buffers in the pool, leaving them unpinned, since as long as you aren't up against the bandwidth limit of the drive, a larger read is generally as fast as multiple smaller one on modern hardware. I've tried doing this with io_uring as well, but found just eating the preadv syscall cost was faster.
  - lossolo 2mo ago
    > a larger read is generally as fast as multiple smaller one on modern hardware. Not always if by modern you mean NVMe drives. One synchronous preadv() for 256 KiB gives the kernel/device one big request but 16 independent asynchronous 16 KiB reads can be serviced concurrently. So the latter gives the NVMe controller 16 operations it can schedule in parallel. So depending on the workload and hardware, offsets, filesystem and request sizes that can give you lower aggregate latency or higher throughput.
    - marginalia_nu 2mo ago
      If you submit 16 contiguous read requests the system will just merge them into one large read request. Modern SSDs tolerate moderate queue depths very well, but piling on the I/O queue also incurs tail latency jitter unless you're able to ensure the queue depth stays in the moderate range and never goes higher. All else being equal, fewer larger requests is better for I/O latency (though read amplification for the sake of reading more data obviously doesn't help anyone). Though in this scenario, we're mostly comparing the syscall overhead of a single preadv against io_uring bookkeeping for multiple preads, regardless of how you submit the reads they end up being the same operation.
      - vlovich123 2mo ago
        Doesn’t the Linux kernel automatically merge requests for contiguous regions into a single request? I’m not sure 16 read I/Os submitted to io_uring for a contiguous 64kib region will behave differently at the disk level than a single 64kib request. There’s a little bit extra work the kernel has to do to merge the requests but that should be negligible. In a poorly written micro benchmark test it’s possible the kernel will fail to coalesce all of them because your submissions aren’t visible all at once (ie it starts submitting requests and doesn’t have an opportunity to merge). Whether that actually is possible to happen requires digging a bit more into the Linux kernel source.
        marginalia_nu 2mo ago
        Yes that is correct. You can look at 'rrqm/s' and '%rrqm' in `iostat -x` to get statistics for how many requests are merged, and `rareq-sz` for the average request size.
    - ComputerGuru 2mo ago
      I think that’s only true on paper; in practice it’ll be true only when you have competing reads (at a thousand-foot view) and it might be possible to algorithmically bundle a portion thereof. It originally let SCSI controllers attached to spinning rust HDDs optimize physical manipulation of the disk heads to optimized queued reads of data in a “traveling salesman” sort of way, but modern nand flash can only internally read a full page at a time (which may be much greater than even the apparent physical sector size) anyway and with a strictly constant cost regardless of the “physical location” of the data on the non-existent platter. Old drives had optimization constraints like higher sequential read speeds at the outside of the platter (more bytes per physical rotation) and extremely pathological cases for data written to the innermost tracks of the platter. Individual requests were much finer-grained and the latency was much more varied, so a request from app/thread X for as little as 512 bytes from one location could be cheaply piggy-backed on an existing request from app/thread Y to read multiple megabytes from a physically proximate source that would otherwise have seriously delayed or starved the queued waiting read while the outstanding request was serviced. In fact, one consistently sees higher bulk IO numbers when using physical media that has been formatted with a large sector size compared to the old 512 byte fixed emulated size. You’d routinely see lower latency and higher IOPs with 4kn (HDDs or SSDs) than you would with 512e disks, even with SCSI or AHCI controllers that featured similar pipelining support to today’s NVME controllers (or even if you place a spinning rust HDD behind NVMe today!).
  - vlovich123 2mo ago
    Do this with io_uring with the preadv syscall. It’ll be the same or faster (faster only if you can do something else while waiting for I/O or you can submit multiple requests simultaneously - a single io_uring will be basically identical)
    - marginalia_nu 2mo ago
      io_uring is significantly faster for certain workloads, but you can't expect that merely running a syscall via io_uring will somehow magically make it faster. Replacing single sycalls with their equivalents in io_uring is generally slower than just making the syscall directly. io_uring still uses syscalls after all. io_uring generally only wins if you can amortize its overhead across multiple simultaneous operations. Implementing readahead would be such a case, except you can accomplish the same amortization with a single preadv instead, which again turns into a single syscall for multiple reads.
      - vlovich123 2mo ago
        That’s not the only case. A read call makes your thread unable to do anything for the duration of the read. Io_uring lets that same thread continue to handle other requests which themselves might generate more I/O that gets amortized. The fair comparison isn’t 1 syscall on a single thread processing 1 task against io_uring. That would be insane because you clearly don’t have any performance requirements in such a workload already. The closest realistic equivalent would be using Tokio’s spawn_blocking to do that 1 syscall vs doing that syscall in io_uring. It’s probably still more efficient if your benchmark literally is the cost of 1 syscall at a time but not by as much and io_uring in poll mode doesn’t even enter the kernel so it can actually outperform the syscall offloaded to a background thread (even though yes under the hood it’s the same kernel code).
        marginalia_nu 2mo ago
        DBMS workloads aren't asynchronous like that though. There generally isn't anything you can do while you wait for the buffer pool to execute a read, since what you want to do next depends on the data being read.
- wahern 2mo ago
  > Nothing about preadv lets you indicate you only want to read what's already in the page cache. preadv2 + RWF_NOWAIT
  - vlovich123 2mo ago
    That’s not going to work for read ahead unless you’re really only requesting 1 page at a time which is terrible for performance across the board. That syscall will return EAGAIN if any data is missing to satisfy the read. Meaning if you issue a request for 128kib, it’ll error even if it has 124kib. There really aren’t optimal user space APIs. What you want ideally is a way to register a callback on a memory region so that the kernel is able to treat it like page cache and then on a fault you regenerate it using said callback. Thats even more efficient because usually you have to do some processing of the on disk representation (eg if it’s compressed).
Asmod4n 2mo ago
I found out that using mmap and just telling uring to read form there to beat anything else.
- marginalia_nu 2mo ago
  Depends a lot on the memory pressure. If you can be fairly certain the data is (or will be) resident in memory, mmap is basically unbeatable. If you can't (because the data is larger than RAM or there's other stuff competing for RAM), mmap can have gnarly system-wide performance implications[1]. [1] Mandatory mmap=poop-emoji link: https://db.cs.cmu.edu/mmap-cidr2022/ https://db.cs.cmu.edu/mmap-cidr2022/
  - Asmod4n 2mo ago
    In my case: its a PROT_READ, MAP_PRIVAT map and i cannot get a SIGBUS since i tell the kernel to handle it all for me, thanks to liburing, instead i get a short send in that case.
- 0c3ca83 2mo ago
  Why would you use uring to read from an mmap? Couldn't you just memcpy?
  - jeffbee 2mo ago
    Maybe they meant using IORING_OP_MADVISE with MADV_WILLNEED to bring ranges into memory?
    - marginalia_nu 2mo ago
      Though at that point you can just use FADV_WILLNEED.
      - jeffbee 2mo ago
        I'm just steelmanning here ¯\_(ツ)_/¯

amluto 2mo ago

I’m curious why the choice is between syscalls and, specifically, io_uring with O_DIRECT. AFAIK Turso is like SQLite and supports multiple processes accessing the same database, and I would expect buffering to be a huge win in some workloads. What’s wrong with io_uring without direct? There’s also the middle ground of RWF_DONTCACHE.

adrian_b 2mo ago
You can use io_uring very well with sequential files, where the kernel will do read-ahead for you, possibly helped by you with fadvise. O_DIRECT is to be used with files that will be accessed in a random order (random meaning that the order is not predictable by the kernel) and with buffer sizes per access great enough that you want to avoid their copying between kernel and application (e.g. at least a few kByte per access). Whenever O_DIRECT is used, the programmer takes responsibility to implement an adequate form of read-ahead, based on the access pattern that is predicted for the application, and which cannot be guessed by the kernel. If the programmer did not implement read-ahead, like it was the case before doing the update described in TFA, that was an incompletely written program. One should not enable O_DIRECT without a complete implementation for its requirements, as that can lead only to lower performance than standard I/O.

weatherlite 2mo ago

[flagged]

gdiskcfg 2mo ago

[dead]

ComputerGuru 2mo ago

Interesting article but it gave me a bit of a panic attack. Benchmarking (with TPC or otherwise) is NOT the way to determine the correct approach here; that is strictly only to be used for databases (typically RDBMS) effectively “owning” the complete hardware they are running on. An embedded database might be used in that manner if it’s operating as the backend for a pure crud application that performs ~zero server side rendering, parsing, validation, etc and is essentially just an async http-to-SQLite interface. But more likely than not, an embedded db will be used and deployed on machines (not necessarily even servers) serving many a purpose, and need to perform best both within the confines of the resources available to the machine and in relative terms, necessarily making tradeoffs that might sacrifice performance for “value” in terms of CPU or memory usage. This isn’t just with regards to benchmarking, it’s an essential consideration *any* time you are taking ownership of the cache away from the kernel, which is the only piece in the stack that has viability into the global state and can be trusted to give back memory under pressure to ensure everything plays nice together. It’s not limited to just databases or even just memory, for example FreeBSD has had greater than its fair share of issues that trace back to the ZFS having a separate cache from the kernel (despite the much tighter integration between the two and presence of various mechanisms to address pathological cases). You can also refer to any comparison or benchmark between the use of spinlocks vs mutexes: spin locks consistently perform better in/on (micro)benchmarks but are almost always actually the worse choice in the grand scheme of things because the benchmarks falsely assume complete and uncontended ownership over system resources. This isn’t even io_uring specific and I’m hardly the first to bring this up in the context of O_DIRECT.

pbowyer 2mo ago
What would be the correct way to do benchmarking/profiling here? I ask as I've started to run some for Vinyl cache to see what performance issues I can find, and I've quickly learned just how hard it is to get good, repeatable benchmarks that isolate the right thing.
- ComputerGuru 2mo ago
  No easy answer! You’d have to have a myriad of different test suites that as closely resemble your real world usage as possible. Or ship it behind a gate and let your users give you real world feedback if you’re at SQLite (not turso) scale - they’ve done that with a lot of ambiguous optimizations.

andddd 2mo ago

[dead]

ErroneousBosh 2mo ago

This has scrolled past in my feed and every time I've read it as "Io_uring without Radiohead", and I mean you could but what would be the point?

brianhliou 2mo ago

[flagged]