6 ms·

Axboe Achieves 8M IOPS Per-Core with Newest Linux Optimization Patches

111 points by marcodiego 5y ago

jpgvm 5y ago
I remember when he first hit 1M when he was at Fusion-IO. Man how far things have come since those days.
Aissen 5y ago
That's 125 ns per IO, on a single CPU core, with two devices. Mind boggling.
- megous 5y ago
  IO is 512B sectors here? Or 4KiB? EDIT: 512, looking at the screenshot..
- guerrilla 5y ago
  Faster than DRAM from the 90's.
  - spicybright 5y ago
    I'd hope so lol
  - formerly_proven 5y ago
    The access time has actually not changed a lot, it's just that transferring the smallest read a contemporary processor would issue - one cache line (~64 bytes) - at somewhere between 500-1000 MB/s takes around 100 ns already. That transfer-related latency has been reduced drastically.
    - guerrilla 5y ago
      Didn't /RAS timing drop from like 120ns to 10ns too? That's what I was thinking about. Then DDR, and that bus size is also 16 times wider.
      - formerly_proven 5y ago
        Wasn't tRAS always around 2-3x tCL?
  - terafo 5y ago
    Modern SSDs generally have more bandwidth than DDR2.
- devit 5y ago
  At around 4 GHz and 2 IPC, it's about 1000 CPU instructions per I/O. Assuming you have an hardware DMA ring buffer for I/O that is directly mapped in userspace, the only thing that is really needed is to write the operation type, size, disk position and memory position to the ring buffer, update the buffer position and check for flush, doable in around 8 CISC instructions (plus the slowpath), so around 100x inefficient. Without the direct mapped ring buffer and with a filesystem, you need a kernel to translate from uring to the hardware ring buffer, and here it still seems around 10x inefficient as around 100 instructions should be enough to do the translation (assuming pages already mapped in the IOMMU, that you have the file block map in cache, and that the whole system is architected to maximize the efficiency of this operation).
  - throwawaylinux 5y ago
    > Assuming you have an hardware DMA ring buffer for I/O that is directly mapped in userspace, the only thing that is really needed is to write the operation type, size, disk position and memory position to the ring buffer, update the buffer position and check for flush, doable in around 8 CISC instructions (plus the slowpath), so around 100x inefficient. This type of thing is not measured in instructions anymore, it's measured in off-board operations and latencies. Cache misses, latencies to poke MMIO and get back interrupts if necessary, DMA transfer to complete, device access time. In this case it seems the hardware is theoretically capable of about 12M so the core mostly be just waiting for that.
  - Flow 5y ago
    Will DMA from external devices pass L3 or some other CPU cache? Or are all accesses to the freshy DMAed-to memory be cold
- anarazel 5y ago
  It's worth noting that there's some batching involved, so it's not 125ns for processing one io, then again 125ns for the next. Both combining some of the work, and superscalar execution is necessary to reach this high numbers...
  - benlwalker 5y ago
    To expand on the batching details, I haven't seen exactly what he's doing here, but historically the numbers quoted are from benchmarks that submit something like 128 total queue depth in alternating batches of 64. The disks hit max performance with a much lower queue depth than 64. So it's very unrealistic perfect batching that you'll never see in a real application. The workload is only a useful tool for optimizing the code, which is exactly what he's using it for.
    - wtallis 5y ago
      The Intel Optane drives he's testing with hit max performance at pretty low queue depths, but flash-based SSDs with similar throughput will require those high queue depths.
    - anarazel 5y ago
      I think you need batching for the nvme doorbell alone at anything close to these kinds of rates... For some workloads it's not that hard to have that deep queues. What's harder is to know when to use them and when not. There's really not enough information available to make any of this self-tuning.
junon 5y ago
I know this will be down voted but I've learned to be a bit skeptical of Axboe's monumental claims. There have been some unfair benchmarks posted in the past surrounding io_uring that were called out by people on the liburing repositories. Take these with a grain of salt - his results are notoriously very difficult to reproduce, even on identical hardware. Nevertheless, io_uring is certainly the better design and I'm happy to see progress still being made.
- Datagenerator 5y ago
  Nitpicking aside he has twenty years of Linux kernel development experience and does great things for everyones benefit. Congratulations Jens Axboe!
  - junon 5y ago
    This wasn't nitpicking. It was claiming a nontrivial speedup over epoll for identical test cases. What was the point of your comment?
    - yuffaduffa 5y ago
      What was the point of yours? “Take benchmarks with a grain of salt” is like saying refrigerating food is important or, more appropriately, that it’s important to verify surprising claims. (You made it a personal observation for some reason, but your whole point is still about as insightful as both of those.) The person you’re going after here probably felt compelled to counter the needless personal nature of your remarks. It’s difficult to experimentally verify relativity but we don’t criticize Einstein as a result.
      - junon 5y ago
        Comparing very verifiable, applied science to very theoretical science is a strawman. Science doesn't care about feelings or tenure, so while his experience is relevant, it does not excuse the provably inflated performance figures that have been boasted historically.
reacharavindh 5y ago
Does anyone know whether these optimisations in block layer and iouring benefit major IO hitters like PostgreSQL, ZFS, NFS writes etc? Will it take several years and monumental effort before they adopt iouring first?
- wtallis 5y ago
  io_uring is an improved interface between userspace and the kernel, so it doesn't provide any direct benefits to in-kernel filesystem operations, and the performance benefits io_uring does provide should be largely filesystem-agnostic. That said, some of these recent optimizations may be low enough in the kernel's io stack to also benefit io originating within the kernel itself. Userspace applications that already have some support for asynchronous disk IO (either through the old libaio APIs or as a cleanly-abstracted thread pool) should be able to switch to using io_uring as their backend without too much trouble, and reap the benefits of async that actually works reliably (if switching from libaio) and with vastly lower overhead (if switching from a thread pool). Databases like PostgreSQL were some of the few applications that attempted to deal with the limitations of libaio, but I'm not sure how close they are to having a production-quality io_uring backend.
  - sumtechguy 5y ago
    From an earlier article https://www.phoronix.com/scan.php?page=news_item&px=Linux-Approaching-7M-IOPS https://www.phoronix.com/scan.php?page=news_item&px=Linux-Ap... "His patches pushing the greater performance have been changes to the block code, NVMe, multi-queue blk-mq, and IO_uring." https://git.kernel.dk/cgit/linux-block/log/?h=perf-wip https://git.kernel.dk/cgit/linux-block/log/?h=perf-wip So it looks like he is playing with a good portion of the IO block stack with a very recent concentration on io_uring. So maybe some of it?...
egberts1 5y ago
I look forward to a faster `cp` command.
- the8472 5y ago
  io_uring certainly could help there, but (conditional on using SSDs) even switching to something parallel like xcp or fcp would speed things up compared to single-threaded gnu cp.
- kerneltrap 5y ago
  The fastest `cp` is actually doing no data copies at all (relying on copy-on-write), on filesystems with reflink support. Incidentally, coreutils v9.0 cp switched to doing reflinks by default [1], so there's already a faster cp. [1] https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=25725f9d41735d176d73a757430739fb71c7d043 https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=2...
  - egberts1 5y ago
    True. But, but, … BUT when it comes to files found on differing (physical/logical/virtual) disk partitions, ye gonna need that block copying mechanism within ‘cp’.
    - kerneltrap 5y ago
      Not necessarily, while reflink works only within the boundaries of the same filesystem there are other optimizations that can operate across filesystems (even networked ones). For example copy_file_range() [1] allows for copy offloading (depending on the underlying fs/storage), which was incidentally also added to coreutils v9.0 [2]. There are also optimizations on other layers, e.g. transparent compression (e.g. btrfs) and online block-level dedup (e.g. dm-vdo), that can aid in minimizing and/or avoiding actual block copies. When all else fails, cp will anyway fallback to buffered IO, so it won't be bound by the rate of IO (as long as writeback can keep up with memory pressure). Of course improvements on all layers are always welcome and there are certainly workloads that can benefit from an optimized raw throughput, low-latency and scalability. Demanding workloads are typically using dedicated IO stacks and bypass the kernel altogether (e.g SPDK), but the amazing efforts of Axboe and everyone else in the kernel community have been continuously bringing linux up to par in terms of IO performance. [1] https://man7.org/linux/man-pages/man2/copy_file_range.2.html https://man7.org/linux/man-pages/man2/copy_file_range.2.html [2] https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=4b04a0c3b792d27909670a81d21f2a3b3e0ea563 https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=4...
tomc1985 5y ago
I keep seeing Axboe as a typo'd version of Adobe and it's really bugging me edit- It's someone's name, I thought it was a company or product
- noir_lord 5y ago
  It is a wlel kwnon pohnnomeen taht you can sawp all the ltteers in a wrod and as lnog as the fsrit and lsat is ccrreot it wlil sitll be cibemreplohnse. https://www.mrc-cbu.cam.ac.uk/people/matt.davis/cmabridge/ https://www.mrc-cbu.cam.ac.uk/people/matt.davis/cmabridge/
  - tomc1985 5y ago
    It's wild that I was able to read that in my head at full speed, with the correct word pronunciation (except the last word... wonder what the upper limit on that is anyway?)
    - noir_lord 5y ago
      Useful reminder that the human brain is a pattern recognition machine par excellence.
- cteiosanu 5y ago
  Glad I'm not the only one.
terafo 5y ago
He just hit 9M https://twitter.com/axboe/status/1450188650852065291 https://twitter.com/axboe/status/1450188650852065291
- sliken 5y ago
  Anyone know exactly how to replicate this? I see the command line, but it looks like t/io_ring is maybe a wrapper around fio? Or maybe a binary/git repo so I can compile my own?
  - zerd 5y ago
    Didn't see the t/io_uring script, but the branch they're working on is here https://git.kernel.dk/cgit/linux-block/log/?h=perf-wip https://git.kernel.dk/cgit/linux-block/log/?h=perf-wip
  - kerneltrap 5y ago
    https://git.kernel.dk/cgit/fio/tree/t/io_uring.c https://git.kernel.dk/cgit/fio/tree/t/io_uring.c

111 points by marcodiego 5y ago

jpgvm 5y ago

I remember when he first hit 1M when he was at Fusion-IO. Man how far things have come since those days.

Aissen 5y ago

That's 125 ns per IO, on a single CPU core, with two devices. Mind boggling.

megous 5y ago
IO is 512B sectors here? Or 4KiB? EDIT: 512, looking at the screenshot..
guerrilla 5y ago
Faster than DRAM from the 90's.
- spicybright 5y ago
  I'd hope so lol
- formerly_proven 5y ago
  The access time has actually not changed a lot, it's just that transferring the smallest read a contemporary processor would issue - one cache line (~64 bytes) - at somewhere between 500-1000 MB/s takes around 100 ns already. That transfer-related latency has been reduced drastically.
  - guerrilla 5y ago
    Didn't /RAS timing drop from like 120ns to 10ns too? That's what I was thinking about. Then DDR, and that bus size is also 16 times wider.
    - formerly_proven 5y ago
      Wasn't tRAS always around 2-3x tCL?
- terafo 5y ago
  Modern SSDs generally have more bandwidth than DDR2.
devit 5y ago
At around 4 GHz and 2 IPC, it's about 1000 CPU instructions per I/O. Assuming you have an hardware DMA ring buffer for I/O that is directly mapped in userspace, the only thing that is really needed is to write the operation type, size, disk position and memory position to the ring buffer, update the buffer position and check for flush, doable in around 8 CISC instructions (plus the slowpath), so around 100x inefficient. Without the direct mapped ring buffer and with a filesystem, you need a kernel to translate from uring to the hardware ring buffer, and here it still seems around 10x inefficient as around 100 instructions should be enough to do the translation (assuming pages already mapped in the IOMMU, that you have the file block map in cache, and that the whole system is architected to maximize the efficiency of this operation).
- throwawaylinux 5y ago
  > Assuming you have an hardware DMA ring buffer for I/O that is directly mapped in userspace, the only thing that is really needed is to write the operation type, size, disk position and memory position to the ring buffer, update the buffer position and check for flush, doable in around 8 CISC instructions (plus the slowpath), so around 100x inefficient. This type of thing is not measured in instructions anymore, it's measured in off-board operations and latencies. Cache misses, latencies to poke MMIO and get back interrupts if necessary, DMA transfer to complete, device access time. In this case it seems the hardware is theoretically capable of about 12M so the core mostly be just waiting for that.
- Flow 5y ago
  Will DMA from external devices pass L3 or some other CPU cache? Or are all accesses to the freshy DMAed-to memory be cold
anarazel 5y ago
It's worth noting that there's some batching involved, so it's not 125ns for processing one io, then again 125ns for the next. Both combining some of the work, and superscalar execution is necessary to reach this high numbers...
- benlwalker 5y ago
  To expand on the batching details, I haven't seen exactly what he's doing here, but historically the numbers quoted are from benchmarks that submit something like 128 total queue depth in alternating batches of 64. The disks hit max performance with a much lower queue depth than 64. So it's very unrealistic perfect batching that you'll never see in a real application. The workload is only a useful tool for optimizing the code, which is exactly what he's using it for.
  - wtallis 5y ago
    The Intel Optane drives he's testing with hit max performance at pretty low queue depths, but flash-based SSDs with similar throughput will require those high queue depths.
  - anarazel 5y ago
    I think you need batching for the nvme doorbell alone at anything close to these kinds of rates... For some workloads it's not that hard to have that deep queues. What's harder is to know when to use them and when not. There's really not enough information available to make any of this self-tuning.

junon 5y ago

I know this will be down voted but I've learned to be a bit skeptical of Axboe's monumental claims. There have been some unfair benchmarks posted in the past surrounding io_uring that were called out by people on the liburing repositories. Take these with a grain of salt - his results are notoriously very difficult to reproduce, even on identical hardware. Nevertheless, io_uring is certainly the better design and I'm happy to see progress still being made.

Datagenerator 5y ago
Nitpicking aside he has twenty years of Linux kernel development experience and does great things for everyones benefit. Congratulations Jens Axboe!
- junon 5y ago
  This wasn't nitpicking. It was claiming a nontrivial speedup over epoll for identical test cases. What was the point of your comment?
  - yuffaduffa 5y ago
    What was the point of yours? “Take benchmarks with a grain of salt” is like saying refrigerating food is important or, more appropriately, that it’s important to verify surprising claims. (You made it a personal observation for some reason, but your whole point is still about as insightful as both of those.) The person you’re going after here probably felt compelled to counter the needless personal nature of your remarks. It’s difficult to experimentally verify relativity but we don’t criticize Einstein as a result.
    - junon 5y ago
      Comparing very verifiable, applied science to very theoretical science is a strawman. Science doesn't care about feelings or tenure, so while his experience is relevant, it does not excuse the provably inflated performance figures that have been boasted historically.

reacharavindh 5y ago

Does anyone know whether these optimisations in block layer and iouring benefit major IO hitters like PostgreSQL, ZFS, NFS writes etc? Will it take several years and monumental effort before they adopt iouring first?

wtallis 5y ago
io_uring is an improved interface between userspace and the kernel, so it doesn't provide any direct benefits to in-kernel filesystem operations, and the performance benefits io_uring does provide should be largely filesystem-agnostic. That said, some of these recent optimizations may be low enough in the kernel's io stack to also benefit io originating within the kernel itself. Userspace applications that already have some support for asynchronous disk IO (either through the old libaio APIs or as a cleanly-abstracted thread pool) should be able to switch to using io_uring as their backend without too much trouble, and reap the benefits of async that actually works reliably (if switching from libaio) and with vastly lower overhead (if switching from a thread pool). Databases like PostgreSQL were some of the few applications that attempted to deal with the limitations of libaio, but I'm not sure how close they are to having a production-quality io_uring backend.
- sumtechguy 5y ago
  From an earlier article https://www.phoronix.com/scan.php?page=news_item&px=Linux-Approaching-7M-IOPS https://www.phoronix.com/scan.php?page=news_item&px=Linux-Ap... "His patches pushing the greater performance have been changes to the block code, NVMe, multi-queue blk-mq, and IO_uring." https://git.kernel.dk/cgit/linux-block/log/?h=perf-wip https://git.kernel.dk/cgit/linux-block/log/?h=perf-wip So it looks like he is playing with a good portion of the IO block stack with a very recent concentration on io_uring. So maybe some of it?...

egberts1 5y ago

I look forward to a faster `cp` command.

the8472 5y ago
io_uring certainly could help there, but (conditional on using SSDs) even switching to something parallel like xcp or fcp would speed things up compared to single-threaded gnu cp.
kerneltrap 5y ago
The fastest `cp` is actually doing no data copies at all (relying on copy-on-write), on filesystems with reflink support. Incidentally, coreutils v9.0 cp switched to doing reflinks by default [1], so there's already a faster cp. [1] https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=25725f9d41735d176d73a757430739fb71c7d043 https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=2...
- egberts1 5y ago
  True. But, but, … BUT when it comes to files found on differing (physical/logical/virtual) disk partitions, ye gonna need that block copying mechanism within ‘cp’.
  - kerneltrap 5y ago
    Not necessarily, while reflink works only within the boundaries of the same filesystem there are other optimizations that can operate across filesystems (even networked ones). For example copy_file_range() [1] allows for copy offloading (depending on the underlying fs/storage), which was incidentally also added to coreutils v9.0 [2]. There are also optimizations on other layers, e.g. transparent compression (e.g. btrfs) and online block-level dedup (e.g. dm-vdo), that can aid in minimizing and/or avoiding actual block copies. When all else fails, cp will anyway fallback to buffered IO, so it won't be bound by the rate of IO (as long as writeback can keep up with memory pressure). Of course improvements on all layers are always welcome and there are certainly workloads that can benefit from an optimized raw throughput, low-latency and scalability. Demanding workloads are typically using dedicated IO stacks and bypass the kernel altogether (e.g SPDK), but the amazing efforts of Axboe and everyone else in the kernel community have been continuously bringing linux up to par in terms of IO performance. [1] https://man7.org/linux/man-pages/man2/copy_file_range.2.html https://man7.org/linux/man-pages/man2/copy_file_range.2.html [2] https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=4b04a0c3b792d27909670a81d21f2a3b3e0ea563 https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=4...

tomc1985 5y ago

I keep seeing Axboe as a typo'd version of Adobe and it's really bugging me edit- It's someone's name, I thought it was a company or product

noir_lord 5y ago
It is a wlel kwnon pohnnomeen taht you can sawp all the ltteers in a wrod and as lnog as the fsrit and lsat is ccrreot it wlil sitll be cibemreplohnse. https://www.mrc-cbu.cam.ac.uk/people/matt.davis/cmabridge/ https://www.mrc-cbu.cam.ac.uk/people/matt.davis/cmabridge/
- tomc1985 5y ago
  It's wild that I was able to read that in my head at full speed, with the correct word pronunciation (except the last word... wonder what the upper limit on that is anyway?)
  - noir_lord 5y ago
    Useful reminder that the human brain is a pattern recognition machine par excellence.
cteiosanu 5y ago
Glad I'm not the only one.

terafo 5y ago

He just hit 9M https://twitter.com/axboe/status/1450188650852065291 https://twitter.com/axboe/status/1450188650852065291

sliken 5y ago
Anyone know exactly how to replicate this? I see the command line, but it looks like t/io_ring is maybe a wrapper around fio? Or maybe a binary/git repo so I can compile my own?
- zerd 5y ago
  Didn't see the t/io_uring script, but the branch they're working on is here https://git.kernel.dk/cgit/linux-block/log/?h=perf-wip https://git.kernel.dk/cgit/linux-block/log/?h=perf-wip
- kerneltrap 5y ago
  https://git.kernel.dk/cgit/fio/tree/t/io_uring.c https://git.kernel.dk/cgit/fio/tree/t/io_uring.c