4 ms·

I have not adopted io_uring yet because it isn't clear that it will provide useful performance improvements over linux aio in cases where the disk I/O subsystem

I have not adopted io_uring yet because it isn't clear that it will provide useful performance improvements over linux aio in cases where the disk I/O subsystem is already highly optimized. Where io_uring seems to show a benefit relative to linux aio is more naive software design, which adds a lot of value but is a somewhat different value proposition than has been expressed. For software that is already capable of driving storage hardware at its theoretical limit, the benefit is less immediate and offset by the requirement of having a very recent Linux kernel.

tyingq 7y ago
If I understand the premise right, it should be fewer syscalls per IO. So even if it doesn't improve disk I/O, it might reduce CPU utilization.
- matheusmoreira 7y ago
  It could also be nearly zero system calls per I/O operation. The kernel can poll the submission queue for new entries. This eliminates system call overhead at the cost of higher CPU utilization.
- jandrewrogers 7y ago
  This is true, but for most intentionally optimized storage engines that syscall overhead is below the noise floor in practice, even on NVMe storage. A single core can easily drive gigabytes per second using the old Linux AIO interface. It appears to primarily be an optimization for storage that was not well-optimized to begin with. It is not obvious that it would make high-performance storage engines faster.
shuss 7y ago
For regular files, aio works async only if they are opened in unbuffered mode. I think this is a huge limitation. io_uring on the other hand, can provide a uniform interface for all file descriptors whether they are sockets or regular files. This should be a decent win, IMO.
- wbl 7y ago
  And works is used advisedly. Certain filesystem edge conditions particularly metadata changes due to block allocation can result in blocking behavior.
- jandrewrogers 7y ago
  That was kind of my point. While all of this is true, these are not material limitations for the implementation of high-performance storage engines. For example, using unbuffered file descriptors is a standard design element of databases for performance reasons that remain true. Being able to drive networking over io_uring would be a big advantage but my understanding from people using it is that part is still a bit broken.
  - shuss 7y ago
    True. Have to agree, here. Although one advantage over aio for block I/O that io_uring will still have is to use polling mode to almost completely avoid system calls.
  - g8oz 7y ago
    The ScyllaDB developers wrote up their take here: https://www.scylladb.com/2020/05/05/how-io_uring-and-ebpf-will-revolutionize-programming-in-linux/ https://www.scylladb.com/2020/05/05/how-io_uring-and-ebpf-wi...
    - jabl 7y ago
      Those benchmark results are pretty impressive. In particular, io_uring gets the best performance both when the data is in the page cache and when bypassing the cache.
loeg 7y ago
aio doesn't have facilities for synchronous filesystem metadata operations like open, rename, unlink, etc. If your workload is largely metadata-static, aio is ok. If you need to do even a little filesystem manipulation, io_uring seems like it can provide some benefits.

tyingq 7y ago

If I understand the premise right, it should be fewer syscalls per IO. So even if it doesn't improve disk I/O, it might reduce CPU utilization.

matheusmoreira 7y ago
It could also be nearly zero system calls per I/O operation. The kernel can poll the submission queue for new entries. This eliminates system call overhead at the cost of higher CPU utilization.
jandrewrogers 7y ago
This is true, but for most intentionally optimized storage engines that syscall overhead is below the noise floor in practice, even on NVMe storage. A single core can easily drive gigabytes per second using the old Linux AIO interface. It appears to primarily be an optimization for storage that was not well-optimized to begin with. It is not obvious that it would make high-performance storage engines faster.

shuss 7y ago

For regular files, aio works async only if they are opened in unbuffered mode. I think this is a huge limitation. io_uring on the other hand, can provide a uniform interface for all file descriptors whether they are sockets or regular files. This should be a decent win, IMO.

wbl 7y ago
And works is used advisedly. Certain filesystem edge conditions particularly metadata changes due to block allocation can result in blocking behavior.
jandrewrogers 7y ago
That was kind of my point. While all of this is true, these are not material limitations for the implementation of high-performance storage engines. For example, using unbuffered file descriptors is a standard design element of databases for performance reasons that remain true. Being able to drive networking over io_uring would be a big advantage but my understanding from people using it is that part is still a bit broken.
- shuss 7y ago
  True. Have to agree, here. Although one advantage over aio for block I/O that io_uring will still have is to use polling mode to almost completely avoid system calls.
- g8oz 7y ago
  The ScyllaDB developers wrote up their take here: https://www.scylladb.com/2020/05/05/how-io_uring-and-ebpf-will-revolutionize-programming-in-linux/ https://www.scylladb.com/2020/05/05/how-io_uring-and-ebpf-wi...
  - jabl 7y ago
    Those benchmark results are pretty impressive. In particular, io_uring gets the best performance both when the data is in the page cache and when bypassing the cache.

loeg 7y ago

aio doesn't have facilities for synchronous filesystem metadata operations like open, rename, unlink, etc. If your workload is largely metadata-static, aio is ok. If you need to do even a little filesystem manipulation, io_uring seems like it can provide some benefits.