4 ms·

Awesome, do you have any data for performance or RAM usage of epoll vs. io_uring in a real-world scenario for BIND9?

fanf2 3y ago
Not yet :-)
destructionator 3y ago
This test doesn't include epoll in any significant fashion at all, and also doesn't apply to network operations. The libuv thing is specifically about disk files, which linux doesn't really support with epoll (epoll just always considers them ready, regardless of if the requested page is in cache or not, meaning it is not helpful for the async case. by contrast, a socket without data immediately available in the kernel's buffer will not show readiness). Since you can't tell if a disk io function would have to actually hit the disk or use the cache on older linuxes, libuv just always dispatches said operations to a worker thread pool which then calls the normal function. When it returns, it sends a message back to the other thread to indicate its completion. If data is already present in the cache, this adds significant overhead vs just calling `read` directly - instead calling `read`, it will post a request to the worker thread queue (which may require synchronization, im not sure exactly how the impl does it tbh), wake the worker thread which calls `read`, which does not block significantly since the data is indeed already present, then it posts an event back to the main thread (which, sure, it is listening for this event on `epoll`, along with whatever else it is listening to, but that's an insignificant detail at this point) which calls the on-complete handler. All that work instead of just transferring the data from the cache to the result. Significant slowdown. If the data is not already present in the cache, it does exactly those same steps, except this time `read` actually does block for some time, so the main thread can continue with other work in the mean time. (Of course, if the main thread has no other work to do, you've gained nothing from this!) Hence why I said in my other comment that you ought not draw any general conclusion from this. This speedup has nothing to do with epoll vs io_uring (except maybe that linux file i/o is not really epoll compatible) and everything to do with libuv's block device implementation specifically. It is totally inapplicable to network loads entirely. edit: in the first version I said "disk file" but it is technically block devices, of which disk files are just the most common example, but of course /dev/zero is a block device that is not a disk file.... and worth noting that /dev/zero is never going to actually load off a disk meaning it is "in the cache" all the time.
- eklitzke 3y ago
  It doesn't really make sense to have an API to tell if reading from a file would block or not because page cache data can be evicted at any time. Even if there was an API that could tell you your next read would be non-blocking, there could be a race where the page cache entry could be evicted before you could do the read.
  - Filligree 3y ago
    That sort of race condition is only relevant if it affects correctness. Here it means a read might be unexpectedly slow, but you'll still get the same data. 99.9% of the time the data will not be evicted prior to the read, so it's still going to be a win overall.
    - sgtnoodle 3y ago
      If you're writing a latency sensitive program, 99.9% might as well be 0% though; you're going to end up with an asynchronous anyway.
    - mort96 3y ago
      Remember that a filesystem read might be a disk read in the background. You don't really want to block the main thread for 2 seconds while sshfs does its thing.
      - mort96 3y ago
        Damn, I obviously meant "network read" not a "disk read". Oh well.
  - viraptor 3y ago
    Depends how you define that API. It could do something like "check if available, and if so, pin it until the next read (which will happen right now)". Not saying that's a good idea, but if such API was needed, it is possible.
    - 3y ago
tomohawk 3y ago
epoll vs io_uring discussion: https://github.com/axboe/liburing/issues/189 https://github.com/axboe/liburing/issues/189

Not yet :-)

This test doesn't include epoll in any significant fashion at all, and also doesn't apply to network operations. The libuv thing is specifically about disk files, which linux doesn't really support with epoll (epoll just always considers them ready, regardless of if the requested page is in cache or not, meaning it is not helpful for the async case. by contrast, a socket without data immediately available in the kernel's buffer will not show readiness). Since you can't tell if a disk io function would have to actually hit the disk or use the cache on older linuxes, libuv just always dispatches said operations to a worker thread pool which then calls the normal function. When it returns, it sends a message back to the other thread to indicate its completion. If data is already present in the cache, this adds significant overhead vs just calling `read` directly - instead calling `read`, it will post a request to the worker thread queue (which may require synchronization, im not sure exactly how the impl does it tbh), wake the worker thread which calls `read`, which does not block significantly since the data is indeed already present, then it posts an event back to the main thread (which, sure, it is listening for this event on `epoll`, along with whatever else it is listening to, but that's an insignificant detail at this point) which calls the on-complete handler. All that work instead of just transferring the data from the cache to the result. Significant slowdown. If the data is not already present in the cache, it does exactly those same steps, except this time `read` actually does block for some time, so the main thread can continue with other work in the mean time. (Of course, if the main thread has no other work to do, you've gained nothing from this!) Hence why I said in my other comment that you ought not draw any general conclusion from this. This speedup has nothing to do with epoll vs io_uring (except maybe that linux file i/o is not really epoll compatible) and everything to do with libuv's block device implementation specifically. It is totally inapplicable to network loads entirely. edit: in the first version I said "disk file" but it is technically block devices, of which disk files are just the most common example, but of course /dev/zero is a block device that is not a disk file.... and worth noting that /dev/zero is never going to actually load off a disk meaning it is "in the cache" all the time.

eklitzke 3y ago
It doesn't really make sense to have an API to tell if reading from a file would block or not because page cache data can be evicted at any time. Even if there was an API that could tell you your next read would be non-blocking, there could be a race where the page cache entry could be evicted before you could do the read.
- Filligree 3y ago
  That sort of race condition is only relevant if it affects correctness. Here it means a read might be unexpectedly slow, but you'll still get the same data. 99.9% of the time the data will not be evicted prior to the read, so it's still going to be a win overall.
  - sgtnoodle 3y ago
    If you're writing a latency sensitive program, 99.9% might as well be 0% though; you're going to end up with an asynchronous anyway.
  - mort96 3y ago
    Remember that a filesystem read might be a disk read in the background. You don't really want to block the main thread for 2 seconds while sshfs does its thing.
    - mort96 3y ago
      Damn, I obviously meant "network read" not a "disk read". Oh well.
- viraptor 3y ago
  Depends how you define that API. It could do something like "check if available, and if so, pin it until the next read (which will happen right now)". Not saying that's a good idea, but if such API was needed, it is possible.
  - 3y ago

tomohawk 3y ago

epoll vs io_uring discussion: https://github.com/axboe/liburing/issues/189 https://github.com/axboe/liburing/issues/189