4 ms·

I am not surprised that the cost of context switching due to I/O readiness can often be roughly equal between async tasks and kernel threads. Normal blocking I/

I am not surprised that the cost of context switching due to I/O readiness can often be roughly equal between async tasks and kernel threads. Normal blocking I/O can be surprisingly efficient because of various factors, such as a reduced need for system calls. Think about it this way—if you have a user-space thread which wakes up due to I/O readiness, then this means that the relevant kernel thread woke up from epoll_wait() or something similar. With blocking I/O, you call read(), and the kernel wakes up your thread when the read() completes. With non-blocking I/O, you call read(), get EAGAIN, call epoll_wait(), the kernel wakes up your thread when data is ready, and then you call read() a second time. In both scenarios, you’re calling a blocking system call and waking up the thread later. Of course, there are scenarios when epoll_wait() returns multiple events, which reduces the number of context switches. But the general result is that it’s not always easy to beat blocking I/O and kernel threads.

khuey 6y ago
io_uring will help a lot here.
- klodolph 6y ago
  Well, it can, but not always. Remember that if you’re waiting for an event to arrive, that generally involves a syscall, the thread being put to sleep, and then the thread being woken up. Any time you’re doing that, think, “Could I just replace this polling system with a call to read()?” What io_uring does do is provide a way to poll without needing to wait, but if you haven’t received new events when you poll, you’re not on the fast path any more. Whether you are often on the fast path for io_uring will depend on the particulars of your application and its I/O patterns.
  - quotemstr 6y ago
    io_uring is specifically designed so that zero system calls are necessary while the system is busy. Userspace and the kernel both update ring buffers, and ring buffers can be checked and drained without entering the kernel at all.
    - klodolph 6y ago
      Yes, that’s what “if you haven’t received new events when you poll” means.
  - scottlamb 6y ago
    > What io_uring does do is provide a way to poll without needing to wait, but if you haven’t received new events when you poll, you’re not on the fast path any more. Isn't "not on the fast path any more" a bit absolutist? io_uring's "slow" path is roughly one syscall per iteration, right? That's still many fewer syscalls than one syscall per IO operation (or more if any return EAGAIN/EWOULDBLOCK) as you'd be doing without it. I'm not sure I really care about eliminating that last syscall per iteration; it seems minor in comparison.
    - klodolph 6y ago
      io_uring’s slow path is making one blocking syscall every time you would ordinarily make a blocking syscall. I am a bit baffled how this could possibly be considered an “absolutist” viewpoint—I am just saying that there exist scenarios where io_uring is not helpful. This should be uncontroversial.
      - scottlamb 6y ago
        You mean when there's only one thing to do per iteration? I'd describe that as when mostly idle. As the system gets more loaded, the one syscall per iteration matters less and less.
      - jorangreef 6y ago
        > io_uring’s slow path is making one blocking syscall every time you would ordinarily make a blocking syscall. That's not correct, io_uring was "absolutely" designed, at least in the technical sense, for zero syscalls in the slow path (if you want to): IORING_SETUP_SQPOLL When this flag is specified, a kernel thread is created to perform submission queue polling. An io_uring instance configured in this way enables an application to issue I/O without ever context switching into the kernel. By using the submission queue to fill in new submission queue entries and watching for completions on the completion queue, the application can submit and reap I/Os without doing a single system call. From the man page: https://manpages.debian.org/unstable/liburing-dev/io_uring_setup.2.en.html https://manpages.debian.org/unstable/liburing-dev/io_uring_s... This mode required privileges in early kernel versions but that's already changed. Things are moving fast.
- the8472 6y ago
  io_uring doesn't mean async though. You can also use it for blocking batch execution of syscalls. E.g. when you need to stat hundreds of files or wait for several child processes at once. So with some batch-oriented convenience wrappers it can help threaded code too.
dmw_ng 6y ago
Linux is likely many years from having anything approaching a fully asynchronous system call interface, if anyone was willing to work on it (io_uring makes a huge dent but I don't think it's intending to reimplement everything). Even where async kernel interfaces exist, without reworking of the kernel-internal implementation still there is often the need for a thread for the kernel side to execute on. For example IIRC this is true for swathes of the vfs implementation at present. So the whole thing is a bit of a false equivalence. Better interfaces that reduce context switches are desirable, but even where they exist often you are just substituting a user thread for a kernel one, and in the general case, there is likely to always be system interfaces that never make it into the brave new world -- take SysV IPC for example (a 1975 era API), it seems doubtful anyone would put the effort into making it async, but there will probably still be times where you might want to consume those interfaces for compatibility or some other obscure reason. Also consider the case where a user program has a need for some substantial thread pools of its own, it might be the case in some scenarios that reusing resources that must already exist in user space and live in warmed caches makes more sense. Neither async or Linux threads are "better", it will always depend on a particular use case, and even then the right answer might well be some combination of both.
- lathiat 6y ago
  I'm not sure about io_uring not reimplementing everything.. it seems to be gaining more and more scope.
- Tuna-Fish 6y ago
  > io_uring makes a huge dent but I don't think it's intending to reimplement everything At this point it's proponents are being pretty unapologic that it will, in fact, reimplement every part of the syscall interface that is actively used.
- throwaway81523 6y ago
  Another possibility at least in the case of lots of network sockets is DPDK, avoiding almost all the context switches if the user side is async.
continuations 6y ago
Then why is it that IO-heavy benchmarks such as the Techempower web benchmark are dominated by async frameworks? The fastest results there are all from async frameworks [1]. And among Rust frameworks the same pattern holds. The fastest Rust frameworks are async while a synchronous frmework such as Rocket is about 20x slower. [1] https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=fortune https://www.techempower.com/benchmarks/#section=data-r20&hw=... [2] https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=fortune&l=yyku7z-sf https://www.techempower.com/benchmarks/#section=data-r20&hw=...
- jacoblambda 6y ago
  I'd argue that it's because even though blocking IO is cheaper, it's very difficult to maximise performance in a multithreaded/concurrent context. You could make faster code with it but I wouldn't want to maintain it and you'd have to throw an obscene amount of man hours at it to get that performance.
- klodolph 6y ago
  Benchmarks are not everything, and the difference between asynch/synchronous operation is not the only thing the benchmark is testing (each of these different frameworks appear to have their own system for parsing and representing HTTP requests). You should know what usage patterns YOUR application sees, understand the relative cost of engineering time and CPU time for YOUR application, and do tests in YOUR environment.
- lmm 6y ago
  Those benchmarks measure one very specific scenario: serving lots of small requests concurrently. Async handles that well because that's exactly the scenario where a single epoll_wait() call will return lots of events.
  - infogulch 6y ago
    Is there a different benchmark that demonstrates a scenario where synchronous syscalls are better suited?
titzer 6y ago
I mostly agree with you (not the least of which is that blocking I/O is a damn fine API), but the reason that people use async I/O is to have lots of outstanding requests. Typically you would use select (or similar) to service whichever one responds first. That way you can multiplex many I/O streams onto a small number of threads. If threads are memory-intensive, you almost certainly have to do this.
singron 6y ago
Google found that the main cost of context switching isn't really in the syscall boundary but in the task scheduling. That's why their linux fork has optional userspace scheduling of kernel threads with the switchto syscalls[0]. Essentially, if your thread already knows which thread should run next, it can context switch to it without having to schedule in the kernel, which is exactly the situation in these bucket brigade benchmarks. This benchmark as written is probably underestimating kernel task scheduling cost since only 1 task is runnable at any 1 time, while a realistic multi-threaded system will have more runnable threads to juggle. 0: http://pdxplumbers.osuosl.org/2013/ocw//system/presentations/1653/original/LPC%20-%20User%20Threading.pdf http://pdxplumbers.osuosl.org/2013/ocw//system/presentations...
- otabdeveloper4 6y ago
  Yes, the point of "async" isn't to save CPU cycles, it's to customize the scheduler so that you can prioritize resource use properly. (E.g., don't switch to the systemd or sshd thread if a customer's web request is timing out.) That said, doing this right is out of reach of the average programmer, and it's doubtful that the compiler has enough domain-specific knowledge to do this automatically. "Async" of the Python and node.js fame is yet another thing, a hack to get around their interpreters' inability to use kernel multitasking features because of global locks.

khuey 6y ago

io_uring will help a lot here.

klodolph 6y ago
Well, it can, but not always. Remember that if you’re waiting for an event to arrive, that generally involves a syscall, the thread being put to sleep, and then the thread being woken up. Any time you’re doing that, think, “Could I just replace this polling system with a call to read()?” What io_uring does do is provide a way to poll without needing to wait, but if you haven’t received new events when you poll, you’re not on the fast path any more. Whether you are often on the fast path for io_uring will depend on the particulars of your application and its I/O patterns.
- quotemstr 6y ago
  io_uring is specifically designed so that zero system calls are necessary while the system is busy. Userspace and the kernel both update ring buffers, and ring buffers can be checked and drained without entering the kernel at all.
  - klodolph 6y ago
    Yes, that’s what “if you haven’t received new events when you poll” means.
- scottlamb 6y ago
  > What io_uring does do is provide a way to poll without needing to wait, but if you haven’t received new events when you poll, you’re not on the fast path any more. Isn't "not on the fast path any more" a bit absolutist? io_uring's "slow" path is roughly one syscall per iteration, right? That's still many fewer syscalls than one syscall per IO operation (or more if any return EAGAIN/EWOULDBLOCK) as you'd be doing without it. I'm not sure I really care about eliminating that last syscall per iteration; it seems minor in comparison.
  - klodolph 6y ago
    io_uring’s slow path is making one blocking syscall every time you would ordinarily make a blocking syscall. I am a bit baffled how this could possibly be considered an “absolutist” viewpoint—I am just saying that there exist scenarios where io_uring is not helpful. This should be uncontroversial.
    - scottlamb 6y ago
      You mean when there's only one thing to do per iteration? I'd describe that as when mostly idle. As the system gets more loaded, the one syscall per iteration matters less and less.
    - jorangreef 6y ago
      > io_uring’s slow path is making one blocking syscall every time you would ordinarily make a blocking syscall. That's not correct, io_uring was "absolutely" designed, at least in the technical sense, for zero syscalls in the slow path (if you want to): IORING_SETUP_SQPOLL When this flag is specified, a kernel thread is created to perform submission queue polling. An io_uring instance configured in this way enables an application to issue I/O without ever context switching into the kernel. By using the submission queue to fill in new submission queue entries and watching for completions on the completion queue, the application can submit and reap I/Os without doing a single system call. From the man page: https://manpages.debian.org/unstable/liburing-dev/io_uring_setup.2.en.html https://manpages.debian.org/unstable/liburing-dev/io_uring_s... This mode required privileges in early kernel versions but that's already changed. Things are moving fast.
the8472 6y ago
io_uring doesn't mean async though. You can also use it for blocking batch execution of syscalls. E.g. when you need to stat hundreds of files or wait for several child processes at once. So with some batch-oriented convenience wrappers it can help threaded code too.

dmw_ng 6y ago

Linux is likely many years from having anything approaching a fully asynchronous system call interface, if anyone was willing to work on it (io_uring makes a huge dent but I don't think it's intending to reimplement everything). Even where async kernel interfaces exist, without reworking of the kernel-internal implementation still there is often the need for a thread for the kernel side to execute on. For example IIRC this is true for swathes of the vfs implementation at present. So the whole thing is a bit of a false equivalence. Better interfaces that reduce context switches are desirable, but even where they exist often you are just substituting a user thread for a kernel one, and in the general case, there is likely to always be system interfaces that never make it into the brave new world -- take SysV IPC for example (a 1975 era API), it seems doubtful anyone would put the effort into making it async, but there will probably still be times where you might want to consume those interfaces for compatibility or some other obscure reason. Also consider the case where a user program has a need for some substantial thread pools of its own, it might be the case in some scenarios that reusing resources that must already exist in user space and live in warmed caches makes more sense. Neither async or Linux threads are "better", it will always depend on a particular use case, and even then the right answer might well be some combination of both.

lathiat 6y ago
I'm not sure about io_uring not reimplementing everything.. it seems to be gaining more and more scope.
Tuna-Fish 6y ago
> io_uring makes a huge dent but I don't think it's intending to reimplement everything At this point it's proponents are being pretty unapologic that it will, in fact, reimplement every part of the syscall interface that is actively used.
throwaway81523 6y ago
Another possibility at least in the case of lots of network sockets is DPDK, avoiding almost all the context switches if the user side is async.

continuations 6y ago

Then why is it that IO-heavy benchmarks such as the Techempower web benchmark are dominated by async frameworks? The fastest results there are all from async frameworks [1]. And among Rust frameworks the same pattern holds. The fastest Rust frameworks are async while a synchronous frmework such as Rocket is about 20x slower. [1] https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=fortune https://www.techempower.com/benchmarks/#section=data-r20&hw=... [2] https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=fortune&l=yyku7z-sf https://www.techempower.com/benchmarks/#section=data-r20&hw=...

jacoblambda 6y ago
I'd argue that it's because even though blocking IO is cheaper, it's very difficult to maximise performance in a multithreaded/concurrent context. You could make faster code with it but I wouldn't want to maintain it and you'd have to throw an obscene amount of man hours at it to get that performance.
klodolph 6y ago
Benchmarks are not everything, and the difference between asynch/synchronous operation is not the only thing the benchmark is testing (each of these different frameworks appear to have their own system for parsing and representing HTTP requests). You should know what usage patterns YOUR application sees, understand the relative cost of engineering time and CPU time for YOUR application, and do tests in YOUR environment.
lmm 6y ago
Those benchmarks measure one very specific scenario: serving lots of small requests concurrently. Async handles that well because that's exactly the scenario where a single epoll_wait() call will return lots of events.
- infogulch 6y ago
  Is there a different benchmark that demonstrates a scenario where synchronous syscalls are better suited?

titzer 6y ago

I mostly agree with you (not the least of which is that blocking I/O is a damn fine API), but the reason that people use async I/O is to have lots of outstanding requests. Typically you would use select (or similar) to service whichever one responds first. That way you can multiplex many I/O streams onto a small number of threads. If threads are memory-intensive, you almost certainly have to do this.

singron 6y ago

Google found that the main cost of context switching isn't really in the syscall boundary but in the task scheduling. That's why their linux fork has optional userspace scheduling of kernel threads with the switchto syscalls[0]. Essentially, if your thread already knows which thread should run next, it can context switch to it without having to schedule in the kernel, which is exactly the situation in these bucket brigade benchmarks. This benchmark as written is probably underestimating kernel task scheduling cost since only 1 task is runnable at any 1 time, while a realistic multi-threaded system will have more runnable threads to juggle. 0: http://pdxplumbers.osuosl.org/2013/ocw//system/presentations/1653/original/LPC%20-%20User%20Threading.pdf http://pdxplumbers.osuosl.org/2013/ocw//system/presentations...

otabdeveloper4 6y ago
Yes, the point of "async" isn't to save CPU cycles, it's to customize the scheduler so that you can prioritize resource use properly. (E.g., don't switch to the systemd or sshd thread if a customer's web request is timing out.) That said, doing this right is out of reach of the average programmer, and it's doubtful that the compiler has enough domain-specific knowledge to do this automatically. "Async" of the Python and node.js fame is yet another thing, a hack to get around their interpreters' inability to use kernel multitasking features because of global locks.