7 ms·

I'm quire confused about the benefits of async I/O over a blocking thread pool. I can't reconcile the many claims I've read. Everyone does seem to agree that mo

by ai-dev 5y ago

I'm quire confused about the benefits of async I/O over a blocking thread pool. I can't reconcile the many claims I've read. Everyone does seem to agree that most people don't need async, but past that... - The overhead of a thread context switch is expensive due to cache thrashing - If the context switch is due to IO readiness, the overhead is equivalent for both - The main benefit of async is the coding model and the power it provides (cancellation, selection) - Threads won't scale past tens of thousands of connections - Async is about better utilizing memory, not performance - A normal-sized Linux server can handle a million threads without much trouble, async isn't worth it (harder coding model, colored functions) Is the context switch overhead the same for both? If so, why can't threads scale to 100k+ connections?

wtallis 5y ago
Whether a single server can handle a million threads depends on how often those threads need to wake up, and how much work they need to do before going back to sleep. Thread pools don't even come close to working well for storage IO, because dozens or hundreds of threads per core each blocking on just a few kB of IO that the SSDs can handle in milliseconds or less means you spend far too much time on context switches. Network IO is usually lower in aggregate throughput and much higher in latency, so each thread spends relatively more time sleeping and therefore you can fit more of them into a single machine's available CPU time. But a million threads on one box imposes significant constraints on how active those threads can be.
- ai-dev 5y ago
  > But a million threads on one box imposes significant constraints on how active those threads can be. Wouldn't the same constraints be imposed on a million async tasks? What allows async tasks to be more active than threads? Is it the scheduling model, the overhead of context switching, or something else? EDIT: I reread and saw that you mentioned context switching. I guess my question would then be the same as here: https://news.ycombinator.com/item?id=29452759 https://news.ycombinator.com/item?id=29452759. Is the claim that the context switching overhead when due to I/O readiness the same true? I'm mostly thinking about network servers, where I think most? context switches would be due to I/O readiness.
  - xxpor 5y ago
    If nothing else, a green thread context switch is much cheaper than a real context switch because you don't have the security overhead of flushing caches/tlb/etc
vlmutolo 5y ago
I'm only familiar with Rust's async story, though I think the following probably applies to other languages as well. Switching async tasks should have a smaller overhead than switching threads. A context switch involves giving control to the OS and then the OS giving it back at some point. Both involve lots of cache trashing, as you said, and both switches involve bookkeeping that the OS has to do. This probably also involves instruction-cache evictions. Switching async tasks involves loading the new task from memory onto the stack. That's it. The program can immediately start doing useful work again. So, the short answer is that switching threads involves: - more cache evictions - bigger, slower cache evictions - OS-level bookkeeping for threads Also creating threads involves setting up a new, expandable stack in the process's memory space. Creating a new task involves allocating a fixed amount of memory on the heap once.
- ai-dev 5y ago
  > Switching async tasks involves loading the new task from memory onto the stack. That's it. One of the points on the list was that if the context switch is due to I/O readiness, then there's more work for the async task to do [0]: > Think about it this way—if you have a user-space thread which wakes up due to I/O readiness, then this means that the relevant kernel thread woke up from epoll_wait() or something similar. With blocking I/O, you call read(), and the kernel wakes up your thread when the read() completes. With non-blocking I/O, you call read(), get EAGAIN, call epoll_wait(), the kernel wakes up your thread when data is ready, and then you call read() a second time. > In both scenarios, you’re calling a blocking system call and waking up the thread later. 0: https://news.ycombinator.com/item?id=26110699 https://news.ycombinator.com/item?id=26110699
  - roblabla 5y ago
    > this means that the relevant kernel thread woke up from epoll_wait() or something similar. With blocking I/O, you call read(), and the kernel wakes up your thread when the read() completes. With non-blocking I/O, you call read(), get EAGAIN, call epoll_wait(), the kernel wakes up your thread when data is ready, and then you call read() a second time. This is true for readiness-based IO (which, admittedly, most current async IO loops are using), but completion-based IO (such as IOCP or io_uring) don't suffer from this problem: you just add your IO operation to the queue, and do a single syscall that will return once one of the operation in your queue is completed AFAIK.
pdhborges 5y ago
I also have some doubts for some specific scenarios: - Imagine you have a monolith that mostly talks to the database. You have a primary with multiple read replicas. How are you going to take advantage of millions of async tasks if your IO concurrency ends up being caped at a couple of thousand database connections? - The second question is: even if you can have a million requests on flight do you really want to have such a large blast radius on a single server?
- akiselev 5y ago
  > - Imagine you have a monolith that mostly talks to the database. You have a primary with multiple read replicas. How are you going to take advantage of millions of async tasks if your IO concurrency ends up being caped at a couple of thousand database connections? You use a connection pool and multiplex queries over a few persistent connections. This problem exists regardless of whether you use async or thread-per-client. Process per client (i.e. PHP CGI) requires an external connection pool like pgpool. > - The second question is: even if you can have a million requests on flight do you really want to have such a large blast radius on a single server? What's the alternative? When there's a billion connections going through round robin DNS to a thousand load balancers and another hundred million going to backups on failover, everyone's going to have to handle that level of traffic. The scale of the internet is huge and it only just barely works because many services (like DNS, LBs, and databases + their read replicas) can operate at that kind of scale. The people solving these kinds of hard problems have a lot of money that ends up sloshing around via open source github projects, thus influencing the wider engineering community. Many naturally gravitate towards these approaches because they are well tested and usually guaranteed overkill, right in that sweet spot between cover-my-ass and proven-but-shiny. In reality, async is one of those "if you have to ask, you [probably] don't need it" kind of things.
- preseinger 5y ago
  I/O perf is usually a measure of throughput, and while concurrent connections are certainly part of that calculus other factors tend to dominate.
anamax 5y ago
The switch overhead need not be the same. When the OS wakes up a thread and schedules a processor to run it, it has to make sure that said processor has whatever state it needs. If said processor was previously doing something else, the OS has to save the relevant state. Note that the OS doesn't know what's important, so both save and restore are conservative. Yes, switching from one activity to another also has costs, but you only pay what you need to pay and much of the overhead is handled automatically. That said, blocking-for-IO threads do have one huge advantage - the OS knows when to wake them up. With async, something has to look for changes in IO state. Check too often, and you're wasting time. Don't check often enough and you're wasting bandwidth.
- convolvatron 5y ago
  interrupts?
  - duped 5y ago
    Polling driven async is faster than callback based async up and down the stack almost without exception, including interrupts. Interrupts are incredibly slow on modern hardware.
    - anamax 5y ago
      Interrupts vs polling hardware is in the OS. Regardless of how the OS handles IO, user code can either poll the OS or have the OS wakeup a thread when there's a change, such as data for a blocking call.
      - egberts1 5y ago
        As a long-time embedded engineer, I violently recoiled at the over-generalization: “in the OS”. Perhaps “user-land” is that missing aspect. Think how zero-buf embedded network software architect deal with no interrupt and no context switching.
        5y ago

by ai-dev 5y ago

wtallis 5y ago

Whether a single server can handle a million threads depends on how often those threads need to wake up, and how much work they need to do before going back to sleep. Thread pools don't even come close to working well for storage IO, because dozens or hundreds of threads per core each blocking on just a few kB of IO that the SSDs can handle in milliseconds or less means you spend far too much time on context switches. Network IO is usually lower in aggregate throughput and much higher in latency, so each thread spends relatively more time sleeping and therefore you can fit more of them into a single machine's available CPU time. But a million threads on one box imposes significant constraints on how active those threads can be.

ai-dev 5y ago
> But a million threads on one box imposes significant constraints on how active those threads can be. Wouldn't the same constraints be imposed on a million async tasks? What allows async tasks to be more active than threads? Is it the scheduling model, the overhead of context switching, or something else? EDIT: I reread and saw that you mentioned context switching. I guess my question would then be the same as here: https://news.ycombinator.com/item?id=29452759 https://news.ycombinator.com/item?id=29452759. Is the claim that the context switching overhead when due to I/O readiness the same true? I'm mostly thinking about network servers, where I think most? context switches would be due to I/O readiness.
- xxpor 5y ago
  If nothing else, a green thread context switch is much cheaper than a real context switch because you don't have the security overhead of flushing caches/tlb/etc

vlmutolo 5y ago

I'm only familiar with Rust's async story, though I think the following probably applies to other languages as well. Switching async tasks should have a smaller overhead than switching threads. A context switch involves giving control to the OS and then the OS giving it back at some point. Both involve lots of cache trashing, as you said, and both switches involve bookkeeping that the OS has to do. This probably also involves instruction-cache evictions. Switching async tasks involves loading the new task from memory onto the stack. That's it. The program can immediately start doing useful work again. So, the short answer is that switching threads involves: - more cache evictions - bigger, slower cache evictions - OS-level bookkeeping for threads Also creating threads involves setting up a new, expandable stack in the process's memory space. Creating a new task involves allocating a fixed amount of memory on the heap once.

ai-dev 5y ago
> Switching async tasks involves loading the new task from memory onto the stack. That's it. One of the points on the list was that if the context switch is due to I/O readiness, then there's more work for the async task to do [0]: > Think about it this way—if you have a user-space thread which wakes up due to I/O readiness, then this means that the relevant kernel thread woke up from epoll_wait() or something similar. With blocking I/O, you call read(), and the kernel wakes up your thread when the read() completes. With non-blocking I/O, you call read(), get EAGAIN, call epoll_wait(), the kernel wakes up your thread when data is ready, and then you call read() a second time. > In both scenarios, you’re calling a blocking system call and waking up the thread later. 0: https://news.ycombinator.com/item?id=26110699 https://news.ycombinator.com/item?id=26110699
- roblabla 5y ago
  > this means that the relevant kernel thread woke up from epoll_wait() or something similar. With blocking I/O, you call read(), and the kernel wakes up your thread when the read() completes. With non-blocking I/O, you call read(), get EAGAIN, call epoll_wait(), the kernel wakes up your thread when data is ready, and then you call read() a second time. This is true for readiness-based IO (which, admittedly, most current async IO loops are using), but completion-based IO (such as IOCP or io_uring) don't suffer from this problem: you just add your IO operation to the queue, and do a single syscall that will return once one of the operation in your queue is completed AFAIK.

pdhborges 5y ago

I also have some doubts for some specific scenarios: - Imagine you have a monolith that mostly talks to the database. You have a primary with multiple read replicas. How are you going to take advantage of millions of async tasks if your IO concurrency ends up being caped at a couple of thousand database connections? - The second question is: even if you can have a million requests on flight do you really want to have such a large blast radius on a single server?

akiselev 5y ago
> - Imagine you have a monolith that mostly talks to the database. You have a primary with multiple read replicas. How are you going to take advantage of millions of async tasks if your IO concurrency ends up being caped at a couple of thousand database connections? You use a connection pool and multiplex queries over a few persistent connections. This problem exists regardless of whether you use async or thread-per-client. Process per client (i.e. PHP CGI) requires an external connection pool like pgpool. > - The second question is: even if you can have a million requests on flight do you really want to have such a large blast radius on a single server? What's the alternative? When there's a billion connections going through round robin DNS to a thousand load balancers and another hundred million going to backups on failover, everyone's going to have to handle that level of traffic. The scale of the internet is huge and it only just barely works because many services (like DNS, LBs, and databases + their read replicas) can operate at that kind of scale. The people solving these kinds of hard problems have a lot of money that ends up sloshing around via open source github projects, thus influencing the wider engineering community. Many naturally gravitate towards these approaches because they are well tested and usually guaranteed overkill, right in that sweet spot between cover-my-ass and proven-but-shiny. In reality, async is one of those "if you have to ask, you [probably] don't need it" kind of things.
preseinger 5y ago
I/O perf is usually a measure of throughput, and while concurrent connections are certainly part of that calculus other factors tend to dominate.

anamax 5y ago

The switch overhead need not be the same. When the OS wakes up a thread and schedules a processor to run it, it has to make sure that said processor has whatever state it needs. If said processor was previously doing something else, the OS has to save the relevant state. Note that the OS doesn't know what's important, so both save and restore are conservative. Yes, switching from one activity to another also has costs, but you only pay what you need to pay and much of the overhead is handled automatically. That said, blocking-for-IO threads do have one huge advantage - the OS knows when to wake them up. With async, something has to look for changes in IO state. Check too often, and you're wasting time. Don't check often enough and you're wasting bandwidth.

convolvatron 5y ago
interrupts?
- duped 5y ago
  Polling driven async is faster than callback based async up and down the stack almost without exception, including interrupts. Interrupts are incredibly slow on modern hardware.
  - anamax 5y ago
    Interrupts vs polling hardware is in the OS. Regardless of how the OS handles IO, user code can either poll the OS or have the OS wakeup a thread when there's a change, such as data for a blocking call.
    - egberts1 5y ago
      As a long-time embedded engineer, I violently recoiled at the over-generalization: “in the OS”. Perhaps “user-land” is that missing aspect. Think how zero-buf embedded network software architect deal with no interrupt and no context switching.
      - 5y ago