4 ms·

How many systems are there that can't just spawn a thread for each task they have to work on concurrently? This has to be a system that is A) CPU or memory boun

by mbid 5mo ago

How many systems are there that can't just spawn a thread for each task they have to work on concurrently? This has to be a system that is A) CPU or memory bound (since async doesn't make disk or network IO faster) and B) must work on ~tens of thousands of tasks concurrently, i.e. can't just queue up tasks and work on only a small number concurrently. The only meaningful example I can come up with are load balancers, embedded software and perhaps something like browsers. But e.g. an application server implementing a REST API that needs to talk to a database anyway to answer each request doesn't really qualify, since the database connection and the work the database itself does are likely much more resource intensive than the overhead of a thread.

ozgrakkurt 5mo ago
Async does make nvme io faster because queueing multiple operations on the nvme itself is faster.
- mbid 5mo ago
  This is outside of my expertise, but wouldn't multiple threads each submitting a single operation in parallel have the same effect?
  - ozgrakkurt 5mo ago
    That is still “async” considering what gp wrote. Because they wrote “thread per task” which I assume to mean something like “each os thread handles the work submitted by one user”. This is beside the point but, something like io_uring is still significantly better than doing threadpool nvme io.
anonymars 5mo ago
I'm not sure this is correct mental model of what async solves Async precisely improves disk/network I/O-bound applications because synchronous code has to waste a whole thread sitting around waiting for an I/O response (each with its own stack memory and scheduler overhead), and in something like an application server there will be many incoming requests doing so in parallel. Cancellation is also easier with async CPU-bound code would not benefit because the CPU is already busy, and async adds overhead See e.g. https://learn.microsoft.com/en-us/aspnet/web-forms/overview/performance-and-caching/using-asynchronous-methods-in-aspnet-45#:~:text=On%20the%20web,the%20server%20hardware https://learn.microsoft.com/en-us/aspnet/web-forms/overview/... and https://learn.microsoft.com/en-us/aspnet/web-forms/overview/performance-and-caching/using-asynchronous-methods-in-aspnet-45#:~:text=In%20general%2C%20use%20asynchronous,service%20request%20to%20complete https://learn.microsoft.com/en-us/aspnet/web-forms/overview/...
- mbid 5mo ago
  I read this argument ("async is for I/O-bound applications") often, but it makes no sense to me. If your app is I/O bound, how does reducing the work the (already idling!) CPU has to spend on context switching improve the performance of the system?
  - ndriscoll 5mo ago
    IO bound might mean latency but not throughput, so you can up concurrency and add batching, both of which require more concurrent requests in flight to hit your real limit. IO bound might also really mean contention for latches on the database, and different types of requests might hit different tables. Basically, I see people say they're IO bound long before they're at the limit of a single disk, so obviously they are not IO bound. Modern drives are absurdly fast. If everyone were really IO bound, we'd need 1/1000 the hardware we needed 10-15 years ago.
  - anonymars 5mo ago
    It sounds like you're assuming both pieces are running on the same server, which may not be the case (and if you're bottlenecked on the database it probably shouldn't be, because you'd want to move that work off the struggling database server) Assuming for the sake of argument that they are together, you're still saving stack memory for every thread that isn't created. In fact you could say it allows the CPU to be idle, by spending less time context switching. On top of that, async/await is a perfect fit for OS overlapped I/O mechanisms for similar reasons, namely not requiring a separate blocking thread for every pending I/O (see e.g. https://en.wikipedia.org/wiki/Overlapped_I/O https://en.wikipedia.org/wiki/Overlapped_I/O, https://stackoverflow.com/a/5283082 https://stackoverflow.com/a/5283082)
    - mbid 5mo ago
      Right, I think the argument should be that transitioning from a synchronous to asynchronous programming model can improve the performance of a previously CPU/Memory-bound system so that it saturates the IO interface.
default-kramer 5mo ago
I think it's another case of the whole industry being driven by the needs of the very small number of systems that need to handle >10k concurrent requests.
- cmrdporcupine 5mo ago
  Or biases inherited from deploying on single or dual core 32-bit systems from 20 years ago. Honestly, it's a mostly obsolete approach. OS threads are fast. We have lots of cores. The cost of bouncing around on the same core and losing L1 cache coherence is higher than the cost of firing up a new OS thread that could land on a new core. The kernel scheduler gets tuned. Language specific async runtimes are unlikely to see so many eyeballs.
- rerdavies 5mo ago
  Also very useful for applications that have heavily animated graphical user interfaces, where you don't want I/O to interfere with UI animations on the UI thread. (e.g. ALL android apps. Or any UI framework that has a message pump (all of them?).
YZF 5mo ago
Pretty much anything that needs performance and has a lot of relatively light operations is not a candidate for spawning a thread. Context switching and the cost of threads is going to kill performance. A server spawning a thread per request for relatively lightweight request is going to be extremely slow. But sure, if every REST call results in a 10s database query then that's not your bottleneck. A query to a database can be very fast though (due to caches, indices, etc.) so it's not a given that just because you're talking to a database you can just spin up new threads and it'll be fine. EDIT: Something else to consider is what if your REST calls needs to make 5 queries. Do you serialize them? Now your latency can be worse. Do you launch a thread per query? Now you need to a) synchornize b) take x5 the thread cost. Async patterns or green threads or coroutines enable more efficient overlapping of operations and potentially better concurrency (though a server that handles lots of concurrent requests may already have "enough" concurrency anyways).
- layer8 5mo ago
  Server applications don’t spawn threads per request, they use thread pools. The extra context switching due to threads waiting for I/O is negligible in practice for most applications. Asynchronous I/O becomes important when the number of simultaneous requests approaches the number of threads you can have on your system. Many applications don’t come close to that in practice. There’s a benefit in being able to code the handling of a request in synchronous logic. A case has to be made for the particular application that it would cause performance or resource issues, before opting for asynchronous code that adds more complexity.
  - acdha 5mo ago
    There’s also a really good operational benefit if you have limits like total RAM, database connections, etc. where being able to reason about resource usage is important. I’ve seen multiple async apps struggle with things like that because async makes it harder to reason about when resources are released.
    - tcfhgj 5mo ago
RustyRussell 5mo ago
I agree: fork is fast, cheap and easy. If you're spawning something for significant work it tends to be in the noise. Linux kernel uses 8k stacks (TBH, it's been a while), but there's also some copy-on-write overhead. Still, this is not the C10k problem...

by mbid 5mo ago

ozgrakkurt 5mo ago

Async does make nvme io faster because queueing multiple operations on the nvme itself is faster.

mbid 5mo ago
This is outside of my expertise, but wouldn't multiple threads each submitting a single operation in parallel have the same effect?
- ozgrakkurt 5mo ago
  That is still “async” considering what gp wrote. Because they wrote “thread per task” which I assume to mean something like “each os thread handles the work submitted by one user”. This is beside the point but, something like io_uring is still significantly better than doing threadpool nvme io.

anonymars 5mo ago

I'm not sure this is correct mental model of what async solves Async precisely improves disk/network I/O-bound applications because synchronous code has to waste a whole thread sitting around waiting for an I/O response (each with its own stack memory and scheduler overhead), and in something like an application server there will be many incoming requests doing so in parallel. Cancellation is also easier with async CPU-bound code would not benefit because the CPU is already busy, and async adds overhead See e.g. https://learn.microsoft.com/en-us/aspnet/web-forms/overview/performance-and-caching/using-asynchronous-methods-in-aspnet-45#:~:text=On%20the%20web,the%20server%20hardware https://learn.microsoft.com/en-us/aspnet/web-forms/overview/... and https://learn.microsoft.com/en-us/aspnet/web-forms/overview/performance-and-caching/using-asynchronous-methods-in-aspnet-45#:~:text=In%20general%2C%20use%20asynchronous,service%20request%20to%20complete https://learn.microsoft.com/en-us/aspnet/web-forms/overview/...

mbid 5mo ago
I read this argument ("async is for I/O-bound applications") often, but it makes no sense to me. If your app is I/O bound, how does reducing the work the (already idling!) CPU has to spend on context switching improve the performance of the system?
- ndriscoll 5mo ago
  IO bound might mean latency but not throughput, so you can up concurrency and add batching, both of which require more concurrent requests in flight to hit your real limit. IO bound might also really mean contention for latches on the database, and different types of requests might hit different tables. Basically, I see people say they're IO bound long before they're at the limit of a single disk, so obviously they are not IO bound. Modern drives are absurdly fast. If everyone were really IO bound, we'd need 1/1000 the hardware we needed 10-15 years ago.
- anonymars 5mo ago
  It sounds like you're assuming both pieces are running on the same server, which may not be the case (and if you're bottlenecked on the database it probably shouldn't be, because you'd want to move that work off the struggling database server) Assuming for the sake of argument that they are together, you're still saving stack memory for every thread that isn't created. In fact you could say it allows the CPU to be idle, by spending less time context switching. On top of that, async/await is a perfect fit for OS overlapped I/O mechanisms for similar reasons, namely not requiring a separate blocking thread for every pending I/O (see e.g. https://en.wikipedia.org/wiki/Overlapped_I/O https://en.wikipedia.org/wiki/Overlapped_I/O, https://stackoverflow.com/a/5283082 https://stackoverflow.com/a/5283082)
  - mbid 5mo ago
    Right, I think the argument should be that transitioning from a synchronous to asynchronous programming model can improve the performance of a previously CPU/Memory-bound system so that it saturates the IO interface.

default-kramer 5mo ago

I think it's another case of the whole industry being driven by the needs of the very small number of systems that need to handle >10k concurrent requests.

cmrdporcupine 5mo ago
Or biases inherited from deploying on single or dual core 32-bit systems from 20 years ago. Honestly, it's a mostly obsolete approach. OS threads are fast. We have lots of cores. The cost of bouncing around on the same core and losing L1 cache coherence is higher than the cost of firing up a new OS thread that could land on a new core. The kernel scheduler gets tuned. Language specific async runtimes are unlikely to see so many eyeballs.
rerdavies 5mo ago
Also very useful for applications that have heavily animated graphical user interfaces, where you don't want I/O to interfere with UI animations on the UI thread. (e.g. ALL android apps. Or any UI framework that has a message pump (all of them?).

YZF 5mo ago

Pretty much anything that needs performance and has a lot of relatively light operations is not a candidate for spawning a thread. Context switching and the cost of threads is going to kill performance. A server spawning a thread per request for relatively lightweight request is going to be extremely slow. But sure, if every REST call results in a 10s database query then that's not your bottleneck. A query to a database can be very fast though (due to caches, indices, etc.) so it's not a given that just because you're talking to a database you can just spin up new threads and it'll be fine. EDIT: Something else to consider is what if your REST calls needs to make 5 queries. Do you serialize them? Now your latency can be worse. Do you launch a thread per query? Now you need to a) synchornize b) take x5 the thread cost. Async patterns or green threads or coroutines enable more efficient overlapping of operations and potentially better concurrency (though a server that handles lots of concurrent requests may already have "enough" concurrency anyways).

layer8 5mo ago
Server applications don’t spawn threads per request, they use thread pools. The extra context switching due to threads waiting for I/O is negligible in practice for most applications. Asynchronous I/O becomes important when the number of simultaneous requests approaches the number of threads you can have on your system. Many applications don’t come close to that in practice. There’s a benefit in being able to code the handling of a request in synchronous logic. A case has to be made for the particular application that it would cause performance or resource issues, before opting for asynchronous code that adds more complexity.
- acdha 5mo ago
  There’s also a really good operational benefit if you have limits like total RAM, database connections, etc. where being able to reason about resource usage is important. I’ve seen multiple async apps struggle with things like that because async makes it harder to reason about when resources are released.
  - tcfhgj 5mo ago

RustyRussell 5mo ago

I agree: fork is fast, cheap and easy. If you're spawning something for significant work it tends to be in the noise. Linux kernel uses 8k stacks (TBH, it's been a while), but there's also some copy-on-write overhead. Still, this is not the C10k problem...