5 ms·

I think you are missing a key benefit of async/await. It can be implemented incredibly efficiently. This is because it is "stackless". Or in other words you kno

I think you are missing a key benefit of async/await. It can be implemented incredibly efficiently. This is because it is "stackless". Or in other words you know the exact amount of "stack" space required and you can allocate this exactly instead of allocating a real stack which can be very much larger. For example if I want to implement a "sleep" with async/await I probably only need to store the wake time as state, if I want to make a virtual thread to do the same I likely need to allocate a large stack just in case I use it. Of course this can be mitigated with stack caching, small stacks, segmented stacks or other tricks. But doing this is still more expensive than knowing how much "stack" you need up-front and allocating only that.

pron 5y ago
That's not a benefit of async/await as the same could be done with user-mode threads. In fact, that's what we do with virtual threads. But it might be a benefit of async/await in some particular languages.
- kevincox 5y ago
  Can you explain how this is done? Is the current stack copied onto the heap (to the size it currently is)? How are new frames allocated once a thread is suspended?
  - pron 5y ago
    A portion of the stack is copied to the heap when the virtual thread is suspended, and upon successive yields those "stack chunks" are either reused or new ones allocated and form a linked list. When resuming a virtual thread, however, we don't copy its entire stack back from the heap to the stack, but we do it lazily, by installing a "return barrier" by patching the return address, so as you return from a method, its caller (or several callers) is lazily "thawed" back from the heap. This copying of small chunks of memory into a region that's likely in the cache is very efficient. The entire mechanism is rather efficient because in Java we don't have pointers into the stack, so we don't need to pin anything to a specific address, and stacks can be freely moved around.
    - SemanticStrengh 5y ago
      I wonder the implications and opportunities for https://github.com/microsoft/openjdk-proposals/blob/main/stack_allocation/Stack_Allocation_JEP.md https://github.com/microsoft/openjdk-proposals/blob/main/sta...
      - U1F984 5y ago
        Apparently it's already being taken into consideration: > The optimization should work with Project Loom when it becomes available.
- kibwen 5y ago
  > But it might be a benefit of async/await in some particular languages. Rather than saying it's a benefit for particular languages, I'd say it's a benefit in particular contexts, e.g. in contexts where you don't have a heap. Of course it's true that some (most) languages don't support such contexts at all (for a host of good reasons), but the languages that do are shaped by that decision.
  - pron 5y ago
    The use case of interest here is having many concurrent operations (hundreds of thousands or millions). If you don't have a heap, where do you store the (unbounded number of) async/await frames? There are other use-cases where stackless coroutines are useful without being plentiful — e.g. generators — but that's not the use-case we're targeting here (and is probably a use-case of lower importance in general). Many languages/runtimes want just a single coroutine/continuation construct to cover both concurrency and generators — which is a good idea in principle — but then they, especially low-level languages, optimise for the less useful of the two. I've seen some very cool demos of C++ coroutines that are useful for very narrow domains, and yet they offer a single construct that sacrifices the more common, more useful, usage for the less common one. There was one particular presentation about context-switching coroutines in the shadow of cache misses. It was extremely impressive, yet amounted to little more than a party trick. For one, it was extremely sensitive to precise sizing of the coroutine frames, which goes against the point of having a simple, transparent language construct, and for another, it simplifies small code that has to be very carefully written and optimised to the instruction level even after the simplification.
    - kibwen 5y ago
      Yes, I am (perhaps a bit sloppily) using "particular contexts" to refer to particular use cases. And while your use case is the C5M problem, since we're bringing up other languages (which optimize for different contexts) I think it's worth emphasizing that these features also lend themselves to other use cases. Here's an example of using Rust's async/await on embedded devices, for reasons other than serving millions of concurrent connections: https://ferrous-systems.com/blog/async-on-embedded/ https://ferrous-systems.com/blog/async-on-embedded/ > Many languages/runtimes want just a single coroutine/continuation construct to cover both concurrency and generators — which is a good idea in principle — but then they, especially low-level languages, optimise for the less useful of the two. Notably Rust appears to be the opposite here, as it is first focusing on providing higher-level async/await support rather than providing general coroutine support, but its async/await is implemented atop a coroutine abstraction which it does hope to expose directly someday. I'm sure you don't need to be told most of this, but I bring all this up to help answer the more general question of why not every language builds in a green thread runtime, and why one approach is not necessarily strictly superior to another.
vbezhenar 5y ago
OS virtual memory ensures that overhead will not be that big. OS will allocate memory page by page as software touches corresponding virtual addresses. So thread stack will use only as much memory as its maximum stack usage requires (rounded by page size). Async/await of course is more efficient, but in real world native stacks might be good enough, especially when RAM is not very expensive.
- addaon 5y ago
  "Rounded by page size" is a pretty huge caveat here, though, no? With a 4 kB minimum page size on most platforms, 5M threads is 20 GB of stack virtual mappings, minimum. And cycling through those threads even once will make every page of that 20 GB resident.
  - astrange 5y ago
    macOS/iOS is a popular platform where the page size is 16KB and RAM is moderately-to-very expensive. It might be interesting to try something like Mesh (https://github.com/plasma-umass/Mesh https://github.com/plasma-umass/Mesh) to share pages.
    - toast0 5y ago
      macOS/iOS aren't a realistic server platform for high loads. They don't even have syncookies, so anything TCP is out.
  - lumost 5y ago
    That still means I can effectively use 5 million threads on a small server. Which is effectively 3 orders of magnitude more threads than I can currently run with Java.
  - Matthias247 5y ago
    Maybe. Realistically it won't matter, because any real world server would either need a lot more memory anyway to actually handle application-specific concerns or support a much lower number of clients. Keep in mind that even with a tiny and receive buffer of 16kB plus maybe some TLS state of > 30kB per connection the baseline memory usage of doing anything useful is already much higher than 4kB - unless the only thing you want to do is building a large-scale TCP ping service.
dboreham 5y ago
> It can be implemented incredibly efficiently. At the cost of breaking : conceptual model of concurrency ; debugging ; performance analysis ; tracing ; logging but yeah...great stuff.
Matthias247 5y ago
This is a reason that is brought up once in a while - but even after working 10 years in the domain of high-concurrency services I've never seen compelling data that shows clearly whether stackless coroutines are more efficient than stackful ones. People unfortunately rarely write applications in both approaches to tell. While the "stack is optimally sized" argument exists, it might not always be true: E.g. implementations could require far more memory being required for the "virtual" stack than what is actually required due to implementation challenges. That for example applies in various situations in Rust. Then a more classical stackless implementation which allocates state for each callback on the heap (like if you manually write boost asio code) which have quite some allocation and memcpy churn. And besides that a "virtual stack" might be more fragmented and less cache friendly than a contiguous stack, which also impacts efficiency.

pron 5y ago

That's not a benefit of async/await as the same could be done with user-mode threads. In fact, that's what we do with virtual threads. But it might be a benefit of async/await in some particular languages.

kevincox 5y ago
Can you explain how this is done? Is the current stack copied onto the heap (to the size it currently is)? How are new frames allocated once a thread is suspended?
- pron 5y ago
  A portion of the stack is copied to the heap when the virtual thread is suspended, and upon successive yields those "stack chunks" are either reused or new ones allocated and form a linked list. When resuming a virtual thread, however, we don't copy its entire stack back from the heap to the stack, but we do it lazily, by installing a "return barrier" by patching the return address, so as you return from a method, its caller (or several callers) is lazily "thawed" back from the heap. This copying of small chunks of memory into a region that's likely in the cache is very efficient. The entire mechanism is rather efficient because in Java we don't have pointers into the stack, so we don't need to pin anything to a specific address, and stacks can be freely moved around.
  - SemanticStrengh 5y ago
    I wonder the implications and opportunities for https://github.com/microsoft/openjdk-proposals/blob/main/stack_allocation/Stack_Allocation_JEP.md https://github.com/microsoft/openjdk-proposals/blob/main/sta...
    - U1F984 5y ago
      Apparently it's already being taken into consideration: > The optimization should work with Project Loom when it becomes available.
kibwen 5y ago
> But it might be a benefit of async/await in some particular languages. Rather than saying it's a benefit for particular languages, I'd say it's a benefit in particular contexts, e.g. in contexts where you don't have a heap. Of course it's true that some (most) languages don't support such contexts at all (for a host of good reasons), but the languages that do are shaped by that decision.
- pron 5y ago
  The use case of interest here is having many concurrent operations (hundreds of thousands or millions). If you don't have a heap, where do you store the (unbounded number of) async/await frames? There are other use-cases where stackless coroutines are useful without being plentiful — e.g. generators — but that's not the use-case we're targeting here (and is probably a use-case of lower importance in general). Many languages/runtimes want just a single coroutine/continuation construct to cover both concurrency and generators — which is a good idea in principle — but then they, especially low-level languages, optimise for the less useful of the two. I've seen some very cool demos of C++ coroutines that are useful for very narrow domains, and yet they offer a single construct that sacrifices the more common, more useful, usage for the less common one. There was one particular presentation about context-switching coroutines in the shadow of cache misses. It was extremely impressive, yet amounted to little more than a party trick. For one, it was extremely sensitive to precise sizing of the coroutine frames, which goes against the point of having a simple, transparent language construct, and for another, it simplifies small code that has to be very carefully written and optimised to the instruction level even after the simplification.
  - kibwen 5y ago
    Yes, I am (perhaps a bit sloppily) using "particular contexts" to refer to particular use cases. And while your use case is the C5M problem, since we're bringing up other languages (which optimize for different contexts) I think it's worth emphasizing that these features also lend themselves to other use cases. Here's an example of using Rust's async/await on embedded devices, for reasons other than serving millions of concurrent connections: https://ferrous-systems.com/blog/async-on-embedded/ https://ferrous-systems.com/blog/async-on-embedded/ > Many languages/runtimes want just a single coroutine/continuation construct to cover both concurrency and generators — which is a good idea in principle — but then they, especially low-level languages, optimise for the less useful of the two. Notably Rust appears to be the opposite here, as it is first focusing on providing higher-level async/await support rather than providing general coroutine support, but its async/await is implemented atop a coroutine abstraction which it does hope to expose directly someday. I'm sure you don't need to be told most of this, but I bring all this up to help answer the more general question of why not every language builds in a green thread runtime, and why one approach is not necessarily strictly superior to another.

vbezhenar 5y ago

OS virtual memory ensures that overhead will not be that big. OS will allocate memory page by page as software touches corresponding virtual addresses. So thread stack will use only as much memory as its maximum stack usage requires (rounded by page size). Async/await of course is more efficient, but in real world native stacks might be good enough, especially when RAM is not very expensive.

addaon 5y ago
"Rounded by page size" is a pretty huge caveat here, though, no? With a 4 kB minimum page size on most platforms, 5M threads is 20 GB of stack virtual mappings, minimum. And cycling through those threads even once will make every page of that 20 GB resident.
- astrange 5y ago
  macOS/iOS is a popular platform where the page size is 16KB and RAM is moderately-to-very expensive. It might be interesting to try something like Mesh (https://github.com/plasma-umass/Mesh https://github.com/plasma-umass/Mesh) to share pages.
  - toast0 5y ago
    macOS/iOS aren't a realistic server platform for high loads. They don't even have syncookies, so anything TCP is out.
- lumost 5y ago
  That still means I can effectively use 5 million threads on a small server. Which is effectively 3 orders of magnitude more threads than I can currently run with Java.
- Matthias247 5y ago
  Maybe. Realistically it won't matter, because any real world server would either need a lot more memory anyway to actually handle application-specific concerns or support a much lower number of clients. Keep in mind that even with a tiny and receive buffer of 16kB plus maybe some TLS state of > 30kB per connection the baseline memory usage of doing anything useful is already much higher than 4kB - unless the only thing you want to do is building a large-scale TCP ping service.

dboreham 5y ago

> It can be implemented incredibly efficiently. At the cost of breaking : conceptual model of concurrency ; debugging ; performance analysis ; tracing ; logging but yeah...great stuff.

Matthias247 5y ago

This is a reason that is brought up once in a while - but even after working 10 years in the domain of high-concurrency services I've never seen compelling data that shows clearly whether stackless coroutines are more efficient than stackful ones. People unfortunately rarely write applications in both approaches to tell. While the "stack is optimally sized" argument exists, it might not always be true: E.g. implementations could require far more memory being required for the "virtual" stack than what is actually required due to implementation challenges. That for example applies in various situations in Rust. Then a more classical stackless implementation which allocates state for each callback on the heap (like if you manually write boost asio code) which have quite some allocation and memcpy churn. And besides that a "virtual stack" might be more fragmented and less cache friendly than a contiguous stack, which also impacts efficiency.