4 ms·

Asynchronous IO isn't about efficiency. The approach the author takes with their language is just threads, but scheduled in userland. This model allows a decou

Asynchronous IO isn't about efficiency. The approach the author takes with their language is just threads, but scheduled in userland. This model allows a decoupling of the performance characteristics of runtime threads from OS threads - which can sometimes be beneficial - but essentially, the programming model is fundamentally still synchronous. Asynchronous programming with async/await is about revealing the time dimension of execution as a first class concept. This allows more opportunities for composition. Take cancellation for example: cancelling tasks under the synchronous programming model requires passing a context object through every part of your code that might call down into an IO operation. This context object is checked for cancellation at each point a task might block, and checked when a blocking operation is interrupted. Timeouts are even trickier to do in this model, especially if your underlying IO only allows you to set per-operation timeouts and you're trying to expose a deadline-style interface instead. Under the asynchronous model, both timeouts and cancellation simply compose. You take a future representing the work you're doing, and spawn a new future that completes after sleeping for some duration, or spawn a new future that waits on a cancel channel. Then you just race these futures. Take whichever completes first and cancel the other. Having done a lot of programming under both paradigms, the synchronous model is so much more clunky and error-prone to work with and involves a lot of tedious manual work, like passing context objects around, that simply disappears under the asynchronous model.

fulafel 2y ago
I'd argue that few usages of async are motivated this way. In Rust land, it's efficiency and in JS land it's the browser scripting language legacy. > cancelling tasks under the synchronous programming model requires passing a context object through every part of your code that might call down into an IO operation. This is true for some but not all implementations. See eg Erlang or Unix processes (and maybe cancellation in pthreads?).
- Yoric 2y ago
  To be more precise, in JS land, we introduced async not directly because of scripting but because of backwards compatibility - prior to async/Promise, the JS + DOM semantics were specified with a single thread of execution in mind, with complex dependencies in both directions (e.g. some DOM operations can cause sync reflow while handling an event, which is... bad) and run-to-completion. Promise made it easier to: - cut monolithic chunks of code-that-needs-to-be-executed-to-completion-before-updating-the-screen into something that didn't cause jank; - introduce background I/O. async/await made Promise more readable. (yes, that's for Promise and async/await on the browser, async callbacks have a different history on Node)
HippoBaro 2y ago
> Under the asynchronous model, both timeouts and cancellation simply compose. You take a future representing the work you're doing, and spawn a new future that completes after sleeping for some duration, or spawn a new future that waits on a cancel channel. Then you just race these futures. Take whichever completes first and cancel the other. That only works when what you're trying to do has no side effect. Consider what happens when you need to cancel a write to a file or a stream. Did you write everything? Something? Nothing? What's the state of the file/stream at this point? Unfortunately, this is intractable: you'll need the underlying system to let you know, which means you will have to wait for it to return. Therefore, if these operations should have a deadline, you'll need to be able to communicate that to the kernel.
dehrmann 2y ago
> Asynchronous programming with async/await is about revealing the time dimension of execution as a first class concept People are more likely to assume their code is fast enough and not worry about the execution time of synchronous data processing, then spend weeks investigating why the p99 latency is 5 seconds with clusters of spikes. Async IO is almost entirely about efficiency. It's telling the OS that you can manage context switches better than it. Usually this means you're making a tradeoff for throughput over latency. That tradeoff is for efficiency is fine, but it needs to be conscious, and most of the time, you actually want lower latency.
- kaba0 2y ago
  Are you sure that the developer is the best at determining these context switches? I mean, for a low-level language like rust, sure. But for higher level programming, e.g. some CRUD backend, should the developer really care about all that added complexity, when the runtime knows just as much, if not more. Like, it’s a DB call? Then just use the async primitive of the OS in the background and schedule another job in its place, until it “returns”. I am not ahead from manually adding points where this could happen. I think the Java virtual thread model is the ideal choice for higher level programming for this reason. Async actually imposes a much stricter order of execution than necessary.
  - thesnide 2y ago
    It is. Until it isn't anymore. Same as we used to do asm, but then the generated code is becoming good enough, or even better than hand written asm. I predict the async trend will fade, as hardware and software will improve. And synchronous programming is higher level than using async. And higer level always prevail given enough time, as management always wants to hire the cheapest devs for the task.
    - aenis 2y ago
      Not sure why the downvotes. Async programming is harder than sync, as one needs not only know one's code, but also all the dependencies. Since the benefits of async are in many scenarios limited[1] I'd expect the simpler abstractions to win. I am a CTO at a large company and I routinely experience tech leads who dont understand what happens under the hood of an async event loop and act surprised when weird and hard to debug p99 issues occur in prod (because some obscure dependency of a dependench does sync file access for something trivial). Abstractions win, people are lazy and most new developers lack understanding of lower level concepts such as interrupts or pooling, nor can predict what code may be cpu bound and unsafe in a async codebase. In a few years, explicit aync will be seen as C is seen today - a low level skill. [1] if your service handles, say, 500qps on average the difference between async and threaded sync might be just 1-2 extra nodes. Does not register on the infra spend.
Veserv 2y ago
One of the other aspects of this is that implementing a synchronous model on top of asynchronous primitives is absolutely trivial. You just wait until the asynchronous operation completes. Any program designed for asynchronous execution can be trivially retrofitted for synchronous execution. In contrast, implementing a asynchronous model on top of synchronous primitives is extremely challenging requiring the current mess of complex implementations such as state machine rewriting and thread pools. Furthermore, retrofitting a program using synchronous execution to use asynchronous execution is a tremendous amount of work as you need to do it bottom up to maintain compatibility during the transition.
- Yoric 2y ago
  > Furthermore, retrofitting a program using synchronous execution to use asynchronous execution is a tremendous amount of work as you need to do it bottom up to maintain compatibility during the transition. Can attest to that. I was part of the team that rewrote Firefox to be fully async. Took us years and we could not maintain compatibility with XUL add-ons (async was not the only reason for this, but that's where the writing on the wall started to become visible).
- tsimionescu 2y ago
  It's also very easy to implement a future/promise style API over a blocking IO primitive, as long as you have cheap threads: you spawn a thread that executes the blocking operation (with cancelation and timeout support as needed) and sets the future's result once the result is done, or some error state. It's really not such a huge problem. I will also note that most async runtimes include much more complex program rewrites and implicit state machines than thread based models. Java style or Go style green threads are much simpler frameworks than C#'s whole async task machinery, or even than Rust's Tokio. And any program written as a series of threads running blocking operations with proper synchronization is also pretty easy to convert to an async model. The difficulty is taking a single-threaded program and making it run in an async model, but that is a completely different discussion. However, I do agree that ultimately you do need the OS to provide async IO primitives to have efficient IO at the application level. Since OS threads can't scale to the required level, even the green threads + blocking IO approach is only realistically implementable with async IO from the OS level. This could change if the OS actually implemented a green threads runtime for blocking operations, but that might still have other inefficiencies related to costs of crossing security boundaries.
kaba0 2y ago
These are just assumptions on your part - the synchronous/threading model doesn’t have to be that primitive, the Thread itself can take on the semantics of cancellation/timeouts just fine. While there are some ergonomic warts in java’s case, it does show that something like interrupts can work reasonably well (with a sufficiently good error handling system, e.g. exceptions). With “structured concurrency”/nurseries it can be much more readable than manually checking interruptions, and you can just do something like fire a bunch of requests with a given timeline, and join them at the end of the “block”.
bjornsing 2y ago
> Take cancellation for example: cancelling tasks under the synchronous programming model requires passing a context object through every part of your code that might call down into an IO operation. Does it? Wouldn’t you just kill the thread in the synchronous model?
- meindnoch 2y ago
  That's a sure way to get leaks and deadlocks.
  - scotty79 2y ago
    Can you have leaks when everything is garbage collected? Do deadlocks occur if killable threads aren't allowed to wait for other threads?
    - pjc50 2y ago
      You can probably make this work if and only if the thread is shared-nothing. If you share any data structure with another thread then you have the possibility of leaving it in an invalid state when killed. This also requires any "channel" primitives you use for inter-thread communication to be tolerant of either thread being killed at any instruction boundary, which is hard to design.
      - meindnoch 2y ago
        Shared-nothing threads? So basically processes.
    - meindnoch 2y ago
      Java threads were killable originally. Then it turned out to be impossible to use safely, so it got deprecated: https://docs.oracle.com/javase/1.5.0/docs/guide/misc/threadPrimitiveDeprecation.html https://docs.oracle.com/javase/1.5.0/docs/guide/misc/threadP...
- pjc50 2y ago
tsimionescu 2y ago
With both synchronous and asynchronous flows, if you want to support cancelation from a high level (e.g. the user can click Cancel in the UI), you need to pass some kind of context from there down to each and every operation that needs to be cancellable. Whether that is done by passing around context objects from the UI down to IO operations, or by ensuring all functions called from the UI down return Task<T> objects, the problem is the same. The context object approach even has the advantage that it also allows you to pass other application-specific things, such as passing progress information up from the bottom of the stack to the UI, or logging ids etc, that the generic Task object won't have. Also, deadline-style contexts aren't as hard as you make them out to be: you keep track of the remaining time, and pass that as a timeout to every blocking operation, then subtract the actual time taken and pass the remaining time to the next blocking task etc. Or, you can do the exact same thing as the async case: you spawn two threads, one handling the blocking operations, the other waiting for a timeout, and both sharing a cancelation context. Whichever finishes first cancels the other. The difficulty of doing cancellations for most real operations is anyway going to be much much higher than these small differences. The real difficulty of cancellations lies in undoing already finished parts of atomic operations that you completed. The effort to do that is going to dominate the effort to get pass down the context object.
anonymoushn 2y ago
Cancellation isn't possible in general. For example, if you've kicked off expensive work on another thread or passed a pointer to your future to the kernel via io_uring, you must add a layer of indirection that's quite similar to a cancelation context. You can't just accept that the expensive work you don't care about will keep happening or resume a future that has been dropped when the cqe entry bearing a pointer to it arrives. The cancellation facility provided by io_uring that guarantees that you won't receive such a thing does so by blocking your thread for a while, which is undesirable. As implemented, async/await typically greatly harms composability. For example see here: https://nullderef.com/blog/rust-async-sync/ https://nullderef.com/blog/rust-async-sync/ In the specific case of Rust, we might begin to be able to write composable libraries a couple decades from now: https://github.com/rust-lang/keyword-generics-initiative https://github.com/rust-lang/keyword-generics-initiative

fulafel 2y ago

I'd argue that few usages of async are motivated this way. In Rust land, it's efficiency and in JS land it's the browser scripting language legacy. > cancelling tasks under the synchronous programming model requires passing a context object through every part of your code that might call down into an IO operation. This is true for some but not all implementations. See eg Erlang or Unix processes (and maybe cancellation in pthreads?).

Yoric 2y ago
To be more precise, in JS land, we introduced async not directly because of scripting but because of backwards compatibility - prior to async/Promise, the JS + DOM semantics were specified with a single thread of execution in mind, with complex dependencies in both directions (e.g. some DOM operations can cause sync reflow while handling an event, which is... bad) and run-to-completion. Promise made it easier to: - cut monolithic chunks of code-that-needs-to-be-executed-to-completion-before-updating-the-screen into something that didn't cause jank; - introduce background I/O. async/await made Promise more readable. (yes, that's for Promise and async/await on the browser, async callbacks have a different history on Node)

HippoBaro 2y ago

> Under the asynchronous model, both timeouts and cancellation simply compose. You take a future representing the work you're doing, and spawn a new future that completes after sleeping for some duration, or spawn a new future that waits on a cancel channel. Then you just race these futures. Take whichever completes first and cancel the other. That only works when what you're trying to do has no side effect. Consider what happens when you need to cancel a write to a file or a stream. Did you write everything? Something? Nothing? What's the state of the file/stream at this point? Unfortunately, this is intractable: you'll need the underlying system to let you know, which means you will have to wait for it to return. Therefore, if these operations should have a deadline, you'll need to be able to communicate that to the kernel.

dehrmann 2y ago

> Asynchronous programming with async/await is about revealing the time dimension of execution as a first class concept People are more likely to assume their code is fast enough and not worry about the execution time of synchronous data processing, then spend weeks investigating why the p99 latency is 5 seconds with clusters of spikes. Async IO is almost entirely about efficiency. It's telling the OS that you can manage context switches better than it. Usually this means you're making a tradeoff for throughput over latency. That tradeoff is for efficiency is fine, but it needs to be conscious, and most of the time, you actually want lower latency.

kaba0 2y ago
Are you sure that the developer is the best at determining these context switches? I mean, for a low-level language like rust, sure. But for higher level programming, e.g. some CRUD backend, should the developer really care about all that added complexity, when the runtime knows just as much, if not more. Like, it’s a DB call? Then just use the async primitive of the OS in the background and schedule another job in its place, until it “returns”. I am not ahead from manually adding points where this could happen. I think the Java virtual thread model is the ideal choice for higher level programming for this reason. Async actually imposes a much stricter order of execution than necessary.
- thesnide 2y ago
  It is. Until it isn't anymore. Same as we used to do asm, but then the generated code is becoming good enough, or even better than hand written asm. I predict the async trend will fade, as hardware and software will improve. And synchronous programming is higher level than using async. And higer level always prevail given enough time, as management always wants to hire the cheapest devs for the task.
  - aenis 2y ago
    Not sure why the downvotes. Async programming is harder than sync, as one needs not only know one's code, but also all the dependencies. Since the benefits of async are in many scenarios limited[1] I'd expect the simpler abstractions to win. I am a CTO at a large company and I routinely experience tech leads who dont understand what happens under the hood of an async event loop and act surprised when weird and hard to debug p99 issues occur in prod (because some obscure dependency of a dependench does sync file access for something trivial). Abstractions win, people are lazy and most new developers lack understanding of lower level concepts such as interrupts or pooling, nor can predict what code may be cpu bound and unsafe in a async codebase. In a few years, explicit aync will be seen as C is seen today - a low level skill. [1] if your service handles, say, 500qps on average the difference between async and threaded sync might be just 1-2 extra nodes. Does not register on the infra spend.

Veserv 2y ago

One of the other aspects of this is that implementing a synchronous model on top of asynchronous primitives is absolutely trivial. You just wait until the asynchronous operation completes. Any program designed for asynchronous execution can be trivially retrofitted for synchronous execution. In contrast, implementing a asynchronous model on top of synchronous primitives is extremely challenging requiring the current mess of complex implementations such as state machine rewriting and thread pools. Furthermore, retrofitting a program using synchronous execution to use asynchronous execution is a tremendous amount of work as you need to do it bottom up to maintain compatibility during the transition.

Yoric 2y ago
> Furthermore, retrofitting a program using synchronous execution to use asynchronous execution is a tremendous amount of work as you need to do it bottom up to maintain compatibility during the transition. Can attest to that. I was part of the team that rewrote Firefox to be fully async. Took us years and we could not maintain compatibility with XUL add-ons (async was not the only reason for this, but that's where the writing on the wall started to become visible).
tsimionescu 2y ago
It's also very easy to implement a future/promise style API over a blocking IO primitive, as long as you have cheap threads: you spawn a thread that executes the blocking operation (with cancelation and timeout support as needed) and sets the future's result once the result is done, or some error state. It's really not such a huge problem. I will also note that most async runtimes include much more complex program rewrites and implicit state machines than thread based models. Java style or Go style green threads are much simpler frameworks than C#'s whole async task machinery, or even than Rust's Tokio. And any program written as a series of threads running blocking operations with proper synchronization is also pretty easy to convert to an async model. The difficulty is taking a single-threaded program and making it run in an async model, but that is a completely different discussion. However, I do agree that ultimately you do need the OS to provide async IO primitives to have efficient IO at the application level. Since OS threads can't scale to the required level, even the green threads + blocking IO approach is only realistically implementable with async IO from the OS level. This could change if the OS actually implemented a green threads runtime for blocking operations, but that might still have other inefficiencies related to costs of crossing security boundaries.

kaba0 2y ago

These are just assumptions on your part - the synchronous/threading model doesn’t have to be that primitive, the Thread itself can take on the semantics of cancellation/timeouts just fine. While there are some ergonomic warts in java’s case, it does show that something like interrupts can work reasonably well (with a sufficiently good error handling system, e.g. exceptions). With “structured concurrency”/nurseries it can be much more readable than manually checking interruptions, and you can just do something like fire a bunch of requests with a given timeline, and join them at the end of the “block”.

bjornsing 2y ago

> Take cancellation for example: cancelling tasks under the synchronous programming model requires passing a context object through every part of your code that might call down into an IO operation. Does it? Wouldn’t you just kill the thread in the synchronous model?

meindnoch 2y ago
That's a sure way to get leaks and deadlocks.
- scotty79 2y ago
  Can you have leaks when everything is garbage collected? Do deadlocks occur if killable threads aren't allowed to wait for other threads?
  - pjc50 2y ago
    You can probably make this work if and only if the thread is shared-nothing. If you share any data structure with another thread then you have the possibility of leaving it in an invalid state when killed. This also requires any "channel" primitives you use for inter-thread communication to be tolerant of either thread being killed at any instruction boundary, which is hard to design.
    - meindnoch 2y ago
      Shared-nothing threads? So basically processes.
  - meindnoch 2y ago
    Java threads were killable originally. Then it turned out to be impossible to use safely, so it got deprecated: https://docs.oracle.com/javase/1.5.0/docs/guide/misc/threadPrimitiveDeprecation.html https://docs.oracle.com/javase/1.5.0/docs/guide/misc/threadP...
pjc50 2y ago

tsimionescu 2y ago

With both synchronous and asynchronous flows, if you want to support cancelation from a high level (e.g. the user can click Cancel in the UI), you need to pass some kind of context from there down to each and every operation that needs to be cancellable. Whether that is done by passing around context objects from the UI down to IO operations, or by ensuring all functions called from the UI down return Task<T> objects, the problem is the same. The context object approach even has the advantage that it also allows you to pass other application-specific things, such as passing progress information up from the bottom of the stack to the UI, or logging ids etc, that the generic Task object won't have. Also, deadline-style contexts aren't as hard as you make them out to be: you keep track of the remaining time, and pass that as a timeout to every blocking operation, then subtract the actual time taken and pass the remaining time to the next blocking task etc. Or, you can do the exact same thing as the async case: you spawn two threads, one handling the blocking operations, the other waiting for a timeout, and both sharing a cancelation context. Whichever finishes first cancels the other. The difficulty of doing cancellations for most real operations is anyway going to be much much higher than these small differences. The real difficulty of cancellations lies in undoing already finished parts of atomic operations that you completed. The effort to do that is going to dominate the effort to get pass down the context object.

anonymoushn 2y ago

Cancellation isn't possible in general. For example, if you've kicked off expensive work on another thread or passed a pointer to your future to the kernel via io_uring, you must add a layer of indirection that's quite similar to a cancelation context. You can't just accept that the expensive work you don't care about will keep happening or resume a future that has been dropped when the cqe entry bearing a pointer to it arrives. The cancellation facility provided by io_uring that guarantees that you won't receive such a thing does so by blocking your thread for a while, which is undesirable. As implemented, async/await typically greatly harms composability. For example see here: https://nullderef.com/blog/rust-async-sync/ https://nullderef.com/blog/rust-async-sync/ In the specific case of Rust, we might begin to be able to write composable libraries a couple decades from now: https://github.com/rust-lang/keyword-generics-initiative https://github.com/rust-lang/keyword-generics-initiative