5 ms·

I have a lot of experience using concurrency in Go, and for the last couple years have been at the bleeding edge of Python async. The tradeoffs between the two

I have a lot of experience using concurrency in Go, and for the last couple years have been at the bleeding edge of Python async. The tradeoffs between the two approaches are immense. With the virtual thread model you have: * No function coloring problem. This also means existing code is easier to port. * possibility of transparent M:N scheduling. * Impedence mismatch with OS primitives. * Much more sophisticated runtime. * Problematic task cancellation. * Lots of care still needed for non-trivial inter-task synchronization. With the async API model you have: * Viral asyncification (the method color problem). * Simpler runtime. * Obvious and safe task cancellation. * Completely orthogonal to parallelism (actually doing more than one thing simultaneously) for good and for bad. * Inter-task coordination is straightforward and low-overhead even for sophisticated use cases. * Higher initial learning curve. I'm leaning toward liking the async approach more, but that might be just because I'm deep in the middle of it. I think the biggest argument in favor of virtual threads is the automatic parallelism; that's also the biggest argument against: free running threads require more expensive synchronization and introduce nondeterminism.

pron 6y ago
* Java offers both user-mode and kernel threads. You pick at creation, and can even plug your own scheduler. * Loom's virtual threads are completely scheduled in library code, written in Java. * FFI that bypasses the JDK and interacts with native code that does either IO or OS-thread synchronization is extremely rare in Java. * Cancellation is the same for both. Also, IMO, coordination is simpler for threads than for async. Where they differ is in their choice of defaults: thread allow scheduling points anywhere except where explicitly excluded; async/await allows scheduling points nowhere except where explicitly allowed. Putting aside that some languages have both, resulting in few if any guarantees, threads' defaults are better for correct concurrency. The reason is that correctness relies on atomicity, or lack of scheduling points in critical sections. When you explicitly exclude them, none of your callees can break your correctness. When you explicitly allow them, any callee can become async and break its caller's logic. True, the type system will show you where the relevant callsites are, but it will not show you whether there is a reliance on atomicity or not. Async/await does, however, make sense for JavaScript, where all existing code already has an implicit assumption of atomicity, so breaking it would have broken the world. For languages that have both, async/await mostly adds a lot of complexity, although sometimes it is needed for implementation reasons.
- ryanworl 6y ago
  Is it possible to override the scheduler from user code? i.e. if you wanted to control scheduling yourself for some reason.
  - pron 6y ago
    Yes, and even on a per-thread basis.
    - ryanworl 6y ago
      Thanks! Is there an example "Hello World" executor that demonstrates the minimum functionality needed? I would like an executor that runs all tasks on a single thread and schedules them deterministically for testing.
      - pron 6y ago
        var myThreadFactory = Thread.builder().virtual(Executors.newSingleThreadExecutor()).factory(); All threads created by this thread factory will be scheduled to the same kernel thread.
        Cojen 6y ago
        Does this example fully answer the question? Does this allow the tasks to be scheduled deterministically? If I'm very careful and write the tasks such that they make blocking calls at specific locations, then yes. Otherwise, is there any feedback to inform me that tasks are switching context at places I didn't expect? Is it possible to define a custom scheduler that can simply print debug messages at every context switch?
        pron 6y ago
        It answers the question with a "hello, world." To do more sophisticated stuff, like what you want, you'll need to replace or wrap the standard single-thread Executor with your own Executor. There is exactly one method you need to implement. For example: Executor ste = Executors.newSingleThreadExecutor(); Executor myExecutor = task -> { if (task instanceof Thread.VirtualThreadTask vtt) System.out.println("Scheduling " + vtt.thread() + " on " + Thread.currentThread()); ste.execute(task); if (task instanceof Thread.VirtualThreadTask vtt) System.out.println("Descheduled " + vtt.thread() + " from " + Thread.currentThread()); } var myThreadFactory = Thread.builder().virtual(myExecutor).factory();
jerf 6y ago
I don't think "task cancellation" is quite the major difference you think. If you model it as thread A wants to cancel thread B, then while threading means that A runs and cancels B, but B may need some time to catch up, the async world has the problem of thread A running at all to cancel B, if B is having a problem that requires cancellation. It's "obvious" and "safe" until it doesn't happen at all. This is a pervasive problem with the async/await model. As it scales up the probability of something failing to yield when it should and blocking everything else continually goes up as the code size increases, and then the whole model, correctness, practicality, and all, just goes out the window. While it is small for small programs, and it the scaling factor often isn't that large, it is still a thing that happens. Entire OSes used to work that way, with the OS and user processes cooperatively yielding, and what killed that model is this problem. Also, I'm writing a lot of code lately where I can peg multiple cores at a time, with a relatively (if not maximally efficient) language like Go; having to also write it as a whole bunch of OS processes separately running because my runtime can only run on one core at a time is a non-starter, and "async/await" basically turns into a threading system if you try to run it on multiple cores in one system anyhow. These two fatal-for-me flaws mean it's a non-starter for a lot of the work I'm doing anyhow, regardless of any other putative advantages. (As I mentioned, I'm using Go, but if you want to see a runtime that really has the asynchronous exceptions thing figured out, go look at Erlang. Having a thread run off into never-never-land and eating a full CPU isn't fun, but being able to log in to your running system, figure out which it is using a REPL, kill just that thread, and reload its code before restarting it to fix the problem, all without taking down the rest of your system is not an experience most of you have had. But it can be done!)
- jnwatson 6y ago
  Async, and cooperative multitasking in general, requires all members participate in the contract: you shall not block and you shall not go too long until yielding. Once a piece of code violates that, all bets are off. Python has explicit debugging mechanisms to help a developer detect the latter. Cancel safety is less about killing an out-of-control task, and more about making sure the state after cancelling a task is consistent.
  - 6y ago
throwaway894345 6y ago
I've had similar experiences, but I don't much care for async Python. In particular, it's way too easy to block the event loop either by accidentally calling some function that, perhaps transitively, does blocking I/O (this could be remedied if there was no sync IO) or simply by calling a function which is unexpectedly CPU-bound. And when this happens, other requests start failing unrelated to the request that is causing the problem, so you go on this wild goose chase to debug. Sync I/O is also a much nicer, more ergonomic interface than async IMO. And then there are the type error problems--it's way too easy to forget to `await` something. Mypy could help with this, but it's still very, very immature. Lastly, last I checked the debugger couldn't cope with async syntax--this is obviously not criticizing the async approach in general, but I wanted to round out my complaining about async Python. I don't mind working with goroutines personally--I use them sparingly, only when I really need concurrency or parallelism. This takes some discipline (e.g., not to go crazy with goroutines and/or channels) and a bit of experience (in the presence of multiple goroutines, what needs to be locked, when to use channels, etc), so if you're relatively new and very impatient or undisciplined you probably won't have a good time (which isn't to say that if you dislike goroutines you must be a novice or undisciplined!). But for me it's nearly an ideal experience.
svieira 6y ago
A couple of excellent articles on the different ways of thinking about the choice (of when to yield) vs. color (virality of choice) problem: * Choice is bad: https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-... * Choice is good: https://glyph.twistedmatrix.com/2014/02/unyielding.html https://glyph.twistedmatrix.com/2014/02/unyielding.html
breatheoften 6y ago
I'm not sure I really think of function coloring as a "problem" ... facebook is experimenting with auto differentiation for Kotlin and looks like it's adding a new "differentiable" function color -- https://ai.facebook.com/blog/paving-the-way-for-software-20-with-kotlin/ https://ai.facebook.com/blog/paving-the-way-for-software-20-... It looks very easy to reason about and use to me ... and i personally find async a similarly useful marker ... It's about being able to push constraints from caller arbitrarily far down the callee stack -- which is really not something that types support at all but provides for a very high confidence variety of constraint -- and high confidence constraints seem to me like they convey a ton of information. I've been wondering actually whether "function colors" might actually just be a good way to create a whole variety of strong statically enforceable constraints for functions. It seems like they lead to very good and simple programmer mental models ... Are there languages that offer "user definable" function colors? I can think of a lot of application domains that would be much better served by these kinds of constraints than oo or other type-centric approaches ... it would be ridiculously useful to be able to mark a function with the "MyDomainBusinessLogic" color and get assurances that such a method can only call other functions annotated with that color ... would provide an easy way to iterate on app specific abstractions provide compiler assistance for the communication of layering intent -- rather than a bunch of poorly specified words in documents that try to communicate layering intent to other developers -- in language that is either sufficiently precise as to be incomprehensible -- or sufficiently vague as to be subject to (mis)interpretation ...
- jdlshore 6y ago
  There was a series of essays about esoteric language features posted here a few weeks ago, and one of them was exactly what you're talking about. It was a functional language, with the ability to mark functions as involving I/O, non-terminating, or (I think) arbitrary custom "colors." It was an academic language, but very interesting. Sadly, I don't remember the name. Maybe somebody else can post the link.
  - nsm 6y ago
    I believe it is https://news.ycombinator.com/item?id=25178437 https://news.ycombinator.com/item?id=25178437 and the language discussed is Koka https://koka-lang.github.io/koka/doc/manual.html#sec-effect-types https://koka-lang.github.io/koka/doc/manual.html#sec-effect-...

pron 6y ago

* Java offers both user-mode and kernel threads. You pick at creation, and can even plug your own scheduler. * Loom's virtual threads are completely scheduled in library code, written in Java. * FFI that bypasses the JDK and interacts with native code that does either IO or OS-thread synchronization is extremely rare in Java. * Cancellation is the same for both. Also, IMO, coordination is simpler for threads than for async. Where they differ is in their choice of defaults: thread allow scheduling points anywhere except where explicitly excluded; async/await allows scheduling points nowhere except where explicitly allowed. Putting aside that some languages have both, resulting in few if any guarantees, threads' defaults are better for correct concurrency. The reason is that correctness relies on atomicity, or lack of scheduling points in critical sections. When you explicitly exclude them, none of your callees can break your correctness. When you explicitly allow them, any callee can become async and break its caller's logic. True, the type system will show you where the relevant callsites are, but it will not show you whether there is a reliance on atomicity or not. Async/await does, however, make sense for JavaScript, where all existing code already has an implicit assumption of atomicity, so breaking it would have broken the world. For languages that have both, async/await mostly adds a lot of complexity, although sometimes it is needed for implementation reasons.

ryanworl 6y ago
Is it possible to override the scheduler from user code? i.e. if you wanted to control scheduling yourself for some reason.
- pron 6y ago
  Yes, and even on a per-thread basis.
  - ryanworl 6y ago
    Thanks! Is there an example "Hello World" executor that demonstrates the minimum functionality needed? I would like an executor that runs all tasks on a single thread and schedules them deterministically for testing.
    - pron 6y ago
      var myThreadFactory = Thread.builder().virtual(Executors.newSingleThreadExecutor()).factory(); All threads created by this thread factory will be scheduled to the same kernel thread.
      - Cojen 6y ago
        Does this example fully answer the question? Does this allow the tasks to be scheduled deterministically? If I'm very careful and write the tasks such that they make blocking calls at specific locations, then yes. Otherwise, is there any feedback to inform me that tasks are switching context at places I didn't expect? Is it possible to define a custom scheduler that can simply print debug messages at every context switch?
        pron 6y ago
        It answers the question with a "hello, world." To do more sophisticated stuff, like what you want, you'll need to replace or wrap the standard single-thread Executor with your own Executor. There is exactly one method you need to implement. For example: Executor ste = Executors.newSingleThreadExecutor(); Executor myExecutor = task -> { if (task instanceof Thread.VirtualThreadTask vtt) System.out.println("Scheduling " + vtt.thread() + " on " + Thread.currentThread()); ste.execute(task); if (task instanceof Thread.VirtualThreadTask vtt) System.out.println("Descheduled " + vtt.thread() + " from " + Thread.currentThread()); } var myThreadFactory = Thread.builder().virtual(myExecutor).factory();

jerf 6y ago

I don't think "task cancellation" is quite the major difference you think. If you model it as thread A wants to cancel thread B, then while threading means that A runs and cancels B, but B may need some time to catch up, the async world has the problem of thread A running at all to cancel B, if B is having a problem that requires cancellation. It's "obvious" and "safe" until it doesn't happen at all. This is a pervasive problem with the async/await model. As it scales up the probability of something failing to yield when it should and blocking everything else continually goes up as the code size increases, and then the whole model, correctness, practicality, and all, just goes out the window. While it is small for small programs, and it the scaling factor often isn't that large, it is still a thing that happens. Entire OSes used to work that way, with the OS and user processes cooperatively yielding, and what killed that model is this problem. Also, I'm writing a lot of code lately where I can peg multiple cores at a time, with a relatively (if not maximally efficient) language like Go; having to also write it as a whole bunch of OS processes separately running because my runtime can only run on one core at a time is a non-starter, and "async/await" basically turns into a threading system if you try to run it on multiple cores in one system anyhow. These two fatal-for-me flaws mean it's a non-starter for a lot of the work I'm doing anyhow, regardless of any other putative advantages. (As I mentioned, I'm using Go, but if you want to see a runtime that really has the asynchronous exceptions thing figured out, go look at Erlang. Having a thread run off into never-never-land and eating a full CPU isn't fun, but being able to log in to your running system, figure out which it is using a REPL, kill just that thread, and reload its code before restarting it to fix the problem, all without taking down the rest of your system is not an experience most of you have had. But it can be done!)

jnwatson 6y ago
Async, and cooperative multitasking in general, requires all members participate in the contract: you shall not block and you shall not go too long until yielding. Once a piece of code violates that, all bets are off. Python has explicit debugging mechanisms to help a developer detect the latter. Cancel safety is less about killing an out-of-control task, and more about making sure the state after cancelling a task is consistent.
- 6y ago

throwaway894345 6y ago

I've had similar experiences, but I don't much care for async Python. In particular, it's way too easy to block the event loop either by accidentally calling some function that, perhaps transitively, does blocking I/O (this could be remedied if there was no sync IO) or simply by calling a function which is unexpectedly CPU-bound. And when this happens, other requests start failing unrelated to the request that is causing the problem, so you go on this wild goose chase to debug. Sync I/O is also a much nicer, more ergonomic interface than async IMO. And then there are the type error problems--it's way too easy to forget to `await` something. Mypy could help with this, but it's still very, very immature. Lastly, last I checked the debugger couldn't cope with async syntax--this is obviously not criticizing the async approach in general, but I wanted to round out my complaining about async Python. I don't mind working with goroutines personally--I use them sparingly, only when I really need concurrency or parallelism. This takes some discipline (e.g., not to go crazy with goroutines and/or channels) and a bit of experience (in the presence of multiple goroutines, what needs to be locked, when to use channels, etc), so if you're relatively new and very impatient or undisciplined you probably won't have a good time (which isn't to say that if you dislike goroutines you must be a novice or undisciplined!). But for me it's nearly an ideal experience.

svieira 6y ago

A couple of excellent articles on the different ways of thinking about the choice (of when to yield) vs. color (virality of choice) problem: * Choice is bad: https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-... * Choice is good: https://glyph.twistedmatrix.com/2014/02/unyielding.html https://glyph.twistedmatrix.com/2014/02/unyielding.html

breatheoften 6y ago

I'm not sure I really think of function coloring as a "problem" ... facebook is experimenting with auto differentiation for Kotlin and looks like it's adding a new "differentiable" function color -- https://ai.facebook.com/blog/paving-the-way-for-software-20-with-kotlin/ https://ai.facebook.com/blog/paving-the-way-for-software-20-... It looks very easy to reason about and use to me ... and i personally find async a similarly useful marker ... It's about being able to push constraints from caller arbitrarily far down the callee stack -- which is really not something that types support at all but provides for a very high confidence variety of constraint -- and high confidence constraints seem to me like they convey a ton of information. I've been wondering actually whether "function colors" might actually just be a good way to create a whole variety of strong statically enforceable constraints for functions. It seems like they lead to very good and simple programmer mental models ... Are there languages that offer "user definable" function colors? I can think of a lot of application domains that would be much better served by these kinds of constraints than oo or other type-centric approaches ... it would be ridiculously useful to be able to mark a function with the "MyDomainBusinessLogic" color and get assurances that such a method can only call other functions annotated with that color ... would provide an easy way to iterate on app specific abstractions provide compiler assistance for the communication of layering intent -- rather than a bunch of poorly specified words in documents that try to communicate layering intent to other developers -- in language that is either sufficiently precise as to be incomprehensible -- or sufficiently vague as to be subject to (mis)interpretation ...

jdlshore 6y ago
There was a series of essays about esoteric language features posted here a few weeks ago, and one of them was exactly what you're talking about. It was a functional language, with the ability to mark functions as involving I/O, non-terminating, or (I think) arbitrary custom "colors." It was an academic language, but very interesting. Sadly, I don't remember the name. Maybe somebody else can post the link.
- nsm 6y ago
  I believe it is https://news.ycombinator.com/item?id=25178437 https://news.ycombinator.com/item?id=25178437 and the language discussed is Koka https://koka-lang.github.io/koka/doc/manual.html#sec-effect-types https://koka-lang.github.io/koka/doc/manual.html#sec-effect-...