6 ms·

Python has actually had concurrency since about 2019: https://docs.python.org/3/library/asyncio.html https://docs.python.org/3/library/asyncio.html. Having used

Python has actually had concurrency since about 2019: https://docs.python.org/3/library/asyncio.html https://docs.python.org/3/library/asyncio.html. Having used it a few times, it seems fairly sane, but tbf my experience with concurrency in other languages is fairly limited. edit: ray https://github.com/ray-project/ray https://github.com/ray-project/ray is also pretty easy to use and powerful for actual parallelism

uniqueuid 4y ago
I have used asyncio in anger quite a bit, and have to say that it seems elegant at first and works very well for some use cases. But when you try to do things that aren't a map-reduce or Pool.map() pattern, it suddenly becomes pretty warty. E.g. scheduling work out to a processpool executor is ugly under the hood and IMO ugly syntactically as well.
- tandav 4y ago
  > scheduling work out to a processpool executor is ugly under the hood and IMO ugly syntactically as well. Are you talking about this example? https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor https://docs.python.org/3/library/asyncio-eventloop.html#asy...
deleted 4y ago
[deleted]
BugsJustFindMe 4y ago
I find asyncio to be horrendous, both because of the silliness of its demands on how you build your code and also because of its arbitrarily limited scope. Thread/ProcessPoolExecutor is personally much nicer to use and universally applicable...unless you need to accommodate Ctrl-C and then it's ugly again. But fixing _that_ stupid problem would have been a better expenditure of effort than asyncio.
- coldtea 4y ago
  >I find asyncio to be horrendous, both because of the silliness of its demands on how you build your code and also because of its arbitrarily limited scope. Do you compare it to threads and pools, or judge it on its merits as an async framework (with you having experience of those that you think are done better elsewhere, e.g. in Javascript, C#, etc)? Because both things you mention "demands on how you build your code" and "limited scope" are part of the course with async in most languages that aren't async-first.
  - BugsJustFindMe 4y ago
    > Because both things you mention "demands on how you build your code" and "limited scope" are part of the course with async in most languages I don't see how "asyncio is annoying and can only be used for a fraction of scenarios everywhere else too, not just here" is anything other than reinforcement of what I said. OS threads and processes already exist, can already be applied universally for everything, and the pool executors can work with existing serial code without needing the underlying code to contort itself in very fundamental ways. Python's version of asyncio being no worse than someone else's version of asyncio does not sound like a strong case for using Python's asyncio vs fixing the better-in-basically-every-way concurrent futures interface that already existed.
    - coldtea 4y ago
      >I don't see how "asyncio is annoying and can only be used for a fraction of scenarios everywhere else too, not just here" is anything other than reinforcement of what I said. Well, I didn't try to refute what you wrote (for one, it's clearly a personal, subjective opinion). I asked what I've asked merely to clarify whether your issue is with Python's asyncio (e.g. Python got it wrong) or with the tradeoffs inherent in async io APIs in general (regardless of Python). And it seems that it's the latter. I, for one, am fine with async APIs in JS, which have the same "problems" as the one you've mentioned for Python's, so don't share the sentiment.
      - 4y ago
fastball 4y ago
asyncio is still single-threaded due to the GIL.
- siekmanj 4y ago
  True, but it's not trying to be multi-threaded, just concurrent.
- ikinsey 4y ago
  While not ideal, this can be mitigated with multiprocessing. Python asyncio exposes interfaces for interacting with multiple processes [1]. [1] https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor https://docs.python.org/3/library/asyncio-eventloop.html#asy...
sgtlaggy 4y ago
Concurrency in Python is a weird topic, since multiprocessing is the only "real" concurrency. Threading is "implicit" context switching all in the same process/thread, asyncio is "explicit" context switching. On top of that, you also have the complication of the GIL. If threads don't release the GIL, then you can't effectively switch contexts.
- dragonwriter 4y ago
  > Concurrency in Python is a weird topic, since multiprocessing is the only "real" concurrency. You are confusing concurrency and parallelism. > Threading is "implicit" context switching all in the same process/thread No, threading is separate native threads but with a lock that prevents execution of Python code in separate threads simultaneously (native code in separate threads, with at most on running Python, can still work.)
- hn_saver 4y ago
  Threading IS concurrency. When you say "real" concurrency, you actually mean parallelism.
  - bornfreddy 4y ago
    Not in CPython it isn't. Threading in CPython doesn't allow 2 threads to run concurrently (because of GIL). As GP correctly stated, you need multiprocessing (in CPython) for concurrency.
    - bulatb 4y ago
      They're emphasizing a precise distinction between "concurrent" (the way it's structured) and "parallel" (the way it runs). Concurrent programs have multiple right answers for "Which line of computation can make progress?" Sequential execution picks one step from one of them and runs it, then another, and so on, until everything is done. Whichever step is chosen from whichever computation, it's one step per moment in time; concurrency is only the ability to choose. Parallel execution of concurrent code picks steps from two or more computations and runs them at once. Because of the GIL, Python on CPython has concurrency but limited parallelism.
dekhn 4y ago
Asyncio violates every aspect of compositional orthogonality just like decorators you can't combine it with anything else without completely rewriting your code around its constrictions. It's also caused a huge amount of pip installation problems around the AWS CLI and boto
- Joker_vD 4y ago
  Having both Task and Future was a pretty strange move; and the lack of static typing certainly doesn't help: the moment you get a Task wrapping another Task wrapping the actual result, you really want some static analysis tool to tell you that you forgot one "await".
stepanhruda 4y ago
I’m a fan of asyncio, the parent probably meant to say parallelism though, since that’s what getting rid of GIL unlocks.
ikinsey 4y ago
I love asyncio! It's a very well put together library. It provides great interfaces to manage event loops, io, and some basic networking. It gives you a lot of freedom to design asynchronous systems as you see fit. However, batteries are not included. For example, it provides no HTTP client/server. It doesn't interop with any synchronous IO tools in the standard library either, making asyncio a very insular environment. For the majority of problems, Go or Node.js may be better options. They have much more mature environments for managing asynchrony.
- 1337shadow 4y ago
  It depends how you see it https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-...
  - ikinsey 4y ago
    This is exactly why Go is a better option for async use cases.
    - int_19h 4y ago
      Until you need to do async FFI. Callbacks and the async/await syntactic sugar on top of them compose nicely across language boundaries. But green threads are VM-specific.
  - notpushkin 4y ago
    It does indeed, but personally, I believe with async/await the main pain point of this post (callback hell) is essentially gone.

uniqueuid 4y ago

I have used asyncio in anger quite a bit, and have to say that it seems elegant at first and works very well for some use cases. But when you try to do things that aren't a map-reduce or Pool.map() pattern, it suddenly becomes pretty warty. E.g. scheduling work out to a processpool executor is ugly under the hood and IMO ugly syntactically as well.

tandav 4y ago
> scheduling work out to a processpool executor is ugly under the hood and IMO ugly syntactically as well. Are you talking about this example? https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor https://docs.python.org/3/library/asyncio-eventloop.html#asy...

deleted 4y ago

[deleted]

BugsJustFindMe 4y ago

I find asyncio to be horrendous, both because of the silliness of its demands on how you build your code and also because of its arbitrarily limited scope. Thread/ProcessPoolExecutor is personally much nicer to use and universally applicable...unless you need to accommodate Ctrl-C and then it's ugly again. But fixing _that_ stupid problem would have been a better expenditure of effort than asyncio.

coldtea 4y ago
>I find asyncio to be horrendous, both because of the silliness of its demands on how you build your code and also because of its arbitrarily limited scope. Do you compare it to threads and pools, or judge it on its merits as an async framework (with you having experience of those that you think are done better elsewhere, e.g. in Javascript, C#, etc)? Because both things you mention "demands on how you build your code" and "limited scope" are part of the course with async in most languages that aren't async-first.
- BugsJustFindMe 4y ago
  > Because both things you mention "demands on how you build your code" and "limited scope" are part of the course with async in most languages I don't see how "asyncio is annoying and can only be used for a fraction of scenarios everywhere else too, not just here" is anything other than reinforcement of what I said. OS threads and processes already exist, can already be applied universally for everything, and the pool executors can work with existing serial code without needing the underlying code to contort itself in very fundamental ways. Python's version of asyncio being no worse than someone else's version of asyncio does not sound like a strong case for using Python's asyncio vs fixing the better-in-basically-every-way concurrent futures interface that already existed.
  - coldtea 4y ago
    >I don't see how "asyncio is annoying and can only be used for a fraction of scenarios everywhere else too, not just here" is anything other than reinforcement of what I said. Well, I didn't try to refute what you wrote (for one, it's clearly a personal, subjective opinion). I asked what I've asked merely to clarify whether your issue is with Python's asyncio (e.g. Python got it wrong) or with the tradeoffs inherent in async io APIs in general (regardless of Python). And it seems that it's the latter. I, for one, am fine with async APIs in JS, which have the same "problems" as the one you've mentioned for Python's, so don't share the sentiment.
    - 4y ago

fastball 4y ago

asyncio is still single-threaded due to the GIL.

siekmanj 4y ago
True, but it's not trying to be multi-threaded, just concurrent.
ikinsey 4y ago
While not ideal, this can be mitigated with multiprocessing. Python asyncio exposes interfaces for interacting with multiple processes [1]. [1] https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor https://docs.python.org/3/library/asyncio-eventloop.html#asy...

sgtlaggy 4y ago

Concurrency in Python is a weird topic, since multiprocessing is the only "real" concurrency. Threading is "implicit" context switching all in the same process/thread, asyncio is "explicit" context switching. On top of that, you also have the complication of the GIL. If threads don't release the GIL, then you can't effectively switch contexts.

dragonwriter 4y ago
> Concurrency in Python is a weird topic, since multiprocessing is the only "real" concurrency. You are confusing concurrency and parallelism. > Threading is "implicit" context switching all in the same process/thread No, threading is separate native threads but with a lock that prevents execution of Python code in separate threads simultaneously (native code in separate threads, with at most on running Python, can still work.)
hn_saver 4y ago
Threading IS concurrency. When you say "real" concurrency, you actually mean parallelism.
- bornfreddy 4y ago
  Not in CPython it isn't. Threading in CPython doesn't allow 2 threads to run concurrently (because of GIL). As GP correctly stated, you need multiprocessing (in CPython) for concurrency.
  - bulatb 4y ago
    They're emphasizing a precise distinction between "concurrent" (the way it's structured) and "parallel" (the way it runs). Concurrent programs have multiple right answers for "Which line of computation can make progress?" Sequential execution picks one step from one of them and runs it, then another, and so on, until everything is done. Whichever step is chosen from whichever computation, it's one step per moment in time; concurrency is only the ability to choose. Parallel execution of concurrent code picks steps from two or more computations and runs them at once. Because of the GIL, Python on CPython has concurrency but limited parallelism.

dekhn 4y ago

Asyncio violates every aspect of compositional orthogonality just like decorators you can't combine it with anything else without completely rewriting your code around its constrictions. It's also caused a huge amount of pip installation problems around the AWS CLI and boto

Joker_vD 4y ago
Having both Task and Future was a pretty strange move; and the lack of static typing certainly doesn't help: the moment you get a Task wrapping another Task wrapping the actual result, you really want some static analysis tool to tell you that you forgot one "await".

stepanhruda 4y ago

I’m a fan of asyncio, the parent probably meant to say parallelism though, since that’s what getting rid of GIL unlocks.

ikinsey 4y ago

I love asyncio! It's a very well put together library. It provides great interfaces to manage event loops, io, and some basic networking. It gives you a lot of freedom to design asynchronous systems as you see fit. However, batteries are not included. For example, it provides no HTTP client/server. It doesn't interop with any synchronous IO tools in the standard library either, making asyncio a very insular environment. For the majority of problems, Go or Node.js may be better options. They have much more mature environments for managing asynchrony.

1337shadow 4y ago
It depends how you see it https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-...
- ikinsey 4y ago
  This is exactly why Go is a better option for async use cases.
  - int_19h 4y ago
    Until you need to do async FFI. Callbacks and the async/await syntactic sugar on top of them compose nicely across language boundaries. But green threads are VM-specific.
- notpushkin 4y ago
  It does indeed, but personally, I believe with async/await the main pain point of this post (callback hell) is essentially gone.