10 ms·

Guide to Concurrency in Python with Asyncio

195 points by LiamPa 6y ago

theelous3 6y ago
Or just use curio or trio and have an infinitely better time with async in python :)
- nurettin 6y ago
  I see no reason to use curio or anything instead of standard asyncio. Works really well for me. Queue tasks, they finish at their leisure, create futures and locks to control flow, all really easy and bugless code. And you get async redis, async http, a lot of asyncio functions for free. So no, I will not change it for some "really well architected library that takes away all your pains" because there aren't any.
  - gjvc 6y ago
    What do you use for HTTP message handling?
    - nurettin 6y ago
      Well, there is no request router in asyncio-http, so I have to use async views in django or if I need something lightweight I go with Quart. But if I was given the proper time, I'd probably invent my own routing on top of asyncio-http using google's re2 to parse the REST requests like I did here some years ago. https://github.com/nurettin/pwned/blob/master/server/server.hpp#L84 https://github.com/nurettin/pwned/blob/master/server/server....
      - takeda 6y ago
        By asyncio http do you mean aiohttp? Besides the standard way, you can also register routes similarly to DJango and Flask: https://docs.aiohttp.org/en/stable/web_quickstart.html#alternative-ways-for-registering-routes https://docs.aiohttp.org/en/stable/web_quickstart.html#alter...
        nurettin 6y ago
        oh I haven't noticed it had a router, thanks
  - downerending 6y ago
    The reason is that curio is well-designed and has a reasonably compact story for how things work. It fits in my head. And the "cool new things I can do with it" to "how hard is it to learn" ratio is pretty high. Asyncio on the other hand, feels like being beaten with a stick. It's so complicated that normal humans are never really going to understand it all. On top of that, it's been morphing all through the 3.0 series, so there isn't even one spec to learn--it's more like a set.
    - nurettin 6y ago
      What do you mean compact story? How do humans not understand asyncio? I think I pretty much do, and I use it in production, with several interconnected services. How has curio been morphing? How is morphing actually a good thing? Isn't it bad that an api is unstable? Overall I understood little to nothing from your comment.
      - downerending 6y ago
        The curio module has not been morphing, which is part of what I like about it. The async module has been changing, which like you, I dislike. The "right" way to do asyncio would have been to properly mull the design into its final form and then introduce it into Python3. Instead, it appears to have been dribbled in piecemeal over several releases. Beyond that, it's just plain complex and confusing. I haven't bothered to use it for anything so far. I'm hoping someone will show up with a machete and cut it down into something usable. Right now, it resembles C++.
        nurettin 6y ago
        Yeah I haven't actually used it until 3.6.9 where it turned into its current final form. What is confusing about asyncio and how does curio solve that problem?
  - theelous3 6y ago
    "I can use this wrench to drive a nail, what would I want a hammer for?" is basically how this reads. I'm not saying software can't be written with asyncio, I'm just saying curio and trio are better.
- gjvc 6y ago
  curio is by David Beazley, [1] which should be reason enough to investigate it. [1] once memorably dubbed "the people's champion" in the introduction to one of his PyCon speeches.
parhamn 6y ago
This looks great. I wish python would have made the event loop generally more transparent to the end user. Why didn't they just use a single global loop? Sorta like javascript and golang. It would have been more pythonic too. Anyone doing heavier context switching could've had their own loop management. Making the event loop self-managed added a ton of clunkiness in aio apis (use-your-own-loop, loop lifetime management, etc) and becomes mentally complex for newcomers. Theres also the issue that these huge aio frameworks rewriting the same TCP clients have emerged only differentiated by their loop management patterns.
- bluedays 6y ago
  The more time I spend with Python the more reasons I find to hate it. However, it's still my main language. Everything I end up making just ends up being written in Python because that's the language I know the best. I could learn another language, and I have learned other languages, but Python seems to be the language that mentally clicks with me the most. I think it's because it's the first language I learned. Suffice to say, there are so many things wrong with Python and they're all compromises that were made to appeal to wider audience. Unfortunately, I don't think that the "wider audience" is those who are looking for concurrency.
  - maallooc 6y ago
    Me too. As I know more and more about languages and computers, the more I hate it but I end up doing 85% of my projects because with it I can focus on ideas, not implementation details.
  - parhamn 6y ago
    We agree. I think python is great. I actually asked because python usually makes things dead simple and straightforward. > Unfortunately, I don't think that the "wider audience" is those who are looking for concurrency. I don't buy this. Many network heavy shops use Python so concurrency is relevant. The language is used in web, finance, cloud, data pipelining/etl, and so many more. Dropbox, Robinhood, Spotify, Instagram, Uber, Google, etc, are the "wider audience" and they certainly care about concurrency.
mcdermott 6y ago
Sorry, but after using Golang with its very simple and powerful goroutines and channels approach to concurrency, this seems like a convoluted mess. When I need concurrency, I certainly don't think of Python as the right tool. Python seems to have lost its way after the 2 to 3 shift and is no longer what I'd consider "pythonic".
- chartpath 6y ago
  If you want to integrate with the data science ecosystem, would you use modern Python for APIs as well, or do Go services with minimal Python bits? EDIT: coming from someone who has had great experiences with FastAPI and Uvicorn combined with Dask and Redis.
- wegs 6y ago
  I think programming language design as a having a large random component. New languages pop up every day, and it's really hard to predict which constructs will resonate well with human brains. Guido was competent, but not nearly as brilliant as, for example, Larry, but Python 2 was dramatically better than Perl. I think this is mostly by chance. Hundreds of new languages come out every year, and by whatever fluke, a few of them really work. Python 2 was that. In an ecosystem like that, past performance is no indication of future performance. There are very few language designers who did well more than once (Guy Steele is the only one I can think of who wasn't a one-hit-wonder). So I too felt like Python 3 was more of a step backwards than forwards linguistically. But at this point, it has features I need which Python 2 lacks, so I'm mostly working in Python 3. But it definitely feels (unnecessarily) less clean.
  - metreo 6y ago
    Python's origin story as being a replacement to Perl is so completely out of scope compared to what modern scripting languages are doing it's laughable.
css 6y ago
One thing I haven't seen any blogs write about is that multiprocessing in 3.8.x uses `spawn()` and not `fork()`[0] on MacOS. Granted, most applications are not running on OS X Server, but an update that changes a low level API like that led to some issues where running code locally will fail when running it remotely will work. The following code will run anywhere except on MacOS on 3.8.x, where it crashes with `_pickle.PicklingError` as it tries to resolve the name `func` that is not in the new stack: import multiprocessing some_data = {1: "one", 2: "two"} func = lambda: some_data.get(2) process = multiprocessing.Process(target=func) process.start() [0]: https://bugs.python.org/issue33725 https://bugs.python.org/issue33725
- jamestimmins 6y ago
  I'm in the process of adding 3.8 support to Airflow. This was the first (but not the last) obstacle in doing so.
jonahbenton 6y ago
Forgive me, but this is such a ball of mud. All of the "easy" introductions into these primitives are basically whitepaper length, with long digressions into the high-level vs the low-level and historical vs modern patterns. And nothing gets into error cases or problems with cooperative scheduling or debugging/troubleshooting or where the GIL applies. In 5 years debugging the rats nest of concurrent python code that people are writing right now will make clear that go really got this right. So sad that python did not. Edit: elsewhere on HN right now: https://nullprogram.com/blog/2020/05/24/ https://nullprogram.com/blog/2020/05/24/
- devxpy 6y ago
  Why would the Gil apply to asyncio, which is strictly single threaded?
  - zomglings 6y ago
    The post also discusses concurrent.futures.
    - joshuamorton 6y ago
      Which is also single (OS) threaded.
      - devxpy 6y ago
        Why do you say that? I believe concurrent.futures is meant as a way for asyncio to spawn threads or processes -- which lets you essentially call sync code from async and not have it block the event loop. From the docs[1] - The concurrent.futures module provides a high-level interface for asynchronously executing callables. The asynchronous execution can be performed with threads, using ThreadPoolExecutor, or separate processes, using ProcessPoolExecutor. Both implement the same interface, which is defined by the abstract Executor class. [1] https://docs.python.org/3.8/library/concurrent.futures.html https://docs.python.org/3.8/library/concurrent.futures.html
mkchoi212 6y ago
Every time I see concurrency and python together, I’m immediately turned off by it. First of all, it probably won’t be a true “concurrency” due to GIL and by the point you are looking for “advanced” libraries that allow you to do whatever you are trying to do, maybe it’s time to switch languages.
- nurettin 6y ago
  Did you mean: not true parallelism?
michalc 6y ago
Shameless plug for what is essentially my own much shorter intro to asyncio: https://charemza.name/blog/posts/python/asyncio/I-like-python-asyncio/ https://charemza.name/blog/posts/python/asyncio/I-like-pytho...
adonese 6y ago
There's this really interesting blog post Im not remembering about nodejs and async concurrency style as compared to golangs one. I cannot remember the author's name, but it goes into length as of comparing async functions and the differences between normal regular func and the async ones, and calling async func only on async scope and so on. Only I remembered I found it retweeted by Brad Fitz. I though maybe someone could guide me to it here
- bobbiechen 6y ago
  Possibly "What Color is Your Function"? One of my favorites on concurrency approaches. https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-...
deleted 6y ago
[deleted]
dilandau 6y ago
Asyncio detractors have been drowned in a stream of assurances that, really, it's not that bad. Obviously it is, or this wouldn't be post #800 that promises to finally make asyncio clear to newcomers. Twisted sucks. It was a joke. Now it's been tacitly blessed to the point you can sprinkle the `async` keyword all over the place. Good luck. I hope you don't forget about any spots. I truly think part of the reason Guido left is because he couldn't defend this shit.
- nessunodoro 6y ago
  Not to stoke the flame, but I never used Twisted, where did it fail?
  - carapace 6y ago
    Bad docs. Everything else about it is solid as hell. Pathlib comes from Twisted IIRC.
- birdyrooster 6y ago
  Yes Twisted had its draw backs but that was well over 15 years ago. Async io is literally like a couple keywords and a library with a few classes you will use for everything. No more callback/errback chaining or function decorators every single method for inline callbacks. No one in these comments are offering any palatable alternatives. A compiled language with no OO is not a replacement for Python's Asyncio.
birdyrooster 6y ago
To those trashing Python in favor of Golang: A compiled language with no OO is not a replacement for Python. Let's talk about the complexity of putting code generators in all of your projects. I've seen real golang projects and the complexity gets moved into your repo.
- uryga 6y ago
  though i'd say that with Go's approach to generics (wrapping everything in `interface{}`), people used to dynamically-typed languages should feel right at home ;)
- rumanator 6y ago
  > To those trashing Python in favor of Golang: A compiled language with no OO is not a replacement for Python. Why? I mean, come on. Who in their right mind would ever claim that Go is not a good choice to develop web services? And are we supposed to turn a blind eye to Python's problems such as sub-par performance and the GIL? I understand how Python fanboys will pick Python over alternatives whether that makes technical sense or not, but asserting that any compiled language cannot be used to deliver software that has been implemented with Python, and that explicit support for OO is a relevant requirement, is something that flies in the face of reason.
  - birdyrooster 6y ago
    With some very simple sharding (we are talking splitting lists and dictionaries) you can bypass the GIL and not balloon your memory usage. To do so you would use aiomultiprocess which inherits from asyncio classes (and therefore interfaces) but lets you await on methods running in their own processes. https://github.com/omnilib/aiomultiprocess https://github.com/omnilib/aiomultiprocess
migueloller 6y ago
I've found this [1] screencast on the low-level principles of async in Python an amazing resource to understand async at a deep level. [1] https://www.youtube.com/watch?v=Y4Gt3Xjd7G8&list=PLKLmIMXskJQ9WpioXF4LPJJ6Tp9hbDXg-&index=2&t=0s https://www.youtube.com/watch?v=Y4Gt3Xjd7G8&list=PLKLmIMXskJ...
citrin_ru 6y ago
I don't see anything about I/O in this guide. Can anyone recommend a guide which explains how to actually do I/O using asyncio: how to accept connections, how to write to and read from a socket?

195 points by LiamPa 6y ago

theelous3 6y ago

Or just use curio or trio and have an infinitely better time with async in python :)

nurettin 6y ago
I see no reason to use curio or anything instead of standard asyncio. Works really well for me. Queue tasks, they finish at their leisure, create futures and locks to control flow, all really easy and bugless code. And you get async redis, async http, a lot of asyncio functions for free. So no, I will not change it for some "really well architected library that takes away all your pains" because there aren't any.
- gjvc 6y ago
  What do you use for HTTP message handling?
  - nurettin 6y ago
    Well, there is no request router in asyncio-http, so I have to use async views in django or if I need something lightweight I go with Quart. But if I was given the proper time, I'd probably invent my own routing on top of asyncio-http using google's re2 to parse the REST requests like I did here some years ago. https://github.com/nurettin/pwned/blob/master/server/server.hpp#L84 https://github.com/nurettin/pwned/blob/master/server/server....
    - takeda 6y ago
      By asyncio http do you mean aiohttp? Besides the standard way, you can also register routes similarly to DJango and Flask: https://docs.aiohttp.org/en/stable/web_quickstart.html#alternative-ways-for-registering-routes https://docs.aiohttp.org/en/stable/web_quickstart.html#alter...
      - nurettin 6y ago
        oh I haven't noticed it had a router, thanks
- downerending 6y ago
  The reason is that curio is well-designed and has a reasonably compact story for how things work. It fits in my head. And the "cool new things I can do with it" to "how hard is it to learn" ratio is pretty high. Asyncio on the other hand, feels like being beaten with a stick. It's so complicated that normal humans are never really going to understand it all. On top of that, it's been morphing all through the 3.0 series, so there isn't even one spec to learn--it's more like a set.
  - nurettin 6y ago
    What do you mean compact story? How do humans not understand asyncio? I think I pretty much do, and I use it in production, with several interconnected services. How has curio been morphing? How is morphing actually a good thing? Isn't it bad that an api is unstable? Overall I understood little to nothing from your comment.
    - downerending 6y ago
      The curio module has not been morphing, which is part of what I like about it. The async module has been changing, which like you, I dislike. The "right" way to do asyncio would have been to properly mull the design into its final form and then introduce it into Python3. Instead, it appears to have been dribbled in piecemeal over several releases. Beyond that, it's just plain complex and confusing. I haven't bothered to use it for anything so far. I'm hoping someone will show up with a machete and cut it down into something usable. Right now, it resembles C++.
      - nurettin 6y ago
        Yeah I haven't actually used it until 3.6.9 where it turned into its current final form. What is confusing about asyncio and how does curio solve that problem?
- theelous3 6y ago
  "I can use this wrench to drive a nail, what would I want a hammer for?" is basically how this reads. I'm not saying software can't be written with asyncio, I'm just saying curio and trio are better.
gjvc 6y ago
curio is by David Beazley, [1] which should be reason enough to investigate it. [1] once memorably dubbed "the people's champion" in the introduction to one of his PyCon speeches.

parhamn 6y ago

This looks great. I wish python would have made the event loop generally more transparent to the end user. Why didn't they just use a single global loop? Sorta like javascript and golang. It would have been more pythonic too. Anyone doing heavier context switching could've had their own loop management. Making the event loop self-managed added a ton of clunkiness in aio apis (use-your-own-loop, loop lifetime management, etc) and becomes mentally complex for newcomers. Theres also the issue that these huge aio frameworks rewriting the same TCP clients have emerged only differentiated by their loop management patterns.

bluedays 6y ago
The more time I spend with Python the more reasons I find to hate it. However, it's still my main language. Everything I end up making just ends up being written in Python because that's the language I know the best. I could learn another language, and I have learned other languages, but Python seems to be the language that mentally clicks with me the most. I think it's because it's the first language I learned. Suffice to say, there are so many things wrong with Python and they're all compromises that were made to appeal to wider audience. Unfortunately, I don't think that the "wider audience" is those who are looking for concurrency.
- maallooc 6y ago
  Me too. As I know more and more about languages and computers, the more I hate it but I end up doing 85% of my projects because with it I can focus on ideas, not implementation details.
- parhamn 6y ago
  We agree. I think python is great. I actually asked because python usually makes things dead simple and straightforward. > Unfortunately, I don't think that the "wider audience" is those who are looking for concurrency. I don't buy this. Many network heavy shops use Python so concurrency is relevant. The language is used in web, finance, cloud, data pipelining/etl, and so many more. Dropbox, Robinhood, Spotify, Instagram, Uber, Google, etc, are the "wider audience" and they certainly care about concurrency.

mcdermott 6y ago

Sorry, but after using Golang with its very simple and powerful goroutines and channels approach to concurrency, this seems like a convoluted mess. When I need concurrency, I certainly don't think of Python as the right tool. Python seems to have lost its way after the 2 to 3 shift and is no longer what I'd consider "pythonic".

chartpath 6y ago
If you want to integrate with the data science ecosystem, would you use modern Python for APIs as well, or do Go services with minimal Python bits? EDIT: coming from someone who has had great experiences with FastAPI and Uvicorn combined with Dask and Redis.
wegs 6y ago
I think programming language design as a having a large random component. New languages pop up every day, and it's really hard to predict which constructs will resonate well with human brains. Guido was competent, but not nearly as brilliant as, for example, Larry, but Python 2 was dramatically better than Perl. I think this is mostly by chance. Hundreds of new languages come out every year, and by whatever fluke, a few of them really work. Python 2 was that. In an ecosystem like that, past performance is no indication of future performance. There are very few language designers who did well more than once (Guy Steele is the only one I can think of who wasn't a one-hit-wonder). So I too felt like Python 3 was more of a step backwards than forwards linguistically. But at this point, it has features I need which Python 2 lacks, so I'm mostly working in Python 3. But it definitely feels (unnecessarily) less clean.
- metreo 6y ago
  Python's origin story as being a replacement to Perl is so completely out of scope compared to what modern scripting languages are doing it's laughable.

css 6y ago

One thing I haven't seen any blogs write about is that multiprocessing in 3.8.x uses `spawn()` and not `fork()`[0] on MacOS. Granted, most applications are not running on OS X Server, but an update that changes a low level API like that led to some issues where running code locally will fail when running it remotely will work. The following code will run anywhere except on MacOS on 3.8.x, where it crashes with `_pickle.PicklingError` as it tries to resolve the name `func` that is not in the new stack: import multiprocessing some_data = {1: "one", 2: "two"} func = lambda: some_data.get(2) process = multiprocessing.Process(target=func) process.start() [0]: https://bugs.python.org/issue33725 https://bugs.python.org/issue33725

jamestimmins 6y ago
I'm in the process of adding 3.8 support to Airflow. This was the first (but not the last) obstacle in doing so.

jonahbenton 6y ago

Forgive me, but this is such a ball of mud. All of the "easy" introductions into these primitives are basically whitepaper length, with long digressions into the high-level vs the low-level and historical vs modern patterns. And nothing gets into error cases or problems with cooperative scheduling or debugging/troubleshooting or where the GIL applies. In 5 years debugging the rats nest of concurrent python code that people are writing right now will make clear that go really got this right. So sad that python did not. Edit: elsewhere on HN right now: https://nullprogram.com/blog/2020/05/24/ https://nullprogram.com/blog/2020/05/24/

devxpy 6y ago
Why would the Gil apply to asyncio, which is strictly single threaded?
- zomglings 6y ago
  The post also discusses concurrent.futures.
  - joshuamorton 6y ago
    Which is also single (OS) threaded.
    - devxpy 6y ago
      Why do you say that? I believe concurrent.futures is meant as a way for asyncio to spawn threads or processes -- which lets you essentially call sync code from async and not have it block the event loop. From the docs[1] - The concurrent.futures module provides a high-level interface for asynchronously executing callables. The asynchronous execution can be performed with threads, using ThreadPoolExecutor, or separate processes, using ProcessPoolExecutor. Both implement the same interface, which is defined by the abstract Executor class. [1] https://docs.python.org/3.8/library/concurrent.futures.html https://docs.python.org/3.8/library/concurrent.futures.html

mkchoi212 6y ago

Every time I see concurrency and python together, I’m immediately turned off by it. First of all, it probably won’t be a true “concurrency” due to GIL and by the point you are looking for “advanced” libraries that allow you to do whatever you are trying to do, maybe it’s time to switch languages.

nurettin 6y ago
Did you mean: not true parallelism?

michalc 6y ago

Shameless plug for what is essentially my own much shorter intro to asyncio: https://charemza.name/blog/posts/python/asyncio/I-like-python-asyncio/ https://charemza.name/blog/posts/python/asyncio/I-like-pytho...

adonese 6y ago

There's this really interesting blog post Im not remembering about nodejs and async concurrency style as compared to golangs one. I cannot remember the author's name, but it goes into length as of comparing async functions and the differences between normal regular func and the async ones, and calling async func only on async scope and so on. Only I remembered I found it retweeted by Brad Fitz. I though maybe someone could guide me to it here

bobbiechen 6y ago
Possibly "What Color is Your Function"? One of my favorites on concurrency approaches. https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-...

deleted 6y ago

[deleted]

dilandau 6y ago

Asyncio detractors have been drowned in a stream of assurances that, really, it's not that bad. Obviously it is, or this wouldn't be post #800 that promises to finally make asyncio clear to newcomers. Twisted sucks. It was a joke. Now it's been tacitly blessed to the point you can sprinkle the `async` keyword all over the place. Good luck. I hope you don't forget about any spots. I truly think part of the reason Guido left is because he couldn't defend this shit.

nessunodoro 6y ago
Not to stoke the flame, but I never used Twisted, where did it fail?
- carapace 6y ago
  Bad docs. Everything else about it is solid as hell. Pathlib comes from Twisted IIRC.
birdyrooster 6y ago
Yes Twisted had its draw backs but that was well over 15 years ago. Async io is literally like a couple keywords and a library with a few classes you will use for everything. No more callback/errback chaining or function decorators every single method for inline callbacks. No one in these comments are offering any palatable alternatives. A compiled language with no OO is not a replacement for Python's Asyncio.

birdyrooster 6y ago

To those trashing Python in favor of Golang: A compiled language with no OO is not a replacement for Python. Let's talk about the complexity of putting code generators in all of your projects. I've seen real golang projects and the complexity gets moved into your repo.

uryga 6y ago
though i'd say that with Go's approach to generics (wrapping everything in `interface{}`), people used to dynamically-typed languages should feel right at home ;)
rumanator 6y ago
> To those trashing Python in favor of Golang: A compiled language with no OO is not a replacement for Python. Why? I mean, come on. Who in their right mind would ever claim that Go is not a good choice to develop web services? And are we supposed to turn a blind eye to Python's problems such as sub-par performance and the GIL? I understand how Python fanboys will pick Python over alternatives whether that makes technical sense or not, but asserting that any compiled language cannot be used to deliver software that has been implemented with Python, and that explicit support for OO is a relevant requirement, is something that flies in the face of reason.
- birdyrooster 6y ago
  With some very simple sharding (we are talking splitting lists and dictionaries) you can bypass the GIL and not balloon your memory usage. To do so you would use aiomultiprocess which inherits from asyncio classes (and therefore interfaces) but lets you await on methods running in their own processes. https://github.com/omnilib/aiomultiprocess https://github.com/omnilib/aiomultiprocess

migueloller 6y ago

I've found this [1] screencast on the low-level principles of async in Python an amazing resource to understand async at a deep level. [1] https://www.youtube.com/watch?v=Y4Gt3Xjd7G8&list=PLKLmIMXskJQ9WpioXF4LPJJ6Tp9hbDXg-&index=2&t=0s https://www.youtube.com/watch?v=Y4Gt3Xjd7G8&list=PLKLmIMXskJ...

citrin_ru 6y ago

I don't see anything about I/O in this guide. Can anyone recommend a guide which explains how to actually do I/O using asyncio: how to accept connections, how to write to and read from a socket?