4 ms·

Yet another asyncio tutorial that shows you to run a few sleep tasks concurrently. Can we finally get one that shows how to do real stuff such like socket progr

Yet another asyncio tutorial that shows you to run a few sleep tasks concurrently. Can we finally get one that shows how to do real stuff such like socket programming, wrapping non-async-compatible libraries and separating cpu-intensive blocking tasks to awaitable threads?

pmlnr 8y ago
...including error handling in async worker loops, please.
sametmax 8y ago
> such like socket programming That's one of my biggest pet peeves (and if you see my other comments, you'll notice I have quite a few). To do socket programming in asyncio, you can either use: - protocols, with a nice reusable API and an interface that clearly tells you where to do what. But you can't use "await". You are back to creating futures and attaching callback like 10 years ago. - streams, where you can use async / await, you but get to write the entire life cycle by yourself all over again. I get that protocols are faster, and match Twisted model, and I get that streams are pure and functional, but none of this is easy. I use Python to make my life easier. If I wanted extreme perfs I'd use C. If I wanted extreme pureness I'd use Haskell. > wrapping non-async-compatible libraries and separating cpu-intensive blocking tasks to awaitable threads That's the one of the things asyncio did right. Executors are incredibly simple to use, robust and well integrated. Problem is: they are badly documented and the API is awkward. I won't write a tutorial in HN, but as a starting point: You can use: loop = asyncio.get_event_loop() future = loop.run_in_executor(executor, callback, arg1, arg2, arg2...) await future If you pass "None" as an executor, it will get the default one, which will run your callback in a thread pool. Very useful for stuff like database calls. But if you want CPU intensive task, you need to create an instance of ProcessPoolExecutor, and pass it to run_in_executor(). I say it's one of the things asyncio did right because the pools not only distribute automatically the callbacks among the workers of the pool (which you can control the number), but you also get a future back which you can await transparently.
- mtrovo 8y ago
  > Problem is: they are badly documented and the API is awkward. That's my main problem with asyncio right now, bumping into problems and trying to find how to fix them by looking into the documentation is rather difficult. The documentation right now feels more like a documentation for an unsupported old library. Also it's awkward that you cannot resolve kwargs on the functions that you pass to asyncio, like the callback in run_in_executor. You have to wrap the function in a partial resolving all kwargs and then send it to the executor.
- windexh8er 8y ago
  I'm curious of your take on Trio or Curio. Do either address the peeves you've outlined? https://github.com/python-trio/trio https://github.com/python-trio/trio https://github.com/dabeaz/curio https://github.com/dabeaz/curio
  - sametmax 8y ago
    Trio is a better Curio, so you don't really need Curio anymore. It was what started everything and deserves credit for that though. As for Trio, it's what asyncio should have been from the beginning, at least for the high level part (although not for the pet peeves of socket programming: it's too low level for Python IMO) The problem with Trio is that it's incompatible with asyncio (minus some verbose quirky bridges), so you get yet another island, yet another ecosystem. So what, now we get twisted, tornado, gevent, qt, asyncio... and trio ? The madness must stop. And that's why I think there is a better way: creating a higher level API for asyncio, which enforces the best practices and make the common things easy, and the hard things decent. A complete rewrite like Trio would be better (e.g: it famously handles Ctrl + C way better and has a smaller API). But this ship has sailed. We have asyncio now. asyncio is pretty good honestly. But it needs some love. So, considering asyncio is what we have to work with, and by experience, it's quite great if you know what you are doing, I advice people to actually write a wrapper around it. If you don't feel like writing a wrapper, I'll plug in the one I'm working on in case people are curious: https://github.com/Tygs/ayo https://github.com/Tygs/ayo It: - is based on asyncio. Not a new framework. Integrated transparently with normal asyncio. - implement some lessons leaned from trio (e.g: nurseries, cancellation, etc) - expose a sweet API (running blocking code is run.aside(callback, args, *kwargs), and automate stuff it can (@ayo.run_as_main setup the event loop and run the function in one row) - make hard things decent: timeout and concurrency limit are just a param away I does need some serious doc, including a rich tutorial which features a pure asyncio part. Also it needs some mix between streams and protocols. I'm not going to skip that part, I think it's mandatory, but I'll need many months to make the whole thing. Now, I am not Nathaniel or Yury, so my work is not nearly as bullet proof as theirs. I would not advice to install ayo in prod now, but I think it's a great proof of concept of how good asyncio can be. And we most certainly can do even better.
luckycharms810 8y ago
It is not in the style of a blog, but I put this together a few years ago to teach people how async programming works. It includes a handful of examples on generators, Coroutines, how to make use of the socket libs non blocking functionality, and how to tie them together to get the yield syntax folks are used to. It doesn’t use any asyncio parts of python but is just meant to show what’s happening under the hood. https://github.com/ltavag/async_presentation?files=1 https://github.com/ltavag/async_presentation?files=1
Sir_Cmpwn 8y ago
If anyone wants to see some small, practical asyncio code in action, here's a little LMTP daemon I wrote recently: https://git.sr.ht/~sircmpwn/lists.sr.ht/tree/lists-srht-lmtp https://git.sr.ht/~sircmpwn/lists.sr.ht/tree/lists-srht-lmtp via https://git.sr.ht/~sircmpwn/lists.sr.ht https://git.sr.ht/~sircmpwn/lists.sr.ht Or getting deeper, another project which implements Synapse's[0] RPC protocol and encapsulates high-level RPC actions in asyncio sugar: https://git.sr.ht/~sircmpwn/broca/tree/broca/connection.py https://git.sr.ht/~sircmpwn/broca/tree/broca/connection.py Code which uses this code: https://git.sr.ht/~sircmpwn/broca/tree/broca/rpc.py https://git.sr.ht/~sircmpwn/broca/tree/broca/rpc.py [0] https://synapse-bt.org https://synapse-bt.org
cup-of-tea 8y ago
If you don't mind a video format, this talk by Robert Smallshire includes that: https://www.youtube.com/watch?v=M-UcUs7IMIM https://www.youtube.com/watch?v=M-UcUs7IMIM
AboutTheWhisles 8y ago
This architecture is made to match software threads to (logical) hardware threads, then have them loop over data separated into chunks that don't depend on each other. If a function is blocking, needs the CPU and isn't thread safe, it can be wrapped in a message passing node that will get skipped over if a thread is already running it. Every separate chunk of data that it creates will be dealt with concurrently and the high level structure can be put together in a graph that uses openGL. https://github.com/LiveAsynchronousVisualizedArchitecture/lava/blob/master/README.md https://github.com/LiveAsynchronousVisualizedArchitecture/la...

pmlnr 8y ago

...including error handling in async worker loops, please.

sametmax 8y ago

> such like socket programming That's one of my biggest pet peeves (and if you see my other comments, you'll notice I have quite a few). To do socket programming in asyncio, you can either use: - protocols, with a nice reusable API and an interface that clearly tells you where to do what. But you can't use "await". You are back to creating futures and attaching callback like 10 years ago. - streams, where you can use async / await, you but get to write the entire life cycle by yourself all over again. I get that protocols are faster, and match Twisted model, and I get that streams are pure and functional, but none of this is easy. I use Python to make my life easier. If I wanted extreme perfs I'd use C. If I wanted extreme pureness I'd use Haskell. > wrapping non-async-compatible libraries and separating cpu-intensive blocking tasks to awaitable threads That's the one of the things asyncio did right. Executors are incredibly simple to use, robust and well integrated. Problem is: they are badly documented and the API is awkward. I won't write a tutorial in HN, but as a starting point: You can use: loop = asyncio.get_event_loop() future = loop.run_in_executor(executor, callback, arg1, arg2, arg2...) await future If you pass "None" as an executor, it will get the default one, which will run your callback in a thread pool. Very useful for stuff like database calls. But if you want CPU intensive task, you need to create an instance of ProcessPoolExecutor, and pass it to run_in_executor(). I say it's one of the things asyncio did right because the pools not only distribute automatically the callbacks among the workers of the pool (which you can control the number), but you also get a future back which you can await transparently.

mtrovo 8y ago
> Problem is: they are badly documented and the API is awkward. That's my main problem with asyncio right now, bumping into problems and trying to find how to fix them by looking into the documentation is rather difficult. The documentation right now feels more like a documentation for an unsupported old library. Also it's awkward that you cannot resolve kwargs on the functions that you pass to asyncio, like the callback in run_in_executor. You have to wrap the function in a partial resolving all kwargs and then send it to the executor.
windexh8er 8y ago
I'm curious of your take on Trio or Curio. Do either address the peeves you've outlined? https://github.com/python-trio/trio https://github.com/python-trio/trio https://github.com/dabeaz/curio https://github.com/dabeaz/curio
- sametmax 8y ago
  Trio is a better Curio, so you don't really need Curio anymore. It was what started everything and deserves credit for that though. As for Trio, it's what asyncio should have been from the beginning, at least for the high level part (although not for the pet peeves of socket programming: it's too low level for Python IMO) The problem with Trio is that it's incompatible with asyncio (minus some verbose quirky bridges), so you get yet another island, yet another ecosystem. So what, now we get twisted, tornado, gevent, qt, asyncio... and trio ? The madness must stop. And that's why I think there is a better way: creating a higher level API for asyncio, which enforces the best practices and make the common things easy, and the hard things decent. A complete rewrite like Trio would be better (e.g: it famously handles Ctrl + C way better and has a smaller API). But this ship has sailed. We have asyncio now. asyncio is pretty good honestly. But it needs some love. So, considering asyncio is what we have to work with, and by experience, it's quite great if you know what you are doing, I advice people to actually write a wrapper around it. If you don't feel like writing a wrapper, I'll plug in the one I'm working on in case people are curious: https://github.com/Tygs/ayo https://github.com/Tygs/ayo It: - is based on asyncio. Not a new framework. Integrated transparently with normal asyncio. - implement some lessons leaned from trio (e.g: nurseries, cancellation, etc) - expose a sweet API (running blocking code is run.aside(callback, args, *kwargs), and automate stuff it can (@ayo.run_as_main setup the event loop and run the function in one row) - make hard things decent: timeout and concurrency limit are just a param away I does need some serious doc, including a rich tutorial which features a pure asyncio part. Also it needs some mix between streams and protocols. I'm not going to skip that part, I think it's mandatory, but I'll need many months to make the whole thing. Now, I am not Nathaniel or Yury, so my work is not nearly as bullet proof as theirs. I would not advice to install ayo in prod now, but I think it's a great proof of concept of how good asyncio can be. And we most certainly can do even better.

luckycharms810 8y ago

It is not in the style of a blog, but I put this together a few years ago to teach people how async programming works. It includes a handful of examples on generators, Coroutines, how to make use of the socket libs non blocking functionality, and how to tie them together to get the yield syntax folks are used to. It doesn’t use any asyncio parts of python but is just meant to show what’s happening under the hood. https://github.com/ltavag/async_presentation?files=1 https://github.com/ltavag/async_presentation?files=1

Sir_Cmpwn 8y ago

If anyone wants to see some small, practical asyncio code in action, here's a little LMTP daemon I wrote recently: https://git.sr.ht/~sircmpwn/lists.sr.ht/tree/lists-srht-lmtp https://git.sr.ht/~sircmpwn/lists.sr.ht/tree/lists-srht-lmtp via https://git.sr.ht/~sircmpwn/lists.sr.ht https://git.sr.ht/~sircmpwn/lists.sr.ht Or getting deeper, another project which implements Synapse's[0] RPC protocol and encapsulates high-level RPC actions in asyncio sugar: https://git.sr.ht/~sircmpwn/broca/tree/broca/connection.py https://git.sr.ht/~sircmpwn/broca/tree/broca/connection.py Code which uses this code: https://git.sr.ht/~sircmpwn/broca/tree/broca/rpc.py https://git.sr.ht/~sircmpwn/broca/tree/broca/rpc.py [0] https://synapse-bt.org https://synapse-bt.org

cup-of-tea 8y ago

If you don't mind a video format, this talk by Robert Smallshire includes that: https://www.youtube.com/watch?v=M-UcUs7IMIM https://www.youtube.com/watch?v=M-UcUs7IMIM

AboutTheWhisles 8y ago

This architecture is made to match software threads to (logical) hardware threads, then have them loop over data separated into chunks that don't depend on each other. If a function is blocking, needs the CPU and isn't thread safe, it can be wrapped in a message passing node that will get skipped over if a thread is already running it. Every separate chunk of data that it creates will be dealt with concurrently and the high level structure can be put together in a graph that uses openGL. https://github.com/LiveAsynchronousVisualizedArchitecture/lava/blob/master/README.md https://github.com/LiveAsynchronousVisualizedArchitecture/la...