6 ms·

Python: Generators, Coroutines, Native Coroutines and Async/Await

147 points by snw 11y ago

mariocesar 11y ago
If you don't have the latest asyncio version, change `ensure_future` to `async`. > asyncio.async(display_date(1, loop)) See this https://github.com/python/asyncio/pull/242 https://github.com/python/asyncio/pull/242 The author is using the latest release of asyncio, will be good to mention that.
harlowja 11y ago
Anyone know when most of the python stdlib (the batteries included part) will start itself using async/await (for any/all things that could block)?? I suppose it will never change due to backwards compat. (which feels odd...)?
- baxter001 11y ago
  The cost of such a change also compares badly with being able to wrap anything you want so easily.
- sametmax 11y ago
  More than that, believe it or not, but synchronicity and blocking code is considered a feature: it's simple and straighforward. Most code doesn't need to be non blocking. We only hear so much about non blocking because the Web is the big thing and Web programming benefits from it. But there are hundreds of other fields in programming where IO is not the main issue, and hence don't need the additional complexity of being async. Granted, asyncio does a good job at making async easier, but imperative code will always be easier to use, learn and teach. These qualities are very important, and made Python famous : easy to grow with, and yet powerful. So the stdlib will remain as-is, the useful, easy and beautiful thing it has been. We may see additions made to it so you can ALSO, when needed, do more stuff async. But really, I think we will see more and more external lib just doing that, so for when you DO need async, you will just pip install. Things I do hope will have an additional async API though: - urllib; - sqlite3; - subprocess.
  - AnkhMorporkian 11y ago
    You know what I've always wanted to see (and it's quite possible one exists; I have never actively searched for one)? A completely asynchronous programming language. Every single operation is carried out asynchronously; everything from addition to while loops. Synchronization would be a nightmare, but I think it would be a fun exercise to see what you could do in that framework.
    - nine_k 11y ago
      Is lazy evaluation asynchronous enough for you? It's more predictable than random asynchronous I/O, though, so synchronization is easier.
    - Natanael_L 11y ago
      There are some doing that by default, but I can't remember the names. Every operation can be done in parallel unless you declare a necessary order for some subset(s) of operations.
    - luhn 11y ago
      How would that work? My understanding of async is that it's a way of interacting with IO. So how would the examples you give (additional, while loops) behave asynchronously, since they don't involve IO? (Except for maybe memory IO..?)
      - chrisseaton 11y ago
        Addition still takes time though doesn't it? The idea is I could start an addition operation, and then be notified in some way when it's done. I think the idea most similar to what the grandparent wants is instruction-level dataflow, where programs are DAGs, and instructions are only run when their inputs are ready, rather than when a program counter reaches them. Physical dataflow machines were built that did this in hardware.
        luhn 11y ago
        A program as a DAG.... Whoa. That's an interesting concept to try to wrap your mind around.
- luhn 11y ago
  I think it's a combination of impracticality and lack of motivation that will keep the stdlib largely blocking. * File IO (`open()`) will remain blocking because there's no practical way to implement it in asyncio. This isn't Python's fault, the OS support is a mess. * Async Socket IO is already implemented in the stdlib. * The stdlib HTTP (urllib, urllib2) is pretty awful and I don't know of anybody who uses it, instead they opt for Requests. aiohttp is the asyncio equivalent of Requests. * SMTP would be nice to see async. * SQLite relies on file IO, so is likewise doomed to always block. I can't think of anything else in the stdlib that is IO-driven. Edit: sametmax mentioned subprocess, which I had overlooked. That would be a nice one to see.
- 0x0 11y ago
  You can't just sprinkle async onto existing functions and have the awaiting code resume magically: Going async also requires an event loop. Adding an event loop to an environment where there previously has not been one is non-trivial. Also, changing existing blocking methods to be non-blocking would break everything that uses (and expects) those APIs to be blocking.
bawana 11y ago
Async is not new. life has been doing it for millennia. Your heart does not stop beating while you breathe, yet both systems are interrelated. The body accomplishes this through the use of 'buffers'. Pause one system long enough and the other will fail. So we have evolved a 'scheduler' - the hindbrain that starts to scream louder and louder when the event queue of one system becomes too long. It might be instructive to model this in silicon. Have one cpu core be the scheduler. Have every set of interrelated processes (that are Async) register their variables with the scheduler. The scheduler then keeps track of these sets. When a variable in a set changes, pause that process for two cycles.
macavity23 11y ago
A good article, but it should mention async network programming (probably with http://www.tornadoweb.org/ http://www.tornadoweb.org/) which is the major driving use case behind the adoption of async programming patterns.

147 points by snw 11y ago

If you don't have the latest asyncio version, change `ensure_future` to `async`. > asyncio.async(display_date(1, loop)) See this https://github.com/python/asyncio/pull/242 https://github.com/python/asyncio/pull/242 The author is using the latest release of asyncio, will be good to mention that.

harlowja 11y ago

Anyone know when most of the python stdlib (the batteries included part) will start itself using async/await (for any/all things that could block)?? I suppose it will never change due to backwards compat. (which feels odd...)?

baxter001 11y ago
The cost of such a change also compares badly with being able to wrap anything you want so easily.
sametmax 11y ago
More than that, believe it or not, but synchronicity and blocking code is considered a feature: it's simple and straighforward. Most code doesn't need to be non blocking. We only hear so much about non blocking because the Web is the big thing and Web programming benefits from it. But there are hundreds of other fields in programming where IO is not the main issue, and hence don't need the additional complexity of being async. Granted, asyncio does a good job at making async easier, but imperative code will always be easier to use, learn and teach. These qualities are very important, and made Python famous : easy to grow with, and yet powerful. So the stdlib will remain as-is, the useful, easy and beautiful thing it has been. We may see additions made to it so you can ALSO, when needed, do more stuff async. But really, I think we will see more and more external lib just doing that, so for when you DO need async, you will just pip install. Things I do hope will have an additional async API though: - urllib; - sqlite3; - subprocess.
- AnkhMorporkian 11y ago
  You know what I've always wanted to see (and it's quite possible one exists; I have never actively searched for one)? A completely asynchronous programming language. Every single operation is carried out asynchronously; everything from addition to while loops. Synchronization would be a nightmare, but I think it would be a fun exercise to see what you could do in that framework.
  - nine_k 11y ago
    Is lazy evaluation asynchronous enough for you? It's more predictable than random asynchronous I/O, though, so synchronization is easier.
  - Natanael_L 11y ago
    There are some doing that by default, but I can't remember the names. Every operation can be done in parallel unless you declare a necessary order for some subset(s) of operations.
  - luhn 11y ago
    How would that work? My understanding of async is that it's a way of interacting with IO. So how would the examples you give (additional, while loops) behave asynchronously, since they don't involve IO? (Except for maybe memory IO..?)
    - chrisseaton 11y ago
      Addition still takes time though doesn't it? The idea is I could start an addition operation, and then be notified in some way when it's done. I think the idea most similar to what the grandparent wants is instruction-level dataflow, where programs are DAGs, and instructions are only run when their inputs are ready, rather than when a program counter reaches them. Physical dataflow machines were built that did this in hardware.
      - luhn 11y ago
        A program as a DAG.... Whoa. That's an interesting concept to try to wrap your mind around.
luhn 11y ago
I think it's a combination of impracticality and lack of motivation that will keep the stdlib largely blocking. * File IO (`open()`) will remain blocking because there's no practical way to implement it in asyncio. This isn't Python's fault, the OS support is a mess. * Async Socket IO is already implemented in the stdlib. * The stdlib HTTP (urllib, urllib2) is pretty awful and I don't know of anybody who uses it, instead they opt for Requests. aiohttp is the asyncio equivalent of Requests. * SMTP would be nice to see async. * SQLite relies on file IO, so is likewise doomed to always block. I can't think of anything else in the stdlib that is IO-driven. Edit: sametmax mentioned subprocess, which I had overlooked. That would be a nice one to see.
0x0 11y ago
You can't just sprinkle async onto existing functions and have the awaiting code resume magically: Going async also requires an event loop. Adding an event loop to an environment where there previously has not been one is non-trivial. Also, changing existing blocking methods to be non-blocking would break everything that uses (and expects) those APIs to be blocking.

bawana 11y ago

Async is not new. life has been doing it for millennia. Your heart does not stop beating while you breathe, yet both systems are interrelated. The body accomplishes this through the use of 'buffers'. Pause one system long enough and the other will fail. So we have evolved a 'scheduler' - the hindbrain that starts to scream louder and louder when the event queue of one system becomes too long. It might be instructive to model this in silicon. Have one cpu core be the scheduler. Have every set of interrelated processes (that are Async) register their variables with the scheduler. The scheduler then keeps track of these sets. When a variable in a set changes, pause that process for two cycles.

macavity23 11y ago

A good article, but it should mention async network programming (probably with http://www.tornadoweb.org/ http://www.tornadoweb.org/) which is the major driving use case behind the adoption of async programming patterns.