5 ms·

Anyone know when most of the python stdlib (the batteries included part) will start itself using async/await (for any/all things that could block)?? I suppose i

Anyone know when most of the python stdlib (the batteries included part) will start itself using async/await (for any/all things that could block)?? I suppose it will never change due to backwards compat. (which feels odd...)?

baxter001 11y ago
The cost of such a change also compares badly with being able to wrap anything you want so easily.
sametmax 11y ago
More than that, believe it or not, but synchronicity and blocking code is considered a feature: it's simple and straighforward. Most code doesn't need to be non blocking. We only hear so much about non blocking because the Web is the big thing and Web programming benefits from it. But there are hundreds of other fields in programming where IO is not the main issue, and hence don't need the additional complexity of being async. Granted, asyncio does a good job at making async easier, but imperative code will always be easier to use, learn and teach. These qualities are very important, and made Python famous : easy to grow with, and yet powerful. So the stdlib will remain as-is, the useful, easy and beautiful thing it has been. We may see additions made to it so you can ALSO, when needed, do more stuff async. But really, I think we will see more and more external lib just doing that, so for when you DO need async, you will just pip install. Things I do hope will have an additional async API though: - urllib; - sqlite3; - subprocess.
- AnkhMorporkian 11y ago
  You know what I've always wanted to see (and it's quite possible one exists; I have never actively searched for one)? A completely asynchronous programming language. Every single operation is carried out asynchronously; everything from addition to while loops. Synchronization would be a nightmare, but I think it would be a fun exercise to see what you could do in that framework.
  - nine_k 11y ago
    Is lazy evaluation asynchronous enough for you? It's more predictable than random asynchronous I/O, though, so synchronization is easier.
  - Natanael_L 11y ago
    There are some doing that by default, but I can't remember the names. Every operation can be done in parallel unless you declare a necessary order for some subset(s) of operations.
  - luhn 11y ago
    How would that work? My understanding of async is that it's a way of interacting with IO. So how would the examples you give (additional, while loops) behave asynchronously, since they don't involve IO? (Except for maybe memory IO..?)
    - chrisseaton 11y ago
      Addition still takes time though doesn't it? The idea is I could start an addition operation, and then be notified in some way when it's done. I think the idea most similar to what the grandparent wants is instruction-level dataflow, where programs are DAGs, and instructions are only run when their inputs are ready, rather than when a program counter reaches them. Physical dataflow machines were built that did this in hardware.
      - luhn 11y ago
        A program as a DAG.... Whoa. That's an interesting concept to try to wrap your mind around.
        harlowja 11y ago
        Just fyi, a library openstack (and myself and others) has created (in python) basically lets u program your workflow (really dataflow) as a DAG, and then the library will run it reliably (and in parallel and so-on). I'm more than willing to answer questions about how it does that if people are interested... http://docs.openstack.org/developer/taskflow/ http://docs.openstack.org/developer/taskflow/ https://pypi.python.org/pypi/taskflow https://pypi.python.org/pypi/taskflow And yes u can do things like built equation solvers to: http://docs.openstack.org/developer/taskflow/examples.html#linear-equation-solver-in-parallel http://docs.openstack.org/developer/taskflow/examples.html#l... And other neat things: http://docs.openstack.org/developer/taskflow/#examples http://docs.openstack.org/developer/taskflow/#examples
- ketralnis 11y ago
  sqlite3 is unlikely because it's only a small bridge to sqlite's own API which is blocking Still, it's possible to write blocking/nonblocking bridges for that stuff
  - sametmax 11y ago
    True. Yet a standard async DP API + wrapper around a process pool to integrate with it would be nice so ORMs move on. Righ now we are stuck with the status quo and all asyncio adapters for peewee / SQLAlchemy reinvent the will by creating some kind of pool and an ugly syntax to write each call.
luhn 11y ago
I think it's a combination of impracticality and lack of motivation that will keep the stdlib largely blocking. * File IO (`open()`) will remain blocking because there's no practical way to implement it in asyncio. This isn't Python's fault, the OS support is a mess. * Async Socket IO is already implemented in the stdlib. * The stdlib HTTP (urllib, urllib2) is pretty awful and I don't know of anybody who uses it, instead they opt for Requests. aiohttp is the asyncio equivalent of Requests. * SMTP would be nice to see async. * SQLite relies on file IO, so is likewise doomed to always block. I can't think of anything else in the stdlib that is IO-driven. Edit: sametmax mentioned subprocess, which I had overlooked. That would be a nice one to see.
0x0 11y ago
You can't just sprinkle async onto existing functions and have the awaiting code resume magically: Going async also requires an event loop. Adding an event loop to an environment where there previously has not been one is non-trivial. Also, changing existing blocking methods to be non-blocking would break everything that uses (and expects) those APIs to be blocking.

baxter001 11y ago

The cost of such a change also compares badly with being able to wrap anything you want so easily.

sametmax 11y ago

More than that, believe it or not, but synchronicity and blocking code is considered a feature: it's simple and straighforward. Most code doesn't need to be non blocking. We only hear so much about non blocking because the Web is the big thing and Web programming benefits from it. But there are hundreds of other fields in programming where IO is not the main issue, and hence don't need the additional complexity of being async. Granted, asyncio does a good job at making async easier, but imperative code will always be easier to use, learn and teach. These qualities are very important, and made Python famous : easy to grow with, and yet powerful. So the stdlib will remain as-is, the useful, easy and beautiful thing it has been. We may see additions made to it so you can ALSO, when needed, do more stuff async. But really, I think we will see more and more external lib just doing that, so for when you DO need async, you will just pip install. Things I do hope will have an additional async API though: - urllib; - sqlite3; - subprocess.

AnkhMorporkian 11y ago
You know what I've always wanted to see (and it's quite possible one exists; I have never actively searched for one)? A completely asynchronous programming language. Every single operation is carried out asynchronously; everything from addition to while loops. Synchronization would be a nightmare, but I think it would be a fun exercise to see what you could do in that framework.
- nine_k 11y ago
  Is lazy evaluation asynchronous enough for you? It's more predictable than random asynchronous I/O, though, so synchronization is easier.
- Natanael_L 11y ago
  There are some doing that by default, but I can't remember the names. Every operation can be done in parallel unless you declare a necessary order for some subset(s) of operations.
- luhn 11y ago
  How would that work? My understanding of async is that it's a way of interacting with IO. So how would the examples you give (additional, while loops) behave asynchronously, since they don't involve IO? (Except for maybe memory IO..?)
  - chrisseaton 11y ago
    Addition still takes time though doesn't it? The idea is I could start an addition operation, and then be notified in some way when it's done. I think the idea most similar to what the grandparent wants is instruction-level dataflow, where programs are DAGs, and instructions are only run when their inputs are ready, rather than when a program counter reaches them. Physical dataflow machines were built that did this in hardware.
    - luhn 11y ago
      A program as a DAG.... Whoa. That's an interesting concept to try to wrap your mind around.
      - harlowja 11y ago
        Just fyi, a library openstack (and myself and others) has created (in python) basically lets u program your workflow (really dataflow) as a DAG, and then the library will run it reliably (and in parallel and so-on). I'm more than willing to answer questions about how it does that if people are interested... http://docs.openstack.org/developer/taskflow/ http://docs.openstack.org/developer/taskflow/ https://pypi.python.org/pypi/taskflow https://pypi.python.org/pypi/taskflow And yes u can do things like built equation solvers to: http://docs.openstack.org/developer/taskflow/examples.html#linear-equation-solver-in-parallel http://docs.openstack.org/developer/taskflow/examples.html#l... And other neat things: http://docs.openstack.org/developer/taskflow/#examples http://docs.openstack.org/developer/taskflow/#examples
ketralnis 11y ago
sqlite3 is unlikely because it's only a small bridge to sqlite's own API which is blocking Still, it's possible to write blocking/nonblocking bridges for that stuff
- sametmax 11y ago
  True. Yet a standard async DP API + wrapper around a process pool to integrate with it would be nice so ORMs move on. Righ now we are stuck with the status quo and all asyncio adapters for peewee / SQLAlchemy reinvent the will by creating some kind of pool and an ugly syntax to write each call.

luhn 11y ago

I think it's a combination of impracticality and lack of motivation that will keep the stdlib largely blocking. * File IO (`open()`) will remain blocking because there's no practical way to implement it in asyncio. This isn't Python's fault, the OS support is a mess. * Async Socket IO is already implemented in the stdlib. * The stdlib HTTP (urllib, urllib2) is pretty awful and I don't know of anybody who uses it, instead they opt for Requests. aiohttp is the asyncio equivalent of Requests. * SMTP would be nice to see async. * SQLite relies on file IO, so is likewise doomed to always block. I can't think of anything else in the stdlib that is IO-driven. Edit: sametmax mentioned subprocess, which I had overlooked. That would be a nice one to see.

0x0 11y ago

You can't just sprinkle async onto existing functions and have the awaiting code resume magically: Going async also requires an event loop. Adding an event loop to an environment where there previously has not been one is non-trivial. Also, changing existing blocking methods to be non-blocking would break everything that uses (and expects) those APIs to be blocking.