11 ms·

PEP 492 – Coroutines with async and await syntax

234 points by 1st1 12y ago

sylvinus 12y ago
New features like this are exactly what we need to encourage more people (including myself...) to invest time porting their projects to Python 3! I look forward to seeing this approved.
- endymi0n 12y ago
  Exciting but toothless "features" like this are exactly what we need to encourage more people to invest time porting their projects to languages with proper concurrency primitives like Go, Rust or Scala... :)
hendzen 12y ago
And here's a proposal for something quite similar in C++, written by the author of Boost Asio: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n4453.pdf http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n445...
jhgg 12y ago
Although this is great and all, what does this do compared to green threads (gevent, eventlet, etc...)?
- nchammas 12y ago
  I'm not sure from what perspective you were looking to compare asyncio to green threads (performance, readability, etc.) but there is an interesting blog post [0] by the lead architect of Twisted about the fundamental difference in programming model between explicit yielding (e.g. asyncio) and implicit yielding (e.g. green threads, regular threads). It may give you some answers. PEP 492, from what I understood, is more about streamlining the syntax, and the major differences between asyncio (pre this proposed syntax) and green threads in terms of programming model still apply. [0] https://glyph.twistedmatrix.com/2014/02/unyielding.html https://glyph.twistedmatrix.com/2014/02/unyielding.html
  - nas 12y ago
    Glyph is a smart guy and has been doing async code for a very long time (at least in computing terms). It would be wise to consider his arguments. I've done a project making heavy use of Python 3's asyncio feature. It is much nicer than doing the same job with threads or call-backs, in my experience. I'd welcome nicer syntax but the current functionality seems to work well. I can't understand why someone would design a language in 2009 (Node.js) that uses callbacks. For simple programs it works but it doesn't take long to end up with an unmaintainable mess (in my experience).
    - deleted 12y ago
      [deleted]
    - pekk 12y ago
      Why is it wise not to consider opposing arguments? Because people who disagree with Glyph are not "smart guys"?
      - nas 12y ago
        Quite the logic you have going on there. Certainly you should also consider other arguments.
    - sylvinus 12y ago
      With all the respect I have for Glyph, he's the least objective person to talk about this, having so much invested in one side of the argument.
  - harlowja 12y ago
    IMHO the following are much more useful to read over: * https://review.openstack.org/#/c/153298/ https://review.openstack.org/#/c/153298/ * http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an... * http://lists.openstack.org/pipermail/openstack-dev/2014-February/thread.html#26237 http://lists.openstack.org/pipermail/openstack-dev/2014-Febr... And others...
- bayesianhorse 12y ago
  The problem with "green threads" is that their context switching is unpredictable. You never know where the context is switched, potentially confusing C APIs and such. Asyncio "threads", on the other hand, never switch context between "yield from" instructions. This makes it usable with the most bungled and complicated C APIs (e.g. Blender...), and also lets us reason more clearly about the concurrent behavior.
  - harlowja 12y ago
    I'm going to go out on a limb and say this is a fallacy, you have just turned yourself (the programmer) into the context switching routine that the operating system (or python itself) does for you. IMHO computers are good (and can be made better) at scheduling and we should really avoid trying to do it ourselves manually... Getting used to thinking about where code can be context switched is IMHO worth more than shoving it under a rug and turning the programmer into the scheduler...
    - bayesianhorse 12y ago
      It's not as bad as you think it is. First thing: Asyncio doesn't force the programmer to schedule anything. The event loop does that. Also, this kind of message-passing concurrency is not meant for cpu-intensive work, but rather for situations where there's nothing to do in a particular task much of the time. Concurrent threads/tasks in frameworks like NodeJs, Tornado and Asyncio are very good at doing nothing. For example in a web server, a traditionally threaded "worker" spends a lot of time waiting for a connection, waiting for database requests to finish, and so on. The operating system can use that time to execute other threads. But the memory of the thread sits idle during that time. Scheduling in asyncio is not imposed by the operating system or the programmer, and only partly by the event loop, but largely because of external circumstances like network latency/bandwith, how long do the database queries take, and how fast the user is acting.
      - harlowja 12y ago
        Who writes the 'yield from' or 'async' or @coroutine (yield points, aka context switching 'places', that u have to pick correctly)...? I thought that was the programmer ;-) As a comparison to glyph http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an... (the creator/author of sqlalchemy) is a pretty good read :-)
deleted 12y ago
[deleted]
rdtsc 12y ago
Looks good, but coming from someone who used Twisted for a few years, I had found deferreds to be messy and switched to yield-type co-routines (http://twistedmatrix.com/documents/10.2.0/api/twisted.internet.defer.inlineCallbacks.html http://twistedmatrix.com/documents/10.2.0/api/twisted.intern...), but eventually I found those pretty verbose as well. Small demo examples always look clean and fun, but large applications at the top level will end up looking basically like this: y = yield f(x) z = yield g(y) w = yield h(z) ... In this case it would be async/await instead of pure yields. The worst thing was having to hunt for Twisted version of libraries. "Oh you want to talk XMPP? Nah, can't use this Python library, have to find the Twisted version of it". It basically split the library ecosystem. Now presumably it will be having to look for async/await version of libraries that do IO.
- nostrademons 12y ago
  I found the same, working with AppEngine NDB in an app that talked to a dozen or so backends. It's still a lot better than the Node.js (pre-ES6) style of: f(x, function(y) { g(y, function(z) { h(z, function(w) { do_something_with_w(w); }); }); }); What would be a better syntax? The Java/C++ way of threads & hidden shared-state concurrency is a total mess; it hides all the potential yield points, so you never know when your flow of control might block or what shared state might need locking. Channels in Golang are better - they at least have some syntactic support - but the reification of the yield point into a concrete channel can end up creating a fair bit of boilerplate in the common case where you make a bunch of one-off async RPCs to remote services. Maybe Erlang has it right, where the pid is an implicit channel you can send messages to - but then you still need to pass that into any async library function, and it gives you no typing discipline. Maybe we really need something like futures where the syntax: y <- f(x) z <- g(y) w <- h(z) means "wait for the promise returned by expression f(z) to resolve, returning control to the executor, and then assign the result to y." The splitting-the-library-ecosystem thing is a big part of it too (and why ES6 won't magically fix the Node ecosystem), but that's why Python is putting async into the language & standard library itself. At least then the stdlib will support it, and there will be strong social pressure to use the same concurrency mechanism in all libraries.
orf 12y ago
This is the kind of feature that will get people really interested in moving code to Python 3. Everyone said it was lacking a blockbuster feature, this could be it.
- saurik 12y ago
  As someone still using Python 2 (due to the very large number of libraries and tools I rely on for Python 2), I tend to look at these kinds of features more with the eye of "why not implement this for the language people are actually using more often, aka Python 2, rather than trying to use it to fight an ideological battle to move people to a language people aren't, aka Python 3?". The mentality just makes me unhappy with the attempts to somehow force everyone to move, rather than suddenly excited to use Python 3. I thereby ask: is there something about Python 3 that makes this feature extremely easier to implement or extremely easier to integrate? If not, wouldn't it be more interesting to have "more impact" by improving the lives of a larger number of people? Python 3 was maybe an interesting experiment, but given how well Ruby 1.8->1.9 went and how poorly Perl 5->6 has been, it seems like people should be learning some lessons and fighting a more winnable battle. Maybe the people who want to encourage people to migrate to Python 3 should he spending their time not providing "killer features" but instead narrowing the gap between the two versions, not by back porting features to Python 2.7, but by making it easier to use code written for Python 2 in Python 3. It should not have taken until 3.3 to see the u'' syntax return for compatibility with Python 2. :( The current strategy of "try to convince an army of people to port a bunch of code from Python 2 to Python 3 while people are at the same time still often writing code for Python 2", which is what we are seeing being asked of people in other posts here over the last few days (there was a post about this for Debian) just seems like a waste of effort leading to tons of lost ground to alternative languages.
  - rspeer 12y ago
    This is new syntax that requires at least a new minor version. It can't be back-ported to Python 2.7. You might even say that what people find so appealing about Python 2.7 these days is that its syntax will never change. Python 3 is not Perl 6, and that's a banal comparison to make. Python 3 has had many releases, and has had most packages ported to it. Perl 6 has had no releases. Ruby 1.9 should be an example that making incompatible changes that force everyone to upgrade is possible as long as you make timely releases. What gap do you still think there is to be narrowed? We're not on 3.2 anymore. Complaining that "it should not have taken until 3.3" doesn't matter now. What most people are arguing is quite the opposite of you -- they're saying they could switch to Python 3, and they would if there were a killer feature that mattered to them, but just being cleaner and making the core developers happier isn't enough for them to change the way they use Python. So I say, bring on the killer features.
sametmax 12y ago
I read it, but I don't see the benefit of it vs yield from and coroutines. Can somebody explain ?
- ceronman 12y ago
  This is explained in the Rationale and Goals[1] section of the PEP: it is easy to confuse coroutines with regular generators, since they share the same syntax; async libraries often attempt to alleviate this by using decorators (e.g. @asyncio.coroutine [1] ); it is not possible to natively define a coroutine which has no yield or yield from statements, again requiring the use of decorators to fix potential refactoring issues; support for asynchronous calls is limited to expressions where yield is allowed syntactically, limiting the usefulness of syntactic features, such as with and for statements. [1] https://www.python.org/dev/peps/pep-0492/#rationale-and-goals https://www.python.org/dev/peps/pep-0492/#rationale-and-goal...
  - sametmax 12y ago
    I just said I read it. Copying it and pasting it doesn't make it more easy to understand. I just can't somebody somebody would like to introduce a whole new syntaxe just for dislike of decorator and 2 edge cases twisted/tornado/greenlet lived with for years. So I though I missed something. Espacially since we have ways to do async and "there should be one way to do it" is as much important as keeping builtins number down. So I'm asking again : am I missing something or is it again on of these "I don't like it so let's do it like in language x" claims like we had for brackets, lambdas et alike ?
    - jacobolus 12y ago
      Here are some comments from python people (from the mailing list): Victor Stinner: """As a contributor to asyncio, I'm a strong supporter of this PEP. IMO it should land into Python 3.5 to confirm that Python promotes asynchronous programming (and is the best language for async programming? :-)) and accelerate the adoption of asyncio "everywhere". ¶The first time Yury shared with me his idea of new keywords (at Pycon Montreal 2014), I understood that it was just syntax sugar. In fact, it adds also "async for" and "async with" which adds new features. They are almost required to make asyncio usage easier. "async for" would help database ORMs or the aiofiles project (run file I/O in threads). "async with" helps also ORMs.""" Marc-Andre Lemberg: """Thanks for proposing this syntax. With async/await added to Python I think I would actually start writing native async code instead of relying on gevent to do all the heavy lifting for me ;-)""" Guido: """I'm in favor of Yuri's PEP too, but I don't want to push too hard for it as I haven't had the time to consider all the consequences.""" etc. For more: https://mail.python.org/pipermail/python-ideas/2015-April/thread.html#33007 https://mail.python.org/pipermail/python-ideas/2015-April/th...
jMyles 12y ago
The "async with" is syntactically pretty similar to crosstown_traffic from hendrix (based on Twisted): http://hendrix.readthedocs.org/en/latest/crosstown_traffic/ http://hendrix.readthedocs.org/en/latest/crosstown_traffic/
ksec 12y ago
Off Topic: Does anyone know if there are anything similar for Ruby in the work? P.S - Ruby's issues system is hard to follow.
- dmgbrn 12y ago
  Yes and no. Ruby has Fibers, but they're roughly where Python is with generators as coroutines. They're wonderful when working with EM, and one time I wrote a massively overengineered Ruby Warrior program that allowed you to do things like "move(:left) until feel(:left).wall?" instead of doing one thing each time a method was called. This is exactly what makes coroutines so attractive in simulation programs. Still, Fibers are clumsy and counterintuitive and require some abstraction to produce a useful interface. Also, there are some problems related to overflowing the Fiber stack, especially in ridiculously deep Rails stacks, not sure if those have been resolved yet. I'm not sure that special syntax is necessary for widespread adoption of Fibers, since Ruby syntax is flexible enough to provide a sort of DSL for working with Fibers.
- riffraff 12y ago
  ruby used to have RCRs as "language level issue tracking", modeled after PEPs[0]. Sadly, it died some years ago. [0] https://www.ruby-lang.org/en/news/2003/12/19/new-ruby-change-request-rcr-process/ https://www.ruby-lang.org/en/news/2003/12/19/new-ruby-change...
mangeletti 12y ago
Dear Python, I love you.
Animats 12y ago
Python 3 does not need cool new features right now. Python 3 needs to get its existing libraries working properly. People aren't using Python 3 because the out-of-the-box experience sucks. We covered this yesterday on HN.[1] [1] https://news.ycombinator.com/item?id=9378898 https://news.ycombinator.com/item?id=9378898
- scrollaway 12y ago
  Who made you BDFL?
bayesianhorse 12y ago
At first look I don't like this. "async def" feels ugly, and I am not sure this is just because of the unfamiliarity. I don't believe coroutines/async routines will ever be practical without decorators, so you might as well keep them and not disturb one of the most basic syntax rules of Python.
- 1st1 12y ago
  > I don't believe coroutines/async routines will ever be practical without decorators Why do you believe so? Have you seen the implementation of asyncio.coroutine? Have you read the PEP thoroughly and saw the downsides of using a decorator/generators?
  - RickHull 12y ago
    For those of us haven't, can you make the downsides clear? (links or text reply would be fantastic)
    - 1st1 12y ago
      Please read the Rationale of the PEP. edit: I can't explain topics better than I did so in the rationale section of the PEP. If I could, I would have explained it better in that section in the first place ;) As for asyncio.coroutine decorator -- it's just a very simple wrapper, that makes sure that the decorated object is a generator-function. If it's not -- it wraps it in one. It also does some magic to enable debug features. But with some serious shortcomings (that is also explained in the PEP in great detail). My point is: there is absolutely no other value in that decorator. There is nothing fundamental that it does, it just fixes the warts. Documentation, tooling, "easier to spot", etc arguments are unfortunately weak. The PEP makes coroutines in python a first-class language concept, with all the benefits you can have from it (better support in IDEs, tooling, sphinx, less questions on StackOverflow). Disclaimer: I'm the PEP author. I'm also python core developer, and I contributed to asyncio a lot.
  - bayesianhorse 12y ago
    The major benefit I see to the decorator syntax is that there is not an extra distinction about functions. So currently we may have normal functions and generator functions. The latter are mostly functions which just return an iterator. Do we have to make a second distinction? Most of this is just "feeling". But good Python makes me feel good, so I sort of expect that from new features if I am to vote them up. I might like "codef" more than "async def", if there absolutely have to be first-class coroutines.
bayesianhorse 12y ago
I am detecting a high level of "Py3K denial syndrome" in this discussion. The greatness of Python would have been impossible without rejecting virtually all feature requests. To move the vision of Python forward, it was necessary to depart with old syntax. Python 3 is happening. Deal with it.
- frik 12y ago
  > Python 3 is happening. Deal with it. Moving a large codebase to v3 is expensive for almost no benefits? Some parts of the community already moves on to Go, etc. Many third party libraries are v2 only and unmaintained. Many languages that broke backwards compatibility ultimately failed (their community moved on) or split their community. Examples: Modula 2/3/Oberon4, Perl 4/5/6, Lua 5.1(LuaJIT)/5.2, Visual Basic 6/.Net, QBasic/Visual Basic 1, VBA/VBA.Net, J++/J.Net/C#, etc. Fibers and coroutines are key features of certain language and API's for decades. WinAPI16 had already Fibers (MS Word), Lua's coroutines, Go's goroutines, etc.
  - pjmlp 12y ago
    Modula-2 and Modula-3 don't have the same authors, even if the name implies otherwise. Also Oberon had a different purpose than Modula-2.
    - frik 12y ago
      What's your point? It's what I meant with split the community. And I meant Oberon 2 (not 4 which doesn't exist) - two different Oberon 2 versions exist. My point was introducing incompatible language changes that splits the community usually isn't that good, in the long run.
      - pjmlp 12y ago
        Those languages are not related other than sharing the Mesa and Cedar language design ideas. In Oberon's case, you have actually Oberon, Oberon-2, Component Pascal, Active Oberon, Zonnon and Oberon-07. It is not the same as the next version of a given language, rather different approaches how to design memory safe languages for systems programming.
pjtr 12y ago
This reminds me very much of C# async and await (although "async for" and "async with" are very nice additions). It seems to me many other languages are now basically adopting wholesale these same features from C#. I'm curious, is this just my warped perception? Did this terminology and feature set previously exist in programming language theory or practice? I know F# had "async" for a while before C#. Was there "prior art" that looked basically the same in Lisp, Haskell, Scala, ML, etc. or in some research language long before that?
- kevingadd 12y ago
  C# has essentially put a common name to many existing CS features and popularized particular forms of them. A bunch of modern C# features are actually lifted from F#, which had them first on the .NET platform. Sometimes the C# version is simplified or has some of the sharp edges filed off (which makes it less powerful, but safer). Async/await is definitely one of these.
  - didibus 12y ago
    I love C#'s names for things. They take often academic names of non obvious meaning, and just replaces it with a common word whose meaning is obvious.
- pjmlp 12y ago
  You have to thank Erik Meijer for that, at least partially. "Confessions of a used programming language salesman (getting the masses hooked on haskell" http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.118.2025 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.118....
anon4 12y ago
Since a. this is functionally equivalent to coroutines implemented via library support through generators b. Python will never ever get rid of the GIL, i.e. won't have real parallelism (given that the JVM/CLR Python implementations are basically dead and PyPy is chasing the pype dream of STM, we're left with CPython the only actual implementation) Is there any reason to add new syntax? Other than wishful thinking that one day someone will write a Python runtime with proper JIT and parallelism support?
- pmalynin 12y ago
  Stackless Python
- ynik 12y ago
  async/await is useful for more than just 'real parallelism': its main use is coordinating concurrency. The GIL isn't a problem when dealing with async IO or async calls into C libraries.
- gamesbrainiac 12y ago
  There's already a good bit of work on getting rid of the GIL in pypy. pypy also has channels (borrowed from stackless).
- anonymoushn 12y ago
  This is a control flow primitive. You wouldn't hold off on implementing "call/cc" or "function calls" or "loops" until after you have multiple threads running the interpreter, imo.
carapace 12y ago
The Python 2.7 code freeze is totally working! All the horrid new BS is getting piled into 3.* and Python 2.7 is becoming sweet FORTRAN. Viva la BDFL! So wise! ;-)
- carapace 12y ago
  Downvotes? Folks I am a Python fanboy, I love it, am devoted to its success, and I'm delighted to use it every day in my work and personal life. I meant the above comment in earnest, not sarcastically. Python 3.* lets the people who like that sort of thing have their cake and eat it too (and I am one of the people who drools at new shiny tech.) But the fact that Python 2.7 is stable for the foreseeable future is something to celebrate in my opinion, not condemn.
Sidnicious 12y ago
This looks great, but it'll be even better with utilities that let you await multiple tasks in parallel: a, b, c = atools.join([f(), g(), h()]) and select among multiple async tasks (e.g. to implement a timeout): with atools.select({"res": f(), "timeout": timeout(10)}) as tasks: res, which = await tasks
- 1st1 12y ago
  There is equivalent to 'atools.join' in asyncio. And I suspect that we'll see libraries like 'atools' in the wild if the PEP is accepted. If those libraries will end up being popular, we'll add one of the to the standard library! I like the idea of 'a, b = await foo(), bar()' syntax. I'll think of how we can implement it and integrate to the PEP. Thanks!

234 points by 1st1 12y ago

New features like this are exactly what we need to encourage more people (including myself...) to invest time porting their projects to Python 3! I look forward to seeing this approved.

endymi0n 12y ago
Exciting but toothless "features" like this are exactly what we need to encourage more people to invest time porting their projects to languages with proper concurrency primitives like Go, Rust or Scala... :)

hendzen 12y ago

And here's a proposal for something quite similar in C++, written by the author of Boost Asio: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n4453.pdf http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n445...

jhgg 12y ago

Although this is great and all, what does this do compared to green threads (gevent, eventlet, etc...)?

nchammas 12y ago
I'm not sure from what perspective you were looking to compare asyncio to green threads (performance, readability, etc.) but there is an interesting blog post [0] by the lead architect of Twisted about the fundamental difference in programming model between explicit yielding (e.g. asyncio) and implicit yielding (e.g. green threads, regular threads). It may give you some answers. PEP 492, from what I understood, is more about streamlining the syntax, and the major differences between asyncio (pre this proposed syntax) and green threads in terms of programming model still apply. [0] https://glyph.twistedmatrix.com/2014/02/unyielding.html https://glyph.twistedmatrix.com/2014/02/unyielding.html
- nas 12y ago
  Glyph is a smart guy and has been doing async code for a very long time (at least in computing terms). It would be wise to consider his arguments. I've done a project making heavy use of Python 3's asyncio feature. It is much nicer than doing the same job with threads or call-backs, in my experience. I'd welcome nicer syntax but the current functionality seems to work well. I can't understand why someone would design a language in 2009 (Node.js) that uses callbacks. For simple programs it works but it doesn't take long to end up with an unmaintainable mess (in my experience).
  - deleted 12y ago
    [deleted]
  - pekk 12y ago
    Why is it wise not to consider opposing arguments? Because people who disagree with Glyph are not "smart guys"?
    - nas 12y ago
      Quite the logic you have going on there. Certainly you should also consider other arguments.
  - sylvinus 12y ago
    With all the respect I have for Glyph, he's the least objective person to talk about this, having so much invested in one side of the argument.
- harlowja 12y ago
  IMHO the following are much more useful to read over: * https://review.openstack.org/#/c/153298/ https://review.openstack.org/#/c/153298/ * http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an... * http://lists.openstack.org/pipermail/openstack-dev/2014-February/thread.html#26237 http://lists.openstack.org/pipermail/openstack-dev/2014-Febr... And others...
bayesianhorse 12y ago
The problem with "green threads" is that their context switching is unpredictable. You never know where the context is switched, potentially confusing C APIs and such. Asyncio "threads", on the other hand, never switch context between "yield from" instructions. This makes it usable with the most bungled and complicated C APIs (e.g. Blender...), and also lets us reason more clearly about the concurrent behavior.
- harlowja 12y ago
  I'm going to go out on a limb and say this is a fallacy, you have just turned yourself (the programmer) into the context switching routine that the operating system (or python itself) does for you. IMHO computers are good (and can be made better) at scheduling and we should really avoid trying to do it ourselves manually... Getting used to thinking about where code can be context switched is IMHO worth more than shoving it under a rug and turning the programmer into the scheduler...
  - bayesianhorse 12y ago
    It's not as bad as you think it is. First thing: Asyncio doesn't force the programmer to schedule anything. The event loop does that. Also, this kind of message-passing concurrency is not meant for cpu-intensive work, but rather for situations where there's nothing to do in a particular task much of the time. Concurrent threads/tasks in frameworks like NodeJs, Tornado and Asyncio are very good at doing nothing. For example in a web server, a traditionally threaded "worker" spends a lot of time waiting for a connection, waiting for database requests to finish, and so on. The operating system can use that time to execute other threads. But the memory of the thread sits idle during that time. Scheduling in asyncio is not imposed by the operating system or the programmer, and only partly by the event loop, but largely because of external circumstances like network latency/bandwith, how long do the database queries take, and how fast the user is acting.
    - harlowja 12y ago
      Who writes the 'yield from' or 'async' or @coroutine (yield points, aka context switching 'places', that u have to pick correctly)...? I thought that was the programmer ;-) As a comparison to glyph http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an... (the creator/author of sqlalchemy) is a pretty good read :-)

deleted 12y ago

[deleted]

rdtsc 12y ago

Looks good, but coming from someone who used Twisted for a few years, I had found deferreds to be messy and switched to yield-type co-routines (http://twistedmatrix.com/documents/10.2.0/api/twisted.internet.defer.inlineCallbacks.html http://twistedmatrix.com/documents/10.2.0/api/twisted.intern...), but eventually I found those pretty verbose as well. Small demo examples always look clean and fun, but large applications at the top level will end up looking basically like this: y = yield f(x) z = yield g(y) w = yield h(z) ... In this case it would be async/await instead of pure yields. The worst thing was having to hunt for Twisted version of libraries. "Oh you want to talk XMPP? Nah, can't use this Python library, have to find the Twisted version of it". It basically split the library ecosystem. Now presumably it will be having to look for async/await version of libraries that do IO.

nostrademons 12y ago
I found the same, working with AppEngine NDB in an app that talked to a dozen or so backends. It's still a lot better than the Node.js (pre-ES6) style of: f(x, function(y) { g(y, function(z) { h(z, function(w) { do_something_with_w(w); }); }); }); What would be a better syntax? The Java/C++ way of threads & hidden shared-state concurrency is a total mess; it hides all the potential yield points, so you never know when your flow of control might block or what shared state might need locking. Channels in Golang are better - they at least have some syntactic support - but the reification of the yield point into a concrete channel can end up creating a fair bit of boilerplate in the common case where you make a bunch of one-off async RPCs to remote services. Maybe Erlang has it right, where the pid is an implicit channel you can send messages to - but then you still need to pass that into any async library function, and it gives you no typing discipline. Maybe we really need something like futures where the syntax: y <- f(x) z <- g(y) w <- h(z) means "wait for the promise returned by expression f(z) to resolve, returning control to the executor, and then assign the result to y." The splitting-the-library-ecosystem thing is a big part of it too (and why ES6 won't magically fix the Node ecosystem), but that's why Python is putting async into the language & standard library itself. At least then the stdlib will support it, and there will be strong social pressure to use the same concurrency mechanism in all libraries.

orf 12y ago

This is the kind of feature that will get people really interested in moving code to Python 3. Everyone said it was lacking a blockbuster feature, this could be it.

saurik 12y ago
As someone still using Python 2 (due to the very large number of libraries and tools I rely on for Python 2), I tend to look at these kinds of features more with the eye of "why not implement this for the language people are actually using more often, aka Python 2, rather than trying to use it to fight an ideological battle to move people to a language people aren't, aka Python 3?". The mentality just makes me unhappy with the attempts to somehow force everyone to move, rather than suddenly excited to use Python 3. I thereby ask: is there something about Python 3 that makes this feature extremely easier to implement or extremely easier to integrate? If not, wouldn't it be more interesting to have "more impact" by improving the lives of a larger number of people? Python 3 was maybe an interesting experiment, but given how well Ruby 1.8->1.9 went and how poorly Perl 5->6 has been, it seems like people should be learning some lessons and fighting a more winnable battle. Maybe the people who want to encourage people to migrate to Python 3 should he spending their time not providing "killer features" but instead narrowing the gap between the two versions, not by back porting features to Python 2.7, but by making it easier to use code written for Python 2 in Python 3. It should not have taken until 3.3 to see the u'' syntax return for compatibility with Python 2. :( The current strategy of "try to convince an army of people to port a bunch of code from Python 2 to Python 3 while people are at the same time still often writing code for Python 2", which is what we are seeing being asked of people in other posts here over the last few days (there was a post about this for Debian) just seems like a waste of effort leading to tons of lost ground to alternative languages.
- rspeer 12y ago
  This is new syntax that requires at least a new minor version. It can't be back-ported to Python 2.7. You might even say that what people find so appealing about Python 2.7 these days is that its syntax will never change. Python 3 is not Perl 6, and that's a banal comparison to make. Python 3 has had many releases, and has had most packages ported to it. Perl 6 has had no releases. Ruby 1.9 should be an example that making incompatible changes that force everyone to upgrade is possible as long as you make timely releases. What gap do you still think there is to be narrowed? We're not on 3.2 anymore. Complaining that "it should not have taken until 3.3" doesn't matter now. What most people are arguing is quite the opposite of you -- they're saying they could switch to Python 3, and they would if there were a killer feature that mattered to them, but just being cleaner and making the core developers happier isn't enough for them to change the way they use Python. So I say, bring on the killer features.

sametmax 12y ago

I read it, but I don't see the benefit of it vs yield from and coroutines. Can somebody explain ?

ceronman 12y ago
This is explained in the Rationale and Goals[1] section of the PEP: it is easy to confuse coroutines with regular generators, since they share the same syntax; async libraries often attempt to alleviate this by using decorators (e.g. @asyncio.coroutine [1] ); it is not possible to natively define a coroutine which has no yield or yield from statements, again requiring the use of decorators to fix potential refactoring issues; support for asynchronous calls is limited to expressions where yield is allowed syntactically, limiting the usefulness of syntactic features, such as with and for statements. [1] https://www.python.org/dev/peps/pep-0492/#rationale-and-goals https://www.python.org/dev/peps/pep-0492/#rationale-and-goal...
- sametmax 12y ago
  I just said I read it. Copying it and pasting it doesn't make it more easy to understand. I just can't somebody somebody would like to introduce a whole new syntaxe just for dislike of decorator and 2 edge cases twisted/tornado/greenlet lived with for years. So I though I missed something. Espacially since we have ways to do async and "there should be one way to do it" is as much important as keeping builtins number down. So I'm asking again : am I missing something or is it again on of these "I don't like it so let's do it like in language x" claims like we had for brackets, lambdas et alike ?
  - jacobolus 12y ago
    Here are some comments from python people (from the mailing list): Victor Stinner: """As a contributor to asyncio, I'm a strong supporter of this PEP. IMO it should land into Python 3.5 to confirm that Python promotes asynchronous programming (and is the best language for async programming? :-)) and accelerate the adoption of asyncio "everywhere". ¶The first time Yury shared with me his idea of new keywords (at Pycon Montreal 2014), I understood that it was just syntax sugar. In fact, it adds also "async for" and "async with" which adds new features. They are almost required to make asyncio usage easier. "async for" would help database ORMs or the aiofiles project (run file I/O in threads). "async with" helps also ORMs.""" Marc-Andre Lemberg: """Thanks for proposing this syntax. With async/await added to Python I think I would actually start writing native async code instead of relying on gevent to do all the heavy lifting for me ;-)""" Guido: """I'm in favor of Yuri's PEP too, but I don't want to push too hard for it as I haven't had the time to consider all the consequences.""" etc. For more: https://mail.python.org/pipermail/python-ideas/2015-April/thread.html#33007 https://mail.python.org/pipermail/python-ideas/2015-April/th...

jMyles 12y ago

The "async with" is syntactically pretty similar to crosstown_traffic from hendrix (based on Twisted): http://hendrix.readthedocs.org/en/latest/crosstown_traffic/ http://hendrix.readthedocs.org/en/latest/crosstown_traffic/

ksec 12y ago

Off Topic: Does anyone know if there are anything similar for Ruby in the work? P.S - Ruby's issues system is hard to follow.

dmgbrn 12y ago
Yes and no. Ruby has Fibers, but they're roughly where Python is with generators as coroutines. They're wonderful when working with EM, and one time I wrote a massively overengineered Ruby Warrior program that allowed you to do things like "move(:left) until feel(:left).wall?" instead of doing one thing each time a method was called. This is exactly what makes coroutines so attractive in simulation programs. Still, Fibers are clumsy and counterintuitive and require some abstraction to produce a useful interface. Also, there are some problems related to overflowing the Fiber stack, especially in ridiculously deep Rails stacks, not sure if those have been resolved yet. I'm not sure that special syntax is necessary for widespread adoption of Fibers, since Ruby syntax is flexible enough to provide a sort of DSL for working with Fibers.
riffraff 12y ago
ruby used to have RCRs as "language level issue tracking", modeled after PEPs[0]. Sadly, it died some years ago. [0] https://www.ruby-lang.org/en/news/2003/12/19/new-ruby-change-request-rcr-process/ https://www.ruby-lang.org/en/news/2003/12/19/new-ruby-change...

mangeletti 12y ago

Dear Python, I love you.

Animats 12y ago

Python 3 does not need cool new features right now. Python 3 needs to get its existing libraries working properly. People aren't using Python 3 because the out-of-the-box experience sucks. We covered this yesterday on HN.[1] [1] https://news.ycombinator.com/item?id=9378898 https://news.ycombinator.com/item?id=9378898

scrollaway 12y ago
Who made you BDFL?

bayesianhorse 12y ago

At first look I don't like this. "async def" feels ugly, and I am not sure this is just because of the unfamiliarity. I don't believe coroutines/async routines will ever be practical without decorators, so you might as well keep them and not disturb one of the most basic syntax rules of Python.

1st1 12y ago
> I don't believe coroutines/async routines will ever be practical without decorators Why do you believe so? Have you seen the implementation of asyncio.coroutine? Have you read the PEP thoroughly and saw the downsides of using a decorator/generators?
- RickHull 12y ago
  For those of us haven't, can you make the downsides clear? (links or text reply would be fantastic)
  - 1st1 12y ago
    Please read the Rationale of the PEP. edit: I can't explain topics better than I did so in the rationale section of the PEP. If I could, I would have explained it better in that section in the first place ;) As for asyncio.coroutine decorator -- it's just a very simple wrapper, that makes sure that the decorated object is a generator-function. If it's not -- it wraps it in one. It also does some magic to enable debug features. But with some serious shortcomings (that is also explained in the PEP in great detail). My point is: there is absolutely no other value in that decorator. There is nothing fundamental that it does, it just fixes the warts. Documentation, tooling, "easier to spot", etc arguments are unfortunately weak. The PEP makes coroutines in python a first-class language concept, with all the benefits you can have from it (better support in IDEs, tooling, sphinx, less questions on StackOverflow). Disclaimer: I'm the PEP author. I'm also python core developer, and I contributed to asyncio a lot.
- bayesianhorse 12y ago
  The major benefit I see to the decorator syntax is that there is not an extra distinction about functions. So currently we may have normal functions and generator functions. The latter are mostly functions which just return an iterator. Do we have to make a second distinction? Most of this is just "feeling". But good Python makes me feel good, so I sort of expect that from new features if I am to vote them up. I might like "codef" more than "async def", if there absolutely have to be first-class coroutines.

bayesianhorse 12y ago

I am detecting a high level of "Py3K denial syndrome" in this discussion. The greatness of Python would have been impossible without rejecting virtually all feature requests. To move the vision of Python forward, it was necessary to depart with old syntax. Python 3 is happening. Deal with it.

frik 12y ago
> Python 3 is happening. Deal with it. Moving a large codebase to v3 is expensive for almost no benefits? Some parts of the community already moves on to Go, etc. Many third party libraries are v2 only and unmaintained. Many languages that broke backwards compatibility ultimately failed (their community moved on) or split their community. Examples: Modula 2/3/Oberon4, Perl 4/5/6, Lua 5.1(LuaJIT)/5.2, Visual Basic 6/.Net, QBasic/Visual Basic 1, VBA/VBA.Net, J++/J.Net/C#, etc. Fibers and coroutines are key features of certain language and API's for decades. WinAPI16 had already Fibers (MS Word), Lua's coroutines, Go's goroutines, etc.
- pjmlp 12y ago
  Modula-2 and Modula-3 don't have the same authors, even if the name implies otherwise. Also Oberon had a different purpose than Modula-2.
  - frik 12y ago
    What's your point? It's what I meant with split the community. And I meant Oberon 2 (not 4 which doesn't exist) - two different Oberon 2 versions exist. My point was introducing incompatible language changes that splits the community usually isn't that good, in the long run.
    - pjmlp 12y ago
      Those languages are not related other than sharing the Mesa and Cedar language design ideas. In Oberon's case, you have actually Oberon, Oberon-2, Component Pascal, Active Oberon, Zonnon and Oberon-07. It is not the same as the next version of a given language, rather different approaches how to design memory safe languages for systems programming.

pjtr 12y ago

This reminds me very much of C# async and await (although "async for" and "async with" are very nice additions). It seems to me many other languages are now basically adopting wholesale these same features from C#. I'm curious, is this just my warped perception? Did this terminology and feature set previously exist in programming language theory or practice? I know F# had "async" for a while before C#. Was there "prior art" that looked basically the same in Lisp, Haskell, Scala, ML, etc. or in some research language long before that?

kevingadd 12y ago
C# has essentially put a common name to many existing CS features and popularized particular forms of them. A bunch of modern C# features are actually lifted from F#, which had them first on the .NET platform. Sometimes the C# version is simplified or has some of the sharp edges filed off (which makes it less powerful, but safer). Async/await is definitely one of these.
- didibus 12y ago
  I love C#'s names for things. They take often academic names of non obvious meaning, and just replaces it with a common word whose meaning is obvious.
pjmlp 12y ago
You have to thank Erik Meijer for that, at least partially. "Confessions of a used programming language salesman (getting the masses hooked on haskell" http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.118.2025 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.118....

anon4 12y ago

Since a. this is functionally equivalent to coroutines implemented via library support through generators b. Python will never ever get rid of the GIL, i.e. won't have real parallelism (given that the JVM/CLR Python implementations are basically dead and PyPy is chasing the pype dream of STM, we're left with CPython the only actual implementation) Is there any reason to add new syntax? Other than wishful thinking that one day someone will write a Python runtime with proper JIT and parallelism support?

pmalynin 12y ago
Stackless Python
ynik 12y ago
async/await is useful for more than just 'real parallelism': its main use is coordinating concurrency. The GIL isn't a problem when dealing with async IO or async calls into C libraries.
gamesbrainiac 12y ago
There's already a good bit of work on getting rid of the GIL in pypy. pypy also has channels (borrowed from stackless).
anonymoushn 12y ago
This is a control flow primitive. You wouldn't hold off on implementing "call/cc" or "function calls" or "loops" until after you have multiple threads running the interpreter, imo.

carapace 12y ago

The Python 2.7 code freeze is totally working! All the horrid new BS is getting piled into 3.* and Python 2.7 is becoming sweet FORTRAN. Viva la BDFL! So wise! ;-)

carapace 12y ago
Downvotes? Folks I am a Python fanboy, I love it, am devoted to its success, and I'm delighted to use it every day in my work and personal life. I meant the above comment in earnest, not sarcastically. Python 3.* lets the people who like that sort of thing have their cake and eat it too (and I am one of the people who drools at new shiny tech.) But the fact that Python 2.7 is stable for the foreseeable future is something to celebrate in my opinion, not condemn.

Sidnicious 12y ago

This looks great, but it'll be even better with utilities that let you await multiple tasks in parallel: a, b, c = atools.join([f(), g(), h()]) and select among multiple async tasks (e.g. to implement a timeout): with atools.select({"res": f(), "timeout": timeout(10)}) as tasks: res, which = await tasks

1st1 12y ago
There is equivalent to 'atools.join' in asyncio. And I suspect that we'll see libraries like 'atools' in the wild if the PEP is accepted. If those libraries will end up being popular, we'll add one of the to the standard library! I like the idea of 'a, b = await foo(), bar()' syntax. I'll think of how we can implement it and integrate to the PEP. Thanks!