5 ms·

Python Coroutines with Async and Await

• midko 11y ago
  relevant: http://lwn.net/Articles/644128/ http://lwn.net/Articles/644128/ ("PEP 492 vs. PEP 3152, new round")
  • ptx 11y ago
    It's linked towards the end of the article, after it's been explained what PEP 3152 is, so reading it after the main article is less confusing. :)
• gamesbrainiac 11y ago
  Python 3.5 is turning out to be a version that people might actually be interested in.
  • pekk 11y ago
    I don't see why this should be a tipping point; this introduces no new capabilities, and a lot of hairy new syntax and magic methods.
    • nine_k 11y ago
      Async operations were the one biggest hurdle when writing efficient I/O-bound programs. (Definitely, Python is not about efficient parallel CPU-bound programming, but it's fine for many I/O-bound tasks.)
• lyschoening 11y ago
  How would one await multiple coroutine/futures? EDIT: await asyncio.gather(fut1, fut2)
  • dcarp 11y ago
    Why not: async asyncio.wait([fut1, fut2])
    • thomasjames 11y ago
      The async keyword is used in coroutine declarations, so it wouldn't be valid syntax.
      • dcarp 11y ago
        My bad... of course I wanted to write await asyncio.wait([fut1, fut2])
• themartorana 11y ago
  Coming from a Go mindset, having moved a significant portion of dev to Go from Python, in many ways for "goroutines", it seems as if there still isn't a parallel, unless I'm missing something. In the simplest case in Go, it's easy for me to toss off some concurrent processing and not wait or care about what happens in those functions. For all intents and purposes, it feels like parallel execution (I know, I know). I can't quite wrap my head around how generators or async/await gives me the same functionality. What am I missing?
  • falcolas 11y ago
    Do you regularly set "GOMAXPROCS"? If not, you're just using coroutines in Go as well, just with less control over the scheduling of the execution.
    • themartorana 11y ago
      I do. But it looks like I'm comparing apples and oranges. Goroutines and Erlang processes feel very much like lightweight threads. Async/Await still pause for return, but doesn't block the calling thread (it seems like). I'm trying to do some research on the patterns (mostly C# vs Go because this PEP is still newish) and it SEEMS like that's what's happening here.
      • Rohansi 11y ago
        Goroutines and C# async/await are quite similar in use and behavior, the implementations are very different though. Goroutines have their own stack but Tasks don't, they store values that cross await boundaries in a compiler-generated class and everything else uses the worker thread's stack. Async/await doesn't feel like lightweight threads because you need to explicitly await tasks. You could invert the keyword usage to match Go (implicit await, explicit go) and end up with something like goroutines.
• kanja 11y ago
  This is pretty great - One feature I really liked from go was channels and the ability for goroutines to communicate with each other. Anyone know there a pep open for a similar idea in the python world?
  • e271828 11y ago
    You can get channel-like behavior using async/await and a custom event loop. I wrote a little example[1] that has a bare-bones implementation of this. I used it to translate the prime sieve example from Go[2] almost directly to Python. The code uses "message = await channel.receive()" to mimic Go's "message <- channel". Instead of using "go func()" to fire off a goroutine, I use loop.run(func()) to add the PEP492 coroutine to my simple event loop. It's not an efficient implementation - it was really meant as a proof of concept that you can use async/await in your own code without any reference to asyncio. [1] https://gist.github.com/vrajivk/c505310fb79d412afcd5#file-sieve-py https://gist.github.com/vrajivk/c505310fb79d412afcd5#file-si... https://gist.github.com/vrajivk/c505310fb79d412afcd5#file-channel-py https://gist.github.com/vrajivk/c505310fb79d412afcd5#file-ch... [2] https://golang.org/doc/play/sieve.go https://golang.org/doc/play/sieve.go
• dom96 11y ago
  This is something that a Python-like language by the name of Nim already has. Interesting to see Python get it now too. http://nim-lang.org/docs/asyncdispatch.html http://nim-lang.org/docs/asyncdispatch.html
• strommen 11y ago
  It's fascinating how async/await are spreading from C# to other languages like JavaScript and Python. As far as I know, no languages had anything like it before C# added it in ~2011. What was the last truly-new language feature that became popular across other languages like this?
  • ceronman 11y ago
    The idea of using coroutines for asynchronous programming is quite old actually. ML has used it since the 80s [1]. Python itself had it way before C# with Twisted's inlineCallbacks [2]. What C# (actually it was F#) introduced was the specific 'await' and 'async' syntax keywords. Those are being copied in other languages, and they're just sugar over coroutines. [1] http://www.cambridge.org/gb/academic/subjects/computer-science/distributed-networked-and-mobile-computing/concurrent-programming-ml http://www.cambridge.org/gb/academic/subjects/computer-scien... [2] http://blog.mekk.waw.pl/archives/14-Twisted-inlineCallbacks-and-deferredGenerator.html http://blog.mekk.waw.pl/archives/14-Twisted-inlineCallbacks-...
  • adultSwim 11y ago
    It's my understanding that this came out of the programming language research community and is just now trickling into production languages (just like generics did). We're basically just getting the up to speed with ML from the 80's.
  • est 11y ago
    In python world, every networking programming trick trace back to Twisted.
• tveita 11y ago
  The Python standard library comes with support for a lot of common network protocols - TLS, HTTP, SMTP, etc. I'm wondering how this will be handled in async. Will everything need to be reimplemented as separate async implementations, or is there a way to have a common implementation that supports both sync and async? Will e.g. async HTTP be added to the standard library? I think this was a topic of discussion when they first added asyncio, but I never saw an official decision.
• mhomde 11y ago
  Having used async & await extensively in C# they're pretty great... but I can't help thinking we're explicitly declaring something that mostly could be implicit if the language was designed right. For example: if I'm calling an IO method that returns a value it should be able to figure that out and orchestrate the code to run asynchronously and await for the value when it's referenced. If I have several asynchronous calls methods in a method it should be able to optimize it so they all run in parallel as optimized as possible. It might need some added declarative syntax on methods but I'd rather prefer that and make the compiler smarter, instead of offloading that complexity on the programmer per default. Sure you should be able to explicitly control asynchronicity if needed but in most cases that doesn't seem optimal nor necessary Adding async/await all over your code adds to the "crud" that makes code more verbose and harder to read and understand
  • tghw 11y ago
    You need that "crud" so that you actually know what your code is doing. If the compiler automatically spins up another thread and finishes there while you return from that function without you knowing it would lead to all sorts of race conditions and other threading headaches. Making them explicit makes it clear what the code is doing.
    • mhomde 11y ago
      that's not necessarily true, in some cases yes. But as I said, control could and should be there for the asking, I don't want a ORM style black box where god knows what happens behind the scenes... but I think you could declaratively markup methods so the compiler knows what is safe to do. For instance marking them as pure, mutex etc etc. It's kinda the same thing except telling the compiler what to do you instruct it what's safe to do, and it can make smart decisions from that. If the compiler/environment detects any ambiguity or risk it might even query the programmer to declare his intent
  • dragonwriter 11y ago
    > Having used async & await extensively in C# they're pretty great... but I can't help thinking we're explicitly declaring something that mostly could be implicit if the language was designed right. Yes, basically, they are explicitly declaring dataflow dependencies. There are languages that handle this automatically, e.g., Oz [0]. OTOH, async/await is a solution that can be retrofitted onto an existing language that isn't fundamentally designed around dataflow dependencies without changing the semantics of existing code. [0] as implemented in the Mozart Programming System, https://mozart.github.io/mozart-v1/doc-1.4.0/ https://mozart.github.io/mozart-v1/doc-1.4.0/
• PopsiclePete 11y ago
  The async/await model gaining traction makes me really sad. I use it all the time and it just seems like a giant hack, a band-aid to hide "scary complexity" from the developers. A typical Microsoft approach to things, but it should stay there. First issue - the necessity for at least two of each method: "DoSomething" and "DoSomethingAsync". Weak. Now as a library developer, you just doubled your work. Sure, sometimes one method could wrap the other. But it's still weak sauce. I much prefer the Go model - you only code "sync" API's, and let the consumer of your API wrap it in a go-routine and make it "async" if they wish. Much simpler, better, composable. Async/await is also gaining traction in Javascript. Really, really sad we still try to avoid concurrent programming and how to do it right vs easy.
  • tghw 11y ago
    It's trivial to wrap existing functions to make an async version. public Task<ReturnType> DoSomethingAsync(args) { return await Task.Run(() => DoSomething(args)); } In fact, it's so easy that I can think of at least two ways to automatically add async methods, either at compile time or at runtime. I would have to look into it a little more, but I think you could even write an extension method on object that would add the async version via reflection. The upside of async/await is that you avoid callback hell. The code stays cleaner and is easier to read, both of which seem like wins to me.
    • PopsiclePete 11y ago
      That's a false choice you're presenting. "Callback hell" or "async/await". We have way more choices and opportunities. Look at Go's CSP implementation, or Erlang's. No callback hell anywhere.
      • tghw 11y ago
        Right, but the languages that are looking at async/await weren't designed with that in mind. You could do a major overhaul of the language, or you could bolt on async/await. While Go/Erlang's way of doing things might be preferable to some, I'm not sure I see async/await as being bad enough to justify a major language overhaul.