12 ms·

Python 3.6 – Asynchronous Comprehensions

125 points by tbarbugli 10y ago

1st1 10y ago
Surprised to see this here :) I'm the author of the PEP, feel free to ask questions.
- sagnew 10y ago
  Just wanted to say excellent work. This is some convenient wizardy!
  - 1st1 10y ago
    Thanks ;)
- orf 10y ago
  Thank you so much for the pep, I've just finished writing a pretty neat tool using all the new 3.6 features and they are awesome, especially the comprehensions. Thanks :)
  - 1st1 10y ago
    Thanks! Did you use new async generators in your tool?
- jchassoul 10y ago
  I just want to say thank you for make this possible! this are the progressive little things that are not always gloriously announce that keep making python what it is step by step.
lima 10y ago
Useful feature. Another important async improvement in 3.6 are asynchronous iterators: https://www.python.org/dev/peps/pep-0525/ https://www.python.org/dev/peps/pep-0525/ (yielding from a coroutine)
jfaucett 10y ago
This is somewhat tangential but could some of you pythonistas opine on the status of concurrency in python as of 3.6? Are we talking JVM level goodness or even some message passing builtins like erlang or is python still meandering around problems other scripting languages like ruby have had in this area? Basically, how easy is it to build concurrent code that actually does improve with added processors assuming you structure your stuff correctly.
- dom0 10y ago
  ~depends Python is still fundamentally a n=1 language, where only one thread can run the interpreter at any time (GIL), like many scripting languages. Thus any gains in performance from using multiple threads has to stem from using external resources (syscalls, calls into C code) -- running pure Python code in multiple threads won't increase performance (but still provide concurrency). Most libraries that do heavy lifting in native code and so on are written with this in mind, and allow thus ample performance gains, should these operations actually limit performance. Of course, nothing keeps one from using entirely different means to concurrency, eg. message passing, like the ZeroMQ approach, RPC, shared memory and so on are all possible with Python. So I'd say it's quite manageable, and that the GIL isn't really a significant limitation for most applications. However, it also means that in many cases process-level parallelism is preferable, eg. in the context of web applications this is the usual approach (and it doesn't have to cost a ton of memory - see uwsgi/pre-forking appservers), besides async.
  - paulddraper 10y ago
    Everything true, though I would add that Python makes multiprocess parallelism fairly painless with the standard multiprocessing package.
jayajay 10y ago
This is str8 dope... when you are accustomed to seeing node.js callbacks pre-es6.
- jayajay 10y ago
  Trigger Warning, do not read if easily offended or if against introspection. Downvotes? Would this have been upvoted?: "Python 3.6 is really cool and looks fairly compact, especially when you're used to nesting callbacks, using promises, or fibers in Node.js. That said, arrow functions provide some of that compactness in ES6." It's interesting how a certain verbal style resonates better or worse in certain communities, even if the content is held approximately fixed: "Python 3.6 is really cool because of comprehensions." "Python 3.6 is lit af bruh cuz dem list comps." "dis lang is dope cuz ONE liners tho." "python is fucking [i for its in me]" For example, if I tried to teach a bunch of inner city kids about Python sounding like the average HackerNews user, they would never learn anything! For those of you that teach kids, you'll probably understand, everyone else will probably take offense to this, don't say I didn't warn you. Communities like this (HackerNews) lack diversity -- we're all the same flavor of nerd, and that's boring as fuck, isn't it? We tend to agree with each other, and are diametrically opposed to people who try to burst our bubble. Clearly we're right about everything because we have been validated by our fancy BS's in honors physics, mathematics, chemistry, biology, and other sciences (oh, and engineering, if you're second-rate (shh! some people don't like mean jokes)). There was a post on here about democracy after the Trump victory (or right before?). And the majority consensus of the nerds on this site was that government should not have unlimited power, and it should make itself easy to change, or replace, if needed. I.e., there should be no runaway governments, which we're stuck with. People should have the power to change things. For the most part, you guys agree with that, which I find incredibly ironic.
  - sametmax 10y ago
    I +1 for the overall insight, but it's logicial and desirable to have small communities of people alike. As long as you have many of them, that are all very different and their members are part of several of them. It know it's the case for me (I'm part active on HN, reddit, imgur, twitter but IRL in sport, charity and porn), and I believe a lot of people are too here. So I think we are OK. HN is indeed a bubble, but it's a fantastic one that would lose a lot of value if the liquid it came from was diluted.
PudgePacket 10y ago
Offtopic but the python website is looking nice, last time I visited it definitely wasn't so shmick.
- noobermin 10y ago
  Hmm, I remember the design from a year or so ago if I'm not mistaken and it seemed the same.
Animats 10y ago
When is this useful? There's no real multiprocessing concurrency in Python. So using this on a computational example such as the one given: result = [i async for i in aiter() if i % 2] won't speed anything up or put multiple CPUs to work on it. Python's new "async" features are useful mostly for servers running many network connections simultaneously. Too many for one thread per connection. That use case used to be handled by "Twisted Python", but now there's language support. That's useful. But a list comprehension doesn't make sense for that use case. The PEP lacks any useful example. The motivation seems to be to make async comprehensions have the same capabilities as regular comprehensions, even though that's not particularly useful.
- yladiz 10y ago
  I could see it being useful for gathering data via an API and returning it in an array, like if an API doesn't return an array of JSON objects but just an object, this could be useful for that use case. In general as long as it's not processor bound like computational tasks but may not be synchronous this could be useful. It also naturally extends the async for syntax to list comprehension. It would be really nice if somehow these kinds of tasks could be tied into the multiprocessing package, so that async could handle processor intensive applications well.
  - rjbwork 10y ago
    This. This is essentially what you can do with the AsParallel() LINQ method, or iterating over an IEnumerable of tasks generated by async methods does in C#.
  - spangry 10y ago
    What about 'as_completed' in concurrent.futures? From my understanding, they fixed the performance issues with ProcessPoolExecutor in 3.4.
- m_mueller 10y ago
  I recently tried to parallelize some non trivial code I have with multiprocessing.map/imap/imap_unordered - the state of parallel computing in python is really poor have to say. If the program's CPU runtime has gotten so long that you'd think about parallelizing, its datastructures tend to be too much to push through the pickle bottleneck and no gain would be had. Except of course number crunching stuff for which there's numpy. Bottom line is, you can do parallel computing, but trying to parallelize after the fact will usually fail, you have to plan for it from the very beginning due to no shared memory, which would add significant time until your first version is up and running I long for a dynamic but strictly typed language with rich standard lib that gets this right. Nim?
dorianm 10y ago
I find the Ruby solution more elegant: https://github.com/bruceadams/pmap https://github.com/bruceadams/pmap e.g.: Replace: require 'open-uri' (1..100).map { |i| open("http://httpbin.org/get?a=#{i}").read } To: require 'open-uri' require 'pmap' (1..100).pmap { |i| open("http://httpbin.org/get?a=#{i}").read } From 17.55s to 0.72s.
- noobermin 10y ago
  As a non-rubyist, this took me a minute to guess through. I prefer list comps. hint: I'm implying familiarity over elegance.
  - dorianm 10y ago
    I guess in Python it would be going from: list(map(lambda x: x**2, [1, 2, 3, 4])) To: list(pmap(lambda x: x**2, [1, 2, 3, 4]))
    - noobermin 10y ago
      Right, and neither of those are pythonic. There is no need for a lambda when you have comprehensions. This comment may relate better to your point: I would personally prefer "afor" over "async for", FWIW. One thing I don't feel "async for" is being more explicit over being more verbose. The symmetry with "await foo" is broken though.
- deleted 10y ago
  [deleted]
- lars512 10y ago
  Parallel map is built in to Python already with pools. import requests import multiprocessing as mp pool = mp.Pool() urls = ["http://httpbin.org/get?a=#{}".format(i) for i in range(100)] pool.map(requests.get, urls)
tejasmanohar 10y ago
I can see a use case for `await` in comprehensions- e.g. responses = [await r() for r in requests] ... but, what's the 'practical' application of `async` in comprehensions? I can't think of one that shouldn't call an independent `async` function for readability.

125 points by tbarbugli 10y ago

1st1 10y ago

Surprised to see this here :) I'm the author of the PEP, feel free to ask questions.

sagnew 10y ago
Just wanted to say excellent work. This is some convenient wizardy!
- 1st1 10y ago
  Thanks ;)
orf 10y ago
Thank you so much for the pep, I've just finished writing a pretty neat tool using all the new 3.6 features and they are awesome, especially the comprehensions. Thanks :)
- 1st1 10y ago
  Thanks! Did you use new async generators in your tool?
jchassoul 10y ago
I just want to say thank you for make this possible! this are the progressive little things that are not always gloriously announce that keep making python what it is step by step.

lima 10y ago

Useful feature. Another important async improvement in 3.6 are asynchronous iterators: https://www.python.org/dev/peps/pep-0525/ https://www.python.org/dev/peps/pep-0525/ (yielding from a coroutine)

jfaucett 10y ago

This is somewhat tangential but could some of you pythonistas opine on the status of concurrency in python as of 3.6? Are we talking JVM level goodness or even some message passing builtins like erlang or is python still meandering around problems other scripting languages like ruby have had in this area? Basically, how easy is it to build concurrent code that actually does improve with added processors assuming you structure your stuff correctly.

dom0 10y ago
~depends Python is still fundamentally a n=1 language, where only one thread can run the interpreter at any time (GIL), like many scripting languages. Thus any gains in performance from using multiple threads has to stem from using external resources (syscalls, calls into C code) -- running pure Python code in multiple threads won't increase performance (but still provide concurrency). Most libraries that do heavy lifting in native code and so on are written with this in mind, and allow thus ample performance gains, should these operations actually limit performance. Of course, nothing keeps one from using entirely different means to concurrency, eg. message passing, like the ZeroMQ approach, RPC, shared memory and so on are all possible with Python. So I'd say it's quite manageable, and that the GIL isn't really a significant limitation for most applications. However, it also means that in many cases process-level parallelism is preferable, eg. in the context of web applications this is the usual approach (and it doesn't have to cost a ton of memory - see uwsgi/pre-forking appservers), besides async.
- paulddraper 10y ago
  Everything true, though I would add that Python makes multiprocess parallelism fairly painless with the standard multiprocessing package.

jayajay 10y ago

This is str8 dope... when you are accustomed to seeing node.js callbacks pre-es6.

jayajay 10y ago
Trigger Warning, do not read if easily offended or if against introspection. Downvotes? Would this have been upvoted?: "Python 3.6 is really cool and looks fairly compact, especially when you're used to nesting callbacks, using promises, or fibers in Node.js. That said, arrow functions provide some of that compactness in ES6." It's interesting how a certain verbal style resonates better or worse in certain communities, even if the content is held approximately fixed: "Python 3.6 is really cool because of comprehensions." "Python 3.6 is lit af bruh cuz dem list comps." "dis lang is dope cuz ONE liners tho." "python is fucking [i for its in me]" For example, if I tried to teach a bunch of inner city kids about Python sounding like the average HackerNews user, they would never learn anything! For those of you that teach kids, you'll probably understand, everyone else will probably take offense to this, don't say I didn't warn you. Communities like this (HackerNews) lack diversity -- we're all the same flavor of nerd, and that's boring as fuck, isn't it? We tend to agree with each other, and are diametrically opposed to people who try to burst our bubble. Clearly we're right about everything because we have been validated by our fancy BS's in honors physics, mathematics, chemistry, biology, and other sciences (oh, and engineering, if you're second-rate (shh! some people don't like mean jokes)). There was a post on here about democracy after the Trump victory (or right before?). And the majority consensus of the nerds on this site was that government should not have unlimited power, and it should make itself easy to change, or replace, if needed. I.e., there should be no runaway governments, which we're stuck with. People should have the power to change things. For the most part, you guys agree with that, which I find incredibly ironic.
- sametmax 10y ago
  I +1 for the overall insight, but it's logicial and desirable to have small communities of people alike. As long as you have many of them, that are all very different and their members are part of several of them. It know it's the case for me (I'm part active on HN, reddit, imgur, twitter but IRL in sport, charity and porn), and I believe a lot of people are too here. So I think we are OK. HN is indeed a bubble, but it's a fantastic one that would lose a lot of value if the liquid it came from was diluted.

PudgePacket 10y ago

Offtopic but the python website is looking nice, last time I visited it definitely wasn't so shmick.

noobermin 10y ago
Hmm, I remember the design from a year or so ago if I'm not mistaken and it seemed the same.

Animats 10y ago

When is this useful? There's no real multiprocessing concurrency in Python. So using this on a computational example such as the one given: result = [i async for i in aiter() if i % 2] won't speed anything up or put multiple CPUs to work on it. Python's new "async" features are useful mostly for servers running many network connections simultaneously. Too many for one thread per connection. That use case used to be handled by "Twisted Python", but now there's language support. That's useful. But a list comprehension doesn't make sense for that use case. The PEP lacks any useful example. The motivation seems to be to make async comprehensions have the same capabilities as regular comprehensions, even though that's not particularly useful.

yladiz 10y ago
I could see it being useful for gathering data via an API and returning it in an array, like if an API doesn't return an array of JSON objects but just an object, this could be useful for that use case. In general as long as it's not processor bound like computational tasks but may not be synchronous this could be useful. It also naturally extends the async for syntax to list comprehension. It would be really nice if somehow these kinds of tasks could be tied into the multiprocessing package, so that async could handle processor intensive applications well.
- rjbwork 10y ago
  This. This is essentially what you can do with the AsParallel() LINQ method, or iterating over an IEnumerable of tasks generated by async methods does in C#.
- spangry 10y ago
  What about 'as_completed' in concurrent.futures? From my understanding, they fixed the performance issues with ProcessPoolExecutor in 3.4.
m_mueller 10y ago
I recently tried to parallelize some non trivial code I have with multiprocessing.map/imap/imap_unordered - the state of parallel computing in python is really poor have to say. If the program's CPU runtime has gotten so long that you'd think about parallelizing, its datastructures tend to be too much to push through the pickle bottleneck and no gain would be had. Except of course number crunching stuff for which there's numpy. Bottom line is, you can do parallel computing, but trying to parallelize after the fact will usually fail, you have to plan for it from the very beginning due to no shared memory, which would add significant time until your first version is up and running I long for a dynamic but strictly typed language with rich standard lib that gets this right. Nim?

dorianm 10y ago

I find the Ruby solution more elegant: https://github.com/bruceadams/pmap https://github.com/bruceadams/pmap e.g.: Replace: require 'open-uri' (1..100).map { |i| open("http://httpbin.org/get?a=#{i}").read } To: require 'open-uri' require 'pmap' (1..100).pmap { |i| open("http://httpbin.org/get?a=#{i}").read } From 17.55s to 0.72s.

noobermin 10y ago
As a non-rubyist, this took me a minute to guess through. I prefer list comps. hint: I'm implying familiarity over elegance.
- dorianm 10y ago
  I guess in Python it would be going from: list(map(lambda x: x**2, [1, 2, 3, 4])) To: list(pmap(lambda x: x**2, [1, 2, 3, 4]))
  - noobermin 10y ago
    Right, and neither of those are pythonic. There is no need for a lambda when you have comprehensions. This comment may relate better to your point: I would personally prefer "afor" over "async for", FWIW. One thing I don't feel "async for" is being more explicit over being more verbose. The symmetry with "await foo" is broken though.
deleted 10y ago
[deleted]
lars512 10y ago
Parallel map is built in to Python already with pools. import requests import multiprocessing as mp pool = mp.Pool() urls = ["http://httpbin.org/get?a=#{}".format(i) for i in range(100)] pool.map(requests.get, urls)

tejasmanohar 10y ago

I can see a use case for `await` in comprehensions- e.g. responses = [await r() for r in requests] ... but, what's the 'practical' application of `async` in comprehensions? I can't think of one that shouldn't call an independent `async` function for readability.