5 ms·

So what this does is introduce a (configurable) buffer between chained async generators. So this does not introduce any kind of multi processor parallelism, jus

by tda 6y ago

So what this does is introduce a (configurable) buffer between chained async generators. So this does not introduce any kind of multi processor parallelism, just makes an async generator easily be able to wait on multiple iterations in parallel. So basically the need for queues is abstracted away, looks quite nice if your use case requires something like this

jacobwilliamroy 6y ago
Yeah the title is an oxymoron. "Parallelized async"
- michalc 6y ago
  Hi, author here, I was slightly torn on the usage of the word "parallelise", in that yes, CPU-wise, no tasks will progress in parallel. However, the point of the library is to allow certain parts of a pipeline to progress at the same time in the real-world. In the example, they are calls to asyncio.sleep, but in real cases, they could be HTTP requests. As in, the bytes of the request/response will be going across the wire at the same time, so the total wall-clock time of (certain) pipelines will go down. So, I claim "parallelise" _is_ appropriate.
  - theelous3 6y ago
    It's just a structural thing though? Like you have a pipeline of [coro1, coro2] where coro2 is using data as it's yielded out of coro1. But you can also just have like, coro2 yield from coro1 or similar. I think I like it. Not sure yet. It might help composibility if you find yourself slapping together a lot of discreet little coros.
  - jacobwilliamroy 6y ago
    I am familiar with all of the cheats, delaying evaluation, kicking the can onto the network and pretending that everything isn't just being fed into a mutex on the other side. And for writing to disk, async is basically the only way to "parallelize" such operations I just think its inappropriate because, if I was looking for a parallelization library like multiprocessing, and I found this, I might waste 15 minutes or so before I figure out that this isn't parallel, it's just async. Async is okay, but it does nothing to improve performance, like what parallelization does. Async provides good UX, parallelization provides good performance. Conflating the two just muddys the water for the people who like to do high-performance-computing.
freeqaz 6y ago
If you do want that massive concurrency you can use PyWren! https://github.com/pywren/pywren https://github.com/pywren/pywren

by tda 6y ago

jacobwilliamroy 6y ago

Yeah the title is an oxymoron. "Parallelized async"

michalc 6y ago
Hi, author here, I was slightly torn on the usage of the word "parallelise", in that yes, CPU-wise, no tasks will progress in parallel. However, the point of the library is to allow certain parts of a pipeline to progress at the same time in the real-world. In the example, they are calls to asyncio.sleep, but in real cases, they could be HTTP requests. As in, the bytes of the request/response will be going across the wire at the same time, so the total wall-clock time of (certain) pipelines will go down. So, I claim "parallelise" _is_ appropriate.
- theelous3 6y ago
  It's just a structural thing though? Like you have a pipeline of [coro1, coro2] where coro2 is using data as it's yielded out of coro1. But you can also just have like, coro2 yield from coro1 or similar. I think I like it. Not sure yet. It might help composibility if you find yourself slapping together a lot of discreet little coros.
- jacobwilliamroy 6y ago
  I am familiar with all of the cheats, delaying evaluation, kicking the can onto the network and pretending that everything isn't just being fed into a mutex on the other side. And for writing to disk, async is basically the only way to "parallelize" such operations I just think its inappropriate because, if I was looking for a parallelization library like multiprocessing, and I found this, I might waste 15 minutes or so before I figure out that this isn't parallel, it's just async. Async is okay, but it does nothing to improve performance, like what parallelization does. Async provides good UX, parallelization provides good performance. Conflating the two just muddys the water for the people who like to do high-performance-computing.

freeqaz 6y ago

If you do want that massive concurrency you can use PyWren! https://github.com/pywren/pywren https://github.com/pywren/pywren