4 ms·
Using a threadpool with asyncio doesn't buy you any parallelism, you have to use a processpool, which is fine for a data processing pipeline. Asyncio does not l
by hyperion2010 11y ago
Using a threadpool with asyncio doesn't buy you any parallelism, you have to use a processpool, which is fine for a data processing pipeline. Asyncio does not let you bypass the gil but it does vastly reduce the cost of spinning up absurd amounts of sockets and the like.
- superbatfish 11y agoI saw this article, enjoyed it, and then said to myself "Now let me go read the inevitable well-meaning but misguided comments about the GIL." And sure enough, the very top comment is a well-meaning but misguided comment about the GIL. The author of this article is talking about CPU-intensive tasks that involve C-extensions. Such extensions typically release the GIL during their execution, which means that you can effectively parallelize such workloads in a ThreadPool. (He explicitly mentions this about Numpy/Scipy, but most other CPU-intensive extension libraries behave the same way.) Heavy users of Numpy use Thread-level parallelism very frequently.
- hyperion2010 11y agoYep. Thought I was missing something since he was taking about cpu bound tasks and threads, if you are using c extensions then it is wasteful to spin up additional interpreters and you can also get fun bugs like numpy reusing the same random seed in child processes!
- 1st1 11y agoYou can use a ProcessPool executor and have nice async/await interface to it.