3 ms·
Won’t CPU-heavy tasks still make everything choke because of the GIL? You want to send CPU-heavy tasks to a different process instead of a different thread, don
by JimDabell 5y ago
Won’t CPU-heavy tasks still make everything choke because of the GIL? You want to send CPU-heavy tasks to a different process instead of a different thread, don’t you?
- joconde 5y agoIn my case no, because I heavily use numpy, PyTorch, and other libraries that use C++ extensions and release the GIL during long operations. I easily use 24 CPU cores fully by submitting to one thread pool. If you use pure Python for long tasks, you're right that a ProcessPoolExecutor would be better.
- ris 5y agoIt depends what you mean by "choke". I think the parents consider "choke" to mean "will focus on one long-running task at the expense of any other task that needs to run". The GIL means that multiple threads won't be able to run python code in parallel, but the design should fairly share the available processing time between all threads that want to run.
- joconde 5y agoIn my case, I'm lucky to be using computation libraries that release the GIL.