4 ms·
It works for I/O bound tasks. For CPU intensive tasks, you have almost no improvement. I use it a lot for web scraping. You can from multiprocessing.pool impor
by iurisilvio 13y ago
It works for I/O bound tasks. For CPU intensive tasks, you have almost no improvement. I use it a lot for web scraping.
You can from multiprocessing.pool import ThreadPool. It is not well documented too, but at least the package makes sense.
- rbanffy 13y agoIt depends on your workload - the GIL is still there after all. If you use the multiple processes you may get away with that. The article mentions this case, but uses PIL, so, the CPU-bound part is running compiled C code with very little GIL interaction.
- goostavos 13y agoI'm not sure I entirely agree with your CPU intensive statement. Multiprocessing definitely brings Python speedups in the CPU bound arena. I've used it for everything from template matching, to building movies from still frames. Heck, my primary solution to speeding things up is throwing more cores at it rather than figuring out why my algorithm sucks ;) A couple of quick benchmarks and you'll be a believer, too!