4 ms·
Just in case someone is interested, I've written a simple fork-based module for processing jobs for Python: https://github.com/jeremysanders/forkqueue https://g
by xioxox 4y ago
Just in case someone is interested, I've written a simple fork-based module for processing jobs for Python: https://github.com/jeremysanders/forkqueue https://github.com/jeremysanders/forkqueue
This has the advantage of allowing work done inside a nested function, allowing large initial datasets to be shared and not have to be passed over pickle.
- itamarst 4y agoSee the article for an explanation of why this is a bad idea. Note that on macOS Python disabled fork()-based multiprocessing as the default long ago (Python 3.8?) because it's so broken, and it will stop being the default for multiprocessing on Linux in 3.14, deprecated in 3.12.
- xioxox 4y agoIt works really well if you avoid threads, and I've found it pretty easy to avoid them in my code.
- itamarst 4y agoYes, but... it's harder than one might think to avoid threads. Merely importing NumPy starts threads, for example.
- mickeyp 4y agoYes, and that's a good thing. Let numpy do its threading unless you know exactly why you don't want it to parallelise your matrix multiplications and what havbe you. Numpy releases the GIL so that works exactly as it should. Threads aren't bad: threads that cause resource contention with the GIL is (possibly) bad. That is almost always done explicitly by the developer and almost never done without your knowledge.
- itamarst 4y agoUnfortunately NumPy's thread pool is only for BLAS, the underlying library for numpy.linalg functions, mostly. Other operations are single-threaded. So you need your own thread pool (or process pool) if you want to parallelize anything else.
- mickeyp 4y agoYes, I know. But your statement makes it sound like numpy using threads is somehow bad or undesired (it can be; but if it is, you'll know how to tell numpy not to thread.)