3 ms·
Anyone else run into problems with Multiprocessing + Numpy? I can somehow only reliably do multiprocessing with numpy when I set: export OPENBLAS_NUM_THREADS=
by amelius 3y ago
Anyone else run into problems with Multiprocessing + Numpy?
I can somehow only reliably do multiprocessing with numpy when I set:
export OPENBLAS_NUM_THREADS=1
Of course, this is a workaround and not a true solution.
- nerdponx 3y agoThis is probably a known bad interaction and might have to do with spawn vs fork as mentioned elsewhere in this thread.
- amelius 3y agoYes, I can remember that I fiddled with spawn versus fork, but still ended with the OPENBLAS_NUM_THREADS=1 solution.
- czbond 3y agoDask can also help with Numpy sets.
- amelius 3y agoWell, it's complicated as I'm not doing multiprocessing with Numpy, actually :) One part of my program uses Numpy, and the other part does multiprocessing. But Python's multiprocessing module causes interference between them (which took quite some time and effort to figure out).