3 ms·

Actually ran into this problem this week, toyed around with multiprocessing.shared_memory (which seems to also rely on mmaped-files, right?) and decided to just

Actually ran into this problem this week, toyed around with multiprocessing.shared_memory (which seems to also rely on mmaped-files, right?) and decided to just embrace the GIL. Multiprocessing is not needed when all of your handful subprocesses are just calling Numpy-code and release their gil anyways. Also some/most Numpy functions are multithreaded (depending on the BLAS implementation, linked against), take advantage of that and schedule huge operations and just let the interpreter sit idle waiting for that result.

blitzar 3y ago
I have seen this movie before, it always ends the same way. Look to go parallel for speed increase, burn a bunch of time and decide its way too hard and just wait the extra few minutes instead.
- fabioz 3y ago
  Actually, you get the speed increase if using threads on python with numpy because numpy will release the GIL internally, so, operations using numpy will actually run in parallel (i.e.: c libraries can release the GIL and do get the performance benefits when using multiple processors -- although yes, when you get back to running python code that won't be parallel due to the GIL).
  - blitzar 3y ago
    My genuine deep love for numpy extends to them doing these things without me knowing anything about it.
  - KeplerBoy 3y ago
    That's something I worry about when reading about future Python releases without GIL. Will people just throw more and more threads at unfathomably slow Python code to get somewhat fast results?
- KeplerBoy 3y ago
  It's actually the opposite. My current job is making a piece of code faster my predecessor tried to make faster using multiprocessing. I now try to pull stuff back into a single process. The code for example loads a file from disk in a separate process to have the main process available for number crunching. The problem: This data is then sent to the main process and another copy is created which takes time. Doing the same with threads is way faster, since data i/o operations are releasing the gil anyways.

blitzar 3y ago

I have seen this movie before, it always ends the same way. Look to go parallel for speed increase, burn a bunch of time and decide its way too hard and just wait the extra few minutes instead.

fabioz 3y ago
Actually, you get the speed increase if using threads on python with numpy because numpy will release the GIL internally, so, operations using numpy will actually run in parallel (i.e.: c libraries can release the GIL and do get the performance benefits when using multiple processors -- although yes, when you get back to running python code that won't be parallel due to the GIL).
- blitzar 3y ago
  My genuine deep love for numpy extends to them doing these things without me knowing anything about it.
- KeplerBoy 3y ago
  That's something I worry about when reading about future Python releases without GIL. Will people just throw more and more threads at unfathomably slow Python code to get somewhat fast results?
KeplerBoy 3y ago
It's actually the opposite. My current job is making a piece of code faster my predecessor tried to make faster using multiprocessing. I now try to pull stuff back into a single process. The code for example loads a file from disk in a separate process to have the main process available for number crunching. The problem: This data is then sent to the main process and another copy is created which takes time. Doing the same with threads is way faster, since data i/o operations are releasing the gil anyways.