3 ms·
Massive overhead of multiprocessing? How have I not noticed this for tens of years? I use coroutines and multiprocessing all the time, and saturate every core
by Demiurge 2y ago
Massive overhead of multiprocessing? How have I not noticed this for tens of years?
I use coroutines and multiprocessing all the time, and saturate every core and all the IO, as needed. I use numpy, pandas, xarray, pytorch, etc.
How did this terrible GIL overhead completely went unnoticed?
- viraptor 2y ago> I use numpy, pandas, xarray, pytorch, etc. That means your code is using python as glue and you do most of your work completely outside of cPython. That's why you don't see the impact - those libraries drop GIL when you use them, so there's much less overhead.
- quietbritishjim 2y agoThe parent commenter said they're using the multiprocessing module, so it's irrelevant to them whether those modules drop the GIL (except for the fact that they are missing an opportunity to using threading instead). The overhead being referred to, whether significant or not, is that of spawning processes and doing IPC.