3 ms·
> All the people that suggest relying on multiprocessing probably haven't done anything that's CPU and Memory intensive because if you have a code that operates
by bildung 7y ago
> All the people that suggest relying on multiprocessing probably haven't done anything that's CPU and Memory intensive because if you have a code that operates on a "world-state" each new process will have to copy that from a parent. If the state takes ~10GB each process will multiply that.
This is wrong, there are multiple ways of python threads working on shared data.
> It's really amazing that the machine learning community has managed to hack around that with C-based libraries like SciPy and NumPy.
Well, the main implementation of the whole language is c-based. I can't see how that implies hackiness.
> If you can't drop GIL because some esoteric features depend on that, then drop them as well.
There have been multiple python implementations without a GIL available for way over 10 years, for example pypy, ironpython and jython. Yet these never went mainstream, which strongly implies the GIL problem actually isn't that much of a problem in the real world.