5 ms·
That only works if you can hide your inner loop in the extension module, and can get rid of all accesses to the Python object model inside the inner loop. This
by ynik 5y ago
That only works if you can hide your inner loop in the extension module, and can get rid of all accesses to the Python object model inside the inner loop.
This works great with numpy if you can work with sufficiently-large arrays so that the GIL can be released for a nontrivial amount of time.
But if your algorithm doesn't have this structure, "call into C and release the GIL" just means the inner loop is constantly releasing+reacquiring the GIL, and multi-threaded execution won't be any faster than single-threaded execution due to the fights over GIL.
Unfortunately we're mostly dealing with complex stuff happening during graph traversals. There is no "inner loop" for us -- just the "outer loop" of the graph traversal. So the only way to get our code to run in parallel is to port the whole algorithm to C. But that's 150k+ lines of codes, there will be barely any Python remaining :(
IMHO Python is a trap, it's easy to get started with but you can stuck in a position you can't get out of without a full port to a different language.
(and yes, we've already tried multi-processing and using shared-memory segments in the C code portions to avoid re-doing work in every process, ... but after more than a man-year of work, >80% of our application remains single-threaded thanks to the GIL, even though in any other language it would be trivial to parallelize the independent graph traversals)
It's neat to have a scripting language for fast prototyping but we would be much much better off if we had picked Javascript instead of Python (the ability to have multiple interpreters in the same process running in parallel would solve our issues).
- quietbritishjim 5y agoAbsolutely agreed, especially the part about needing to hide the inner loop in extension module. If you're using numpy to mutiply 2 x 2 matrices together then the fact it releases the GIL is not that much help. As I said, my experience is that this is not normally a problem in practice. Clearly that's not your experience! I'm not sure whether I've been unusually lucky or you've been unusually unlucky.