4 ms·

Would be nice if we can just get out of this local minima of Python 3.30, and start working on Python 4 that's somewhat backwards compatible and has no GIL and

Would be nice if we can just get out of this local minima of Python 3.30, and start working on Python 4 that's somewhat backwards compatible and has no GIL and has meaningful threading support.

quietbritishjim 5y ago
To rehash my typical HN reply about GIL: I find that 99% of the time that you need to use threads for CPU work (not just IO), that means that you care enough about performance that your Python just calls into some C library (C extension libraries e.g. numpy, scipy, or even bits of the standard library e.g. zipfile operating on an in-memory buffer). The thing is, all these calls (almost) all release the GIL anyway. So threading support in Python is already meaningful, unless you want to run pure-Python CPU intensive code on multiple threads, which I'll admit does happen but in my experience is rare.
- ynik 5y ago
  That only works if you can hide your inner loop in the extension module, and can get rid of all accesses to the Python object model inside the inner loop. This works great with numpy if you can work with sufficiently-large arrays so that the GIL can be released for a nontrivial amount of time. But if your algorithm doesn't have this structure, "call into C and release the GIL" just means the inner loop is constantly releasing+reacquiring the GIL, and multi-threaded execution won't be any faster than single-threaded execution due to the fights over GIL. Unfortunately we're mostly dealing with complex stuff happening during graph traversals. There is no "inner loop" for us -- just the "outer loop" of the graph traversal. So the only way to get our code to run in parallel is to port the whole algorithm to C. But that's 150k+ lines of codes, there will be barely any Python remaining :( IMHO Python is a trap, it's easy to get started with but you can stuck in a position you can't get out of without a full port to a different language. (and yes, we've already tried multi-processing and using shared-memory segments in the C code portions to avoid re-doing work in every process, ... but after more than a man-year of work, >80% of our application remains single-threaded thanks to the GIL, even though in any other language it would be trivial to parallelize the independent graph traversals) It's neat to have a scripting language for fast prototyping but we would be much much better off if we had picked Javascript instead of Python (the ability to have multiple interpreters in the same process running in parallel would solve our issues).
  - quietbritishjim 5y ago
    Absolutely agreed, especially the part about needing to hide the inner loop in extension module. If you're using numpy to mutiply 2 x 2 matrices together then the fact it releases the GIL is not that much help. As I said, my experience is that this is not normally a problem in practice. Clearly that's not your experience! I'm not sure whether I've been unusually lucky or you've been unusually unlucky.

Would be nice if we can just get out of this local minima of Python 3.30, and start working on Python 4 that's somewhat backwards compatible and has no GIL and has meaningful threading support.

quietbritishjim 5y ago

To rehash my typical HN reply about GIL: I find that 99% of the time that you need to use threads for CPU work (not just IO), that means that you care enough about performance that your Python just calls into some C library (C extension libraries e.g. numpy, scipy, or even bits of the standard library e.g. zipfile operating on an in-memory buffer). The thing is, all these calls (almost) all release the GIL anyway. So threading support in Python is already meaningful, unless you want to run pure-Python CPU intensive code on multiple threads, which I'll admit does happen but in my experience is rare.

ynik 5y ago
That only works if you can hide your inner loop in the extension module, and can get rid of all accesses to the Python object model inside the inner loop. This works great with numpy if you can work with sufficiently-large arrays so that the GIL can be released for a nontrivial amount of time. But if your algorithm doesn't have this structure, "call into C and release the GIL" just means the inner loop is constantly releasing+reacquiring the GIL, and multi-threaded execution won't be any faster than single-threaded execution due to the fights over GIL. Unfortunately we're mostly dealing with complex stuff happening during graph traversals. There is no "inner loop" for us -- just the "outer loop" of the graph traversal. So the only way to get our code to run in parallel is to port the whole algorithm to C. But that's 150k+ lines of codes, there will be barely any Python remaining :( IMHO Python is a trap, it's easy to get started with but you can stuck in a position you can't get out of without a full port to a different language. (and yes, we've already tried multi-processing and using shared-memory segments in the C code portions to avoid re-doing work in every process, ... but after more than a man-year of work, >80% of our application remains single-threaded thanks to the GIL, even though in any other language it would be trivial to parallelize the independent graph traversals) It's neat to have a scripting language for fast prototyping but we would be much much better off if we had picked Javascript instead of Python (the ability to have multiple interpreters in the same process running in parallel would solve our issues).
- quietbritishjim 5y ago
  Absolutely agreed, especially the part about needing to hide the inner loop in extension module. If you're using numpy to mutiply 2 x 2 matrices together then the fact it releases the GIL is not that much help. As I said, my experience is that this is not normally a problem in practice. Clearly that's not your experience! I'm not sure whether I've been unusually lucky or you've been unusually unlucky.