4 ms·

Serious question: is the whole deal with the Python GIL solvable if some BigCo decides to throw a ton of money and engineers at it? Like Google with V8, for ins

by jogjayr 10y ago

Serious question: is the whole deal with the Python GIL solvable if some BigCo decides to throw a ton of money and engineers at it? Like Google with V8, for instance. Or is it a truly hard problem that will take something special to solve?

stcredzero 10y ago
For awhile, Python was one of the 4 approved languages at Google. The answer is "probably no." If it were easily solvable, Google would already have thrown money and engineers at it.
- paulmd 10y ago
  See my sibling comment to yours - but care to explain your thoughts more? Jython already fixed the GIL problem. The problem is the legacy codebase built on assumptions of non-concurrency, which is just a matter of engineer time i.e. throwing money at it, plus getting GVR to sign off on it.
  - stcredzero 10y ago
    Jython already fixed the GIL problem. The problem is the legacy codebase built on assumptions of non-concurrency Which is to say that Jython didn't completely fix the GIL problem, from the POV of a lot of people.
paulmd 10y ago
Sure. There's already variants like Jython that have removed the GIL. The larger problem is the existing codebase, which is based around the assumption of non-concurrency. You also have a lot of libraries that use C extensions for performance, and a lot of those are going to break horribly if you suddenly throw them into a concurrent environment where the Python state is mutating underneath them. Again though, "rewriting code for concurrency" is not a fundamentally unsolvable problem, it just takes a lot of engineer time to change everything over. Moving global/static state into instances, adding locks, marking atomic/critical segments, that kind of thing. There's just a lot of things built with Python that would need to be gone over. I really think you could do a switchover on the fly by adding a Java-style "synchronized" attribute. Before you can enter a synchronized method, you set a flag and all other threads must yield at their next return, function call, loop iteration, the end of their atomic segment, or at safe points marked by a "yield" statement (pick some combination of reasonable behavior). While a synchronized method is on the call stack, no other thread may execute. All existing code is marked synchronized - perhaps any code in a .py file is assumed unsafe by default, while any code in a .jy file is assumed safe. Boom, start converting code. The thing that really gets me is that Python 3 is already pushing breaking changes that necessitate a complete overhaul anyway. The failure to thread the interpreter/remove the GIL at the same time is a stunningly idiotic decision. How about since we are making everyone review their code anyway, we have them look at thread safety too? Which leads to the other problem - Python is GVR's baby, and at the end of the day the reason Python 3 has a GIL is because he says so. By all means, Google could go ahead and rewrite everything, but he'd never let them call it Python. It's hard to build momentum for a serious fork like that. And you don't want to spend a bunch of engineer time and end up with an unsupported "toy" that nobody uses. It's a shame, Python hits real close to the mark but concurrency is its Achilles' Heel. I love the language but I am gunshy about using it because you never know if your project will go from "toy" to "real product that may need to adapt/scale" and you need concurrency.
takeda 10y ago
The biggest challenge of removing GIL is to do it without breaking existing code. There's work on that. Here's a technical talk about why is it difficult[1]. [1] https://www.youtube.com/watch?v=fgWUwQVoLHo https://www.youtube.com/watch?v=fgWUwQVoLHo

by jogjayr 10y ago

stcredzero 10y ago

For awhile, Python was one of the 4 approved languages at Google. The answer is "probably no." If it were easily solvable, Google would already have thrown money and engineers at it.

paulmd 10y ago
See my sibling comment to yours - but care to explain your thoughts more? Jython already fixed the GIL problem. The problem is the legacy codebase built on assumptions of non-concurrency, which is just a matter of engineer time i.e. throwing money at it, plus getting GVR to sign off on it.
- stcredzero 10y ago
  Jython already fixed the GIL problem. The problem is the legacy codebase built on assumptions of non-concurrency Which is to say that Jython didn't completely fix the GIL problem, from the POV of a lot of people.

paulmd 10y ago

Sure. There's already variants like Jython that have removed the GIL. The larger problem is the existing codebase, which is based around the assumption of non-concurrency. You also have a lot of libraries that use C extensions for performance, and a lot of those are going to break horribly if you suddenly throw them into a concurrent environment where the Python state is mutating underneath them. Again though, "rewriting code for concurrency" is not a fundamentally unsolvable problem, it just takes a lot of engineer time to change everything over. Moving global/static state into instances, adding locks, marking atomic/critical segments, that kind of thing. There's just a lot of things built with Python that would need to be gone over. I really think you could do a switchover on the fly by adding a Java-style "synchronized" attribute. Before you can enter a synchronized method, you set a flag and all other threads must yield at their next return, function call, loop iteration, the end of their atomic segment, or at safe points marked by a "yield" statement (pick some combination of reasonable behavior). While a synchronized method is on the call stack, no other thread may execute. All existing code is marked synchronized - perhaps any code in a .py file is assumed unsafe by default, while any code in a .jy file is assumed safe. Boom, start converting code. The thing that really gets me is that Python 3 is already pushing breaking changes that necessitate a complete overhaul anyway. The failure to thread the interpreter/remove the GIL at the same time is a stunningly idiotic decision. How about since we are making everyone review their code anyway, we have them look at thread safety too? Which leads to the other problem - Python is GVR's baby, and at the end of the day the reason Python 3 has a GIL is because he says so. By all means, Google could go ahead and rewrite everything, but he'd never let them call it Python. It's hard to build momentum for a serious fork like that. And you don't want to spend a bunch of engineer time and end up with an unsupported "toy" that nobody uses. It's a shame, Python hits real close to the mark but concurrency is its Achilles' Heel. I love the language but I am gunshy about using it because you never know if your project will go from "toy" to "real product that may need to adapt/scale" and you need concurrency.

takeda 10y ago

The biggest challenge of removing GIL is to do it without breaking existing code. There's work on that. Here's a technical talk about why is it difficult[1]. [1] https://www.youtube.com/watch?v=fgWUwQVoLHo https://www.youtube.com/watch?v=fgWUwQVoLHo