4 ms·
I recall digging into Python subinterpreters a decade ago. Abandoned it because the real killer wasn’t shared GIL, it was shared modules. If one subinterpreter
by hhas01 6y ago
I recall digging into Python subinterpreters a decade ago. Abandoned it because the real killer wasn’t shared GIL, it was shared modules. If one subinterpreter imports a module and modifies that module’s state, then every other subinterpreter that uses that module is impacted too.
Article says nothing about that, which makes me very cautious. The GIL only impacts performance; module sharing wrecks robustness.
Even if each subinterpreter does now keep its own module cache, there’s still the challenge of working safely with common C-level resources such as file handles. (Other than telling users “don’t do that”—and GLWT.)
I’ve used Python now for 17 years and it’s been a very productive tool for me. But it definitely has its baked-in limitations and fighting those is an exercise in rapidly diminishing returns.
Something as fundamental as parallelism can’t just be slopped on top of a language as an afterthought; it needs to be designed for it from the start. So rather than trying to retrofit bad parallelism to Python, perhaps it’d be sense to bring the positive parts of Python over to something that already does parallelism right, such as Erlang, and build from there.