3 ms·
I don't think it's possible without breaking changes. But if we're willing to accept some changes to the C API, it's at least possible to make the GIL less glo
by ynik 5y ago
I don't think it's possible without breaking changes.
But if we're willing to accept some changes to the C API, it's at least possible to make the GIL less global.
It really should be obvious that it's possible to have multiple Python interpreters in the same process, where multiple independent threads run Python code in parallel, with occasional calls into a shared data structure implemented as a multi-threaded C extension module.
Currently the GIL makes this impossible: you either can have the multi-threaded C extension module in a single process (but then the Python threads will be limited by Amdahl's law unless you spend >99.9% of CPU time inside the extension module), or you can use multiprocessing which lets the Python code run in parallel, but than the C extension module will have to figure out how to use shared memory.
This is a massive, massive problem for us. We're already invested almost a year of developer time into changing C++ libraries to tell all those pesky std::strings to allocate their data in a shared memory segment. Even then, we've only managed this for some of our shared data structured so far, so this only allows us to parallelize around half of our Python stuff. So in effect, the GIL is causing massive extra complexity for us, and still prevents us from using more than ~2 CPU cores.
We would have been massively better off if 15 years ago we picked Javascript instead of Python. JS is also "single-threaded" like Python, but the JS interpreter lock is much less of a problem because it's not global.
Now Python currently has a "sub-interpreter" effort underway that refactors the interpreter to avoid all that global state and make it per-interpreter. But I somehow doubt that this approach to remove the GIL will be accepted, because again, it will slow down single-threaded execution: e.g. obtaining the `PyObject*` of `None` is currently simply taking the adress of the `_Py_None` global variable. Python will need to split this into one `None` object per interpreter, because (like any Python object) None has a reference count that is mutated all the time. But existing extension modules are accessing the `Py_None` macro -- they don't provide any context which interpreter's `None` they want! This can be solved with some TLS lookups ("what interpreter is the current thread working with"), but I suspect that will be enough of a slowdown that the whole approach will be rejected (since "no performance loss for single-threaded code" seems to be a requirement).
I think Python will get rid of the GIL eventually, but it will take a Python 4.0 with a bunch of breaking changes for C extensions to do so without significant performance loss.