5 ms·
It's relatively simple to make the GIL go away: just compile to some VM that has a good concurrent garbage collector would be one approach. Yes, this will break
by amelius 3y ago
It's relatively simple to make the GIL go away: just compile to some VM that has a good concurrent garbage collector would be one approach. Yes, this will break some assumptions here and there, but not too difficult to overcome especially if you bump the version number to Python 4.
However, that leaves a lot of C code that you can't talk to anymore because the C code requires the old Python FFI. I think this is where the main problem lies.
- notatallshaw 3y ago> It's relatively simple to make the GIL go away: just compile to some VM that has a good concurrent garbage collector would be one approach Sure, if you don't mind paying a 50-90% performance impact on single threaded performance or completely abandon C-API compatibility and have C extensions start from scratch then there are simple approaches. If you look at any example in the past to remove the GIL you would see that keeping these two requirements of not having terrible single threaded performance and not having almost a completely new C-API is actually very complex and takes a lot of expertise to implement.
- saltminer 3y agoThis might be a dumb question, but why would removing the GIL break FFI? Is it just that existing no-GIL implementations/proposals have discarded/ignored it, or is there a fundamental requirement, e.g. C programs unavoidably interact directly with the GIL? (In which case, couldn't a "legacy FFI" wrapper be created?) I know that the C-API is only stable between minor releases [0] compiled in the same manner [1], so it's not like the ecosystem is dependent upon it never changing. I cannot seem to find much discussion about this. I have found a no-GIL interpreter that works with numpy, scikit, etc. [2][3] so it doesn't seem to be a hard limit. (That said, it was not stated if that particular no-GIL implementation requires specially built versions of C-API libs or if it's a drop-in replacement.) [0]: https://docs.python.org/3/c-api/stable.html#c-api-stability https://docs.python.org/3/c-api/stable.html#c-api-stability [1]: https://docs.python.org/3/c-api/stable.html#platform-considerations https://docs.python.org/3/c-api/stable.html#platform-conside... [2]: https://github.com/colesbury/nogil https://github.com/colesbury/nogil [3]: https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional-3-12-updates/26503/28 https://discuss.python.org/t/pep-703-making-the-global-inter...
- kortex 3y ago> C programs unavoidably interact directly with the GIL? Bingo. They don't have to, but often the point of C extensions is performance, which usually means turning on parallelism. E.g. Numpy will release the GIL in order to use machine threads on compute-heavy tasks. I'm not worried about the big 5 (numpy, scipy, pandas, pytorch, and sklearn), they have enough support that they can react to a GILectomy. It's everyone else that touches the GIL but may not have the capacity or ability to update in a timely manner. I don't think this is something which can be shimmed either or ABI-versioned either. It's deeeep and touches huge swaths of the cpython codebase.
- saltminer 3y agoThanks, that explains a lot. Sounds like a task that would have to be done in Python 4, if ever it exists.
- notatallshaw 3y ago> or is there a fundamental requirement, e.g. C programs unavoidably interact directly with the GIL? Both C programs can use the GIL for thread safety and can make assumptions about the safety of interacting with a Python object. Some of those assumptions are not real guarantees from the GIL but in practise are good enough, they would no longer be good enough in a no-GIL world. > I know that the C-API is only stable between minor releases [0] compiled in the same manner [1], so it's not like the ecosystem is dependent upon it never changing. There is a limited API tagged as abi3[1] which is unchanging and doesn't require recompiling and any attempt to remove the GIL so far would break that. > so it's not like the ecosystem is dependent upon it never changing But the wider C-API does not change much between major versions, it's not like the way you interact with the garbage collector completely changes causing you to rethink how you have to write concurrency. This allows the many projects which use Python's C-API to relatively quickly update to new major versions of Python. > I have found a no-GIL interpreter that works with numpy, scikit, etc. [2][3] so it doesn't seem to be a hard limit. The version of nogil Python you are linking is the product of years of work by an expert funded to work full time on this by Meta, the knowledge is sourcing many previous attempts to remove the GIL including the "gilectomy". Also you are linking to the old version based on Python 3.9, there is a new version based on Python 3.12[2] This strays away from the points I was making, but with this specific attempt to remove the GIL if it is adopted it is unlikely to be switched over in a "big bang", e.g. Python 3.13 followed by Python 4.0 with no backwards compatibility on C extensions. The Python community does not want to repeat the mistakes of the Python 2 to 3 transition. So far more likely is to try and find a way to have a bridge version that supports both styles of extensions. There is a lot of complexity in this though, including how to mark these in packaging, how to resolve dependencies between packages which do or do not support nogil, etc. And even this attempt to remove the GIL is likely to make things slower in some applications, both in terms of real-world performance as some benchmarks such as MyPy show a nearly 50% slowdown and there may be even worse edge cases not discovered yet, and in terms of lost development as the Faster CPython project will unlikely be able to land a JIT in 3.13 or 3.14 as they plan right now. [1]: https://docs.python.org/3/c-api/stable.html#c.Py_LIMITED_API https://docs.python.org/3/c-api/stable.html#c.Py_LIMITED_API [2]: https://github.com/colesbury/nogil-3.12 https://github.com/colesbury/nogil-3.12
- brightball 3y agoJRuby is a good path to this in the Ruby world.
- crabbone 3y ago[flagged]
- gyrovagueGeist 3y ago...you mean like how the nogil project already has a working Numpy module?
- crabbone 3y agoNo it doesn't. There are couple dozens of projects which link with NumPy. In order to declare NumPy working, you need to support all of them too.
- apgwoz 3y ago> Also, there aren't good programmers in Python core dev. You seem pretty confident that you know what you are doing.
- deleted 3y ago[deleted]
- zerkten 3y ago>> However, that leaves a lot of C code that you can't talk to anymore because the C code requires the old Python FFI. I think this is where the main problem lies. This is exactly the problem, but people have a hard time grasping this because most people interacting with Python have no understanding of how C code interacts with Python, or don't understand the C module ecosystem. I'm not sure if the Python community has a good accounting of this either because I don't recall seeing much quantitative analysis of how many modules would need to be updated etc. This would help compare with the Python 2 to 3 conversion efforts. Even then, the site listing (shaming?) popular modules with compatibility made a mid-to-late appearance in the process of killing Python 2. Quantification of module updates is obvious thing to have from the get-go for anyone looking to follow through on removing the GIL, but it's not a fun task.
- amelius 3y agoThis needs more thinking but how about a hybrid approach, where you have Thread objects, and GILFreeThread objects? The Thread objects work with old code, but run more slowly. The GILFreeThread objects are fast. If an object is passed from a Thread to a GILFreeThread or the other way around, then special safety code is attached to the object so that manipulating the object from the other side doesn't cause issues. The advantage is that now the module implementers have time to migrate from the old system to the new system. And users can work with both the old modules and "converted" modules in the same system, with minor changes.
- ptx 3y agoThis sounds a bit like COM and its apartment-threaded vs. free-threaded objects. The "special safety code" in that case is a proxy object that sends messages to the thread that owns the actual object when its methods are invoked.
- kortex 3y agoThat sounds like a maintenance and stability nightmare, if it's even possible. You are effectively red/blue splitting the entire codebase. PyObject and the GIL touch everything in the codebase.
- coldtea 3y ago>It's relatively simple to make the GIL go away: just compile to some VM that has a good concurrent garbage collector would be one approach. Yes, this will break some assumptions here and there, but not too difficult to overcome especially if you bump the version number to Python 4. "It's easy to lower the air-conditioning costs of Las Vegas: just move the town to New England". The problem is "how to remove the GIL" in abstract. It's how to remove the GIL, not impact extensions at all (or as little as possible), keep single threaded performance, and have zero impact to user programs. To which the above isn't any kind of solution.
- make3 3y agothis would literally break every single python package out there man