4 ms·
It was ground in to CPython at an early date and every Python C extension ends up with it ground in, too, because it's in the C API. Some things are very diffic
by jerf 10y ago
It was ground in to CPython at an early date and every Python C extension ends up with it ground in, too, because it's in the C API. Some things are very difficult to retrofit on to an existing system. [1]
If you create a new Python implementation from scratch it doesn't need to be there. It is not intrinsic to Python; it is intrinsic to the specific CPython implementation.
It is also, broadly speaking, easy to remove the GIL. However the bar the CPython developers have set is that they will not accept a GIL-removal patch that harms single-threaded performance. This is the bar that has proved difficult to hurdle.
It is also fair to point out that while it's a perennial topic in the Python community, it is not the only scripting language implementation with a GIL or moral equivalent in it.
[1]: For similar reasons, none of the core implementations of the 1990s-era dynamic scripting languages have a very good true multi-threading story. (Some of the alternates can do it, like Jython.) They don't all even have implementations, and last I knew, none of them have anything that you ought to use in production. I don't think this is a fundamental limitation of any of the languages in question, it's just that it's really hard to retrofit threading onto a non-threaded code base after ~ten very heavy years of development. There's a set of other things that are hard to retrofit in to an existing code base if they don't start there from day one, like "unit testability" or "proper string handling".
- eloff 10y agoThis is not true - it is intrinsic to Python - not just CPython. Part of it is the garbage collector, reference counting is expensive to do without the GIL - but that can be changed. IronPython, Jython, some translations of PyPy all attest to that. The other part is C methods, which includes all operations on Python collections like lists and dicts, are atomic, making those collections safe to share and modify across threads. Most python code in the wild that utilizes threads makes use of those guarantees. So you either use fine-grained locking, or lock-free data structures, or STM, etc to protect ALL collection instances, or you've basically made a Python 3000 that breaks compatibility with all existing Python code. So every Python implementation that removes the GIL chooses to go the first route of pessimistically making everything threadsafe - which was tried in Java way back when and was and is a miserable failure. You give up too much performance for it to be worthwhile. So the problem in Python will never be fixed in my opinion - the problem is Python and there's no saving it. Without breaking backwards compatibility it will always be better to use processes instead of threads and keep the GIL.
- fatbird 10y agoYou should watch the video. I won't try to repeat it all here, but Hastings identifies the performance problem areas and lays out some plausible directions for tackling it, and a plausible implementation/changeover path that doesn't create another big breaking change.
- jerf 10y agoIf you carefully read what I said, I think you'll find there's less disagreement between what you posted and what I said than your tone suggests you think. I did, for instance, mention C methods, but you seem to imply that that matters to all Python implementations. It only matters to CPython. Barring what last I knew was some fairly experimental stuff from PyPy, Python extensions are not generally compatible between the implementations. (Where you see libraries that appear to work on multiple implementations, AIUI you're actually seeing multiple implementations of the binding code.) Personally my feeling is that if you care enough about performance to be threading in Python, you care enough about performance to not be using Python anyhow. It's a nice language, it's by far my favorite of the scripting languages of its family, but even with all the PyPy JIT and other magic tech words you can throw at it, it's still a slow language. When that doesn't matter, great; when it does, you eventually reach the point where you're stuck. (I've never been overwhelmed by the "implement it in C then" argument, for many reasons, and nowadays to an increasing degree if you're going to implement your "core" in another language for speed, there's a language you can choose that is both nice to use and already fast for your task, and that's only going to get more true as time goes on.)
- eloff 10y agoThe C methods do unfortunately matter to all implementations. They gave thread safety and almost all Python code in the wild relies on that property, so it's become an undocumented property of the Python language specification and an intrinsic flaw of the language itself.
- andreasvc 10y agoReference counting and the C API are properties of the CPython implementation, not Python the language. Other Python implementations do NOT use reference counting and don't offer the C API, but do make guarantees about atomic operations. I think the problem with backward compatibility is with wanting to preserve the C API and the existing extension modules implemented in C, not necessarily with Python the language.