4 ms·
Oh how I wish Python had just sucked it up at 3.0 and eaten the performance hit for removing the GIL. By now, everybody would have optimized it back to normal
by bsder 7y ago
Oh how I wish Python had just sucked it up at 3.0 and eaten the performance hit for removing the GIL.
By now, everybody would have optimized it back to normal (or better!).
- loeg 7y agoThere was no possible way to do that given the things python lets you get away with "atomically". Requiring explicit synchronization to avoid data races would have been an even bigger language break than the other breaking changes in 3. Python is not a language for people who want to do fine-grained explicit concurrency. And doing it implicitly would require stupidly fine-grained locks on all objects, destroying any performance gains; there's no way to "optimize" that to GIL performance.
- Doxin 7y agoThere's still plenty of programs where you'd want to spawn a thread that basically doesn't touch any of the data of the parent thread. It'd be nice if there was a way around the GIL for those cases. As an example take the multiprocessing.pool.ThreadPool.map function. Most of the use-cases of that function only read from the parent threads memory once to pass the function arguments. After that the thread may very well spend a lot of time only reading memory it reserved itself. It's rather wasteful to have that thread wait on the GIL. Of course I haven't got an obvious solution either but it seems to me that for at least a subset of the uses of threading you could work around the GIL without breaking python too badly. Perhaps split the GIL into a per-thread Thread-interpreter-lock. That way each object can simply be annotated with which lock it belongs to. That way you still get proper atomicity like you're used to in python, but a thread might also actually run concurrently a lot of the time if it only touches objects it created itself.
- loeg 7y ago> There's still plenty of programs where you'd want to spawn a thread that basically doesn't touch any of the data of the parent thread. It'd be nice if there was a way around the GIL for those cases. Subinterpreters gets you that (eventually; not in v1 apparently). > Perhaps split the GIL into a per-thread Thread-interpreter-lock. That way each object can simply be annotated with which lock it belongs to. That way you still get proper atomicity like you're used to in python, but a thread might also actually run concurrently a lot of the time if it only touches objects it created itself. I don't think this scheme would work especially well (or not better than subinterpreters). Message passing still requires copying, or some way to quiesce other threads and recursively change lock ownership of an object graph. To avoid deadlocks, at least one thread will need to drop its own lock to acquire the thread lock of objects owned by a 2nd thread. I'm not sure you could do that and preserve the legacy global atomicity behavior, and it would by definition be slower and bloatier on any individual thread than the existing GIL behavior.
- jashmatthews 7y agoDoes the CPython GIL give you a stronger effective guarantee than the one in CRuby? Because the one in CRuby will absolutely not save you from data races.
- shockinglytrue 7y agoThere really was no hope of this without ditching reference counting, and consequently a much more destablizing change to the C API which was all but entirely unharmed during the 3.x transition. There have been several attempts to bring new science to bear on solving the refcounting problem, I don't think any of them ever got far enough to be taken seriously A significant chunk of Python performance is derived from optimizations made possible by the GIL - for example, so long as it is held, no additional locks are necessary to allocate a small object, unless that entails asking the system allocator for more memory
- __d 7y agoThe big problem with the 2->3 transition was that the cost exceeded the perceived benefit. Removing the GIL would have increased the cost, but also the benefit. I think the core team over-estimated the benefits of the change, hence people's reluctance to pay the cost. I suspect that "threading like Go" might have been a benefit more worth paying the cost.
- CogitoCogito 7y ago> The big problem with the 2->3 transition was that the cost exceeded the perceived benefit. I believe it's generally agreed that the cost of the transition was surprisingly high. Hindsight is 20/20. > Removing the GIL would have increased the cost, but also the benefit. The only attempts I'm aware of resulted in a significant (~50%) slowdown in single-threaded code execution due to the requirement of adding in so many more locks elsewhere (removing the GIL doesn't remove the need for the locks around the various critical sections). Sure maybe it would result in better performance for some programs, but I'm not even sure it would result in something better than a cleaner design. In my experience, the portions of code that could make use of the threading could just be moved to a C/C++-extension and make use of it there (though in 99% of cases I wouldn't go that far and just stick to e.g. numpy). The examples of the code that really should stay in python, but also should make use of concurrency seem to usually sit quite nicely in the pypy framework. That's not to say that removing the GIL wouldn't be nice, but I think the need for it is often overstated.
- klyrs 7y agoOh how I wish Python 3 was named Monty or something. Kill the GIL! Ruthlessly abandon backwards compatibility in the C API! It's a new language, you can do anything! Folks who delayed adoption until this year had a long and glorious period of stability, and are now waking up to a shitshow of incompatibility and half-adopted features.
- acdha 7y ago> a shitshow of incompatibility and half-adopted features. To the extent that this isn’t wrong, it’s because they’re years behind on testing and upgrading from unsupported libraries. The switch isn’t hard on a relatively clean project.
- klyrs 7y agoI did preface that I was talking about people who delayed adoption until the very last minute. The stability was seductive, and IMO the forced transition was long overdue. In hindsight, I wish that had happened sometime around Py36 -- but at this point, I'd rather a new language unfettered by the old C API. But what I said about half-adopted features is due to my experience with rather large bodies of code -- they're not "relatively clean" in any sense, and I often find myself doing long-overdue modernization that hinders my productivity
- acdha 7y agoAgain, is the problem the fraction of a Python 3 upgrade which can’t be done automatically or needing to update old third-party libraries or not having sufficient test coverage to be able to make any changes with confidence?
- klyrs 7y agoI'm not sure what the point of your questions is. Breaking backwards-compatibility makes headaches for maintainers for all of the reasons you list and more. I'd love to see Monty take an even more radical approach, with a "thin enough" compatibility layer. It'd be a great language with all sorts of improvements and mostly-familiar syntax, and old, stable Python libraries would stay stable -- even through a compatibility layer.
- BiteCode_dev 7y agoThere are plenty of things that could have been down, but the Python core dev team is small and very resource limited.