4 ms·

There really was no hope of this without ditching reference counting, and consequently a much more destablizing change to the C API which was all but entirely u

There really was no hope of this without ditching reference counting, and consequently a much more destablizing change to the C API which was all but entirely unharmed during the 3.x transition. There have been several attempts to bring new science to bear on solving the refcounting problem, I don't think any of them ever got far enough to be taken seriously A significant chunk of Python performance is derived from optimizations made possible by the GIL - for example, so long as it is held, no additional locks are necessary to allocate a small object, unless that entails asking the system allocator for more memory

__d 6y ago
The big problem with the 2->3 transition was that the cost exceeded the perceived benefit. Removing the GIL would have increased the cost, but also the benefit. I think the core team over-estimated the benefits of the change, hence people's reluctance to pay the cost. I suspect that "threading like Go" might have been a benefit more worth paying the cost.
- CogitoCogito 6y ago
  > The big problem with the 2->3 transition was that the cost exceeded the perceived benefit. I believe it's generally agreed that the cost of the transition was surprisingly high. Hindsight is 20/20. > Removing the GIL would have increased the cost, but also the benefit. The only attempts I'm aware of resulted in a significant (~50%) slowdown in single-threaded code execution due to the requirement of adding in so many more locks elsewhere (removing the GIL doesn't remove the need for the locks around the various critical sections). Sure maybe it would result in better performance for some programs, but I'm not even sure it would result in something better than a cleaner design. In my experience, the portions of code that could make use of the threading could just be moved to a C/C++-extension and make use of it there (though in 99% of cases I wouldn't go that far and just stick to e.g. numpy). The examples of the code that really should stay in python, but also should make use of concurrency seem to usually sit quite nicely in the pypy framework. That's not to say that removing the GIL wouldn't be nice, but I think the need for it is often overstated.
  - __d 6y ago
    I think it was Greg Stein who made a significant attempt on removing the GIL back in the 1.x timeframe? IIRC, the penalty was much less than 50%, but it was still a significant cost. I think that as time has passed, the inability to simply utilize more than one core becomes more of a problem. The transition to supporting async serves mostly to highlight its inability to use a thread pool like Go, as an example. While I agree that C extensions (and especially, the standard extensions, like eg. socket) really help a lot to limit the impact of of the GIL, I think the usual advice to "just move it to a C extension" or "use multiple processes" are kinda ridiculous to anyone who's not used to the situation. Unfortunately, fixing it will almost certainly require breaking the C API, and we've only just about recovered from the 2->3 breakage: I don't think the language can afford to do it again, perhaps ever. Which is why I think it's a shame that the last transition didn't actually achieve more ...
smabie 6y ago
> A significant chunk of Python performance is derived from optimizations made possible by the GIL Considering that Python has some of the worst performance around, does it really matter? Other languages don't have this problem, and they're orders of magnitude faster.
- shockinglytrue 6y ago
  If performance doesn't matter, why bother attempt such a complex change at all?

__d 6y ago

The big problem with the 2->3 transition was that the cost exceeded the perceived benefit. Removing the GIL would have increased the cost, but also the benefit. I think the core team over-estimated the benefits of the change, hence people's reluctance to pay the cost. I suspect that "threading like Go" might have been a benefit more worth paying the cost.

CogitoCogito 6y ago
> The big problem with the 2->3 transition was that the cost exceeded the perceived benefit. I believe it's generally agreed that the cost of the transition was surprisingly high. Hindsight is 20/20. > Removing the GIL would have increased the cost, but also the benefit. The only attempts I'm aware of resulted in a significant (~50%) slowdown in single-threaded code execution due to the requirement of adding in so many more locks elsewhere (removing the GIL doesn't remove the need for the locks around the various critical sections). Sure maybe it would result in better performance for some programs, but I'm not even sure it would result in something better than a cleaner design. In my experience, the portions of code that could make use of the threading could just be moved to a C/C++-extension and make use of it there (though in 99% of cases I wouldn't go that far and just stick to e.g. numpy). The examples of the code that really should stay in python, but also should make use of concurrency seem to usually sit quite nicely in the pypy framework. That's not to say that removing the GIL wouldn't be nice, but I think the need for it is often overstated.
- __d 6y ago
  I think it was Greg Stein who made a significant attempt on removing the GIL back in the 1.x timeframe? IIRC, the penalty was much less than 50%, but it was still a significant cost. I think that as time has passed, the inability to simply utilize more than one core becomes more of a problem. The transition to supporting async serves mostly to highlight its inability to use a thread pool like Go, as an example. While I agree that C extensions (and especially, the standard extensions, like eg. socket) really help a lot to limit the impact of of the GIL, I think the usual advice to "just move it to a C extension" or "use multiple processes" are kinda ridiculous to anyone who's not used to the situation. Unfortunately, fixing it will almost certainly require breaking the C API, and we've only just about recovered from the 2->3 breakage: I don't think the language can afford to do it again, perhaps ever. Which is why I think it's a shame that the last transition didn't actually achieve more ...

smabie 6y ago

> A significant chunk of Python performance is derived from optimizations made possible by the GIL Considering that Python has some of the worst performance around, does it really matter? Other languages don't have this problem, and they're orders of magnitude faster.

shockinglytrue 6y ago
If performance doesn't matter, why bother attempt such a complex change at all?