6 ms·

Still not encouraged by the no-GIL, "We don't want another Python 2->3 situation", yet very little proffered on how to avoid that scenario. More documentation o

Still not encouraged by the no-GIL, "We don't want another Python 2->3 situation", yet very little proffered on how to avoid that scenario. More documentation on writing thread-safe code, suggested tooling to lint for race conditions (for whatever it is worth), discussions with popular C libraries, dedicated support channels for top tier packages, what about the enormous long-tail of abandoned extensions which still work today, etc.

thiht 3y ago
> Note that if the program imports one single C-extension that uses the GIL on the no-GIL build, it's designed to switch back to the GIL automatically. So this is not a 2=>3 situation where non-compatible code breaks. Sounds good enough to me, am I missing something?
LexiMax 3y ago
In a past life I hacked on PHP for a living, and in the time it took Python 2 to ride off into the sunset, PHP got two major migrations under its belt in 5.2 to 5.3, and then again 5.6 to 7.0. It was amazing to see the contrast between the two languages. PHP gave you plenty of reasons to upgrade, and the amount of incompatible breaking changes was kept to a minimum, often paired with a way to easily shim older code to continue working. I really hope to see no-GIL make it into Python, but in the back of my mind I also worry about what lessons were learned from the 2 to 3 transition. Does the Python team have a more effective plan this time around?
- charrondev 3y ago
  I’ve taken an application codebase from PHP 5.3 to 8.2 now and it was relatively easy the whole way. The real key to minimize the pain was writing effective integration tests with high coverage. We didn’t have a good test suite to start but once we added some utilities to easily call our various endpoints (and internal API client if you will) and make assertions about the coverage came quickly. Popular frameworks like Laravel offer such test utilities out of the box now. That combined with static analysis tools like psalm make it so we can fearlessly move past major upgrades. One thing I was surprised at was just how much crap PHP allowed with just a notice (not even a warning for a long time). A lot of that stuff still works (although over time some notices have progressed to warnings or errors gradually). We have our test suite convert any notices or warnings to exceptions and fail the test case.
  - acdha 3y ago
    > The real key to minimize the pain was writing effective integration tests with high coverage I think this makes it really hard to do comparisons: I’ve done Python 2 to 3 migrations which took an hour or two because the code had tests and was well-maintained, and PHP migrations which were painful slogs without tests and sloppy code (“is this ignored error new or something we should have fixed in the 2000s?”). Most developers don’t have enough data points to say whether the experience they had was due to the language or the culture.
    - charrondev 3y ago
      I’m not familiar enough with the python transition to say much. I can think of a few things that the PHP developers did that helped make the transition easier: - multibyte aware string functions were implemented as a separate (and optional) extension with separately named functions (prefixed with mb) and there was a popular community polyfill from the Symfony project (and is for many new language functions). - Weird sloppy behaviours (like performing array access on a Boolean, or trying to access a property on null, and many more than would silently just turn into null/false) had lengthy deprecation periods and if you had error logging turned on you could clean these up relatively easily even without a big test suite.
      - acdha 3y ago
        > multibyte aware string functions were implemented as a separate (and optional) extension with separately named functions (prefixed with mb) Python had a different take on this with some interesting psychology: you had a new string type which had to explicitly be converted (i.e. concatenating a Unicode string with a byte string causes an exception), which had a stark divide. Projects which had previously handled Unicode correctly converted almost trivially, but the projects which had been sloppy were a morass trying to figure out where Unicode was desirable and where you really needed raw bytes. Almost all of the code I saw where this was a problem didn’t handle Unicode properly but the developers _hated_ the idea of the language forcing them to fix those bugs.
doctoboggan 3y ago
The big and obvious difference is that all the GIL vs no-GIL stuff happens in the background and your average python dev can just ignore it if they want to. The interpreter will note if you have C extensions that don't opt in to no-GIL and then will give you the GIL version. This is _very_ different to the 2-to-3 transition where absolutely every single person, even those who couldn't care less, had to change their code if they wanted to use python 3.
- quickthrower2 3y ago
  But you need to pick your horse. In 5 years time, Python will either be GIL or no GIL, and it is hard to tell which. It might be a setting (which might be more ideal). If you assume nogil, you need to choose dependencies that support that. You may need to trade off: eschew dependencies that aren't looking like they will be nogil compatible by the deadline. You are stuck on Python 3.18 maintenance branch or whatever, rather than the 3.19 (in reality .. 4.0) version. Or choose gil then you can use everything. But is there a prisoners dilemma - everyone picks gil, uses whatever dependencies, library maintainers assuming this don't bother to add nogil support, and then the decision becomes to stick to gil, which if you suspect will happen makes you reason even harder not to support nogil.
  - doctoboggan 3y ago
    I don’t really understand this. Unless I am missing something you should always pick the “no GIL” version as that will work with or without a GIL. Thread safe No GIL code would be totally fine to run on python compiled with the GIL with zero modifications. Because of this I don’t expect there to be multiple versions of any library. Once a library does the (admittedly heavy) lift to no GIL it will just be the main version of that library going forward.
    - quickthrower2 3y ago
      Each library maintainer (probably mostly volunteers) has to decide whether to put effort into making their code thread safe. Clearly it won't be 100% of libraries that "upgrade". Then on top of that, they know their effort might be for nothing if the decision is made to keep Python GIL-only all along (one of the possible 3 outcomes at the end of the 5 years: ["gil", "nogil", "both supported").
threatripper 3y ago
I was assuming that no-GIL will only be enabled if all imported libraries support it. That means that they are marked as no-GIL ready and otherwise the import would throw an exception. Not sure how it is implemented now but that sounded very reasonable to me. The no-GIL compatible code would start with the core libraries and then expand from that. Using legacy libraries just means that you have to revert back to GIL-mode. Any no-GIL enabled library should 100% still function in GIL-mode, so I don't expect the Python 2->3 transition situation to repeat.
rtpg 3y ago
> what about the enormous long-tail of abandoned extensions which still work today, etc. I mean there they're talking about keeping GIL in (and I imagine that will be the case for many many years) so those would still keep working. The fear is if some libraries just drop GIL-ful support, but there too I am hopeful for that not to be the case.

thiht 3y ago

> Note that if the program imports one single C-extension that uses the GIL on the no-GIL build, it's designed to switch back to the GIL automatically. So this is not a 2=>3 situation where non-compatible code breaks. Sounds good enough to me, am I missing something?

LexiMax 3y ago

In a past life I hacked on PHP for a living, and in the time it took Python 2 to ride off into the sunset, PHP got two major migrations under its belt in 5.2 to 5.3, and then again 5.6 to 7.0. It was amazing to see the contrast between the two languages. PHP gave you plenty of reasons to upgrade, and the amount of incompatible breaking changes was kept to a minimum, often paired with a way to easily shim older code to continue working. I really hope to see no-GIL make it into Python, but in the back of my mind I also worry about what lessons were learned from the 2 to 3 transition. Does the Python team have a more effective plan this time around?

charrondev 3y ago
I’ve taken an application codebase from PHP 5.3 to 8.2 now and it was relatively easy the whole way. The real key to minimize the pain was writing effective integration tests with high coverage. We didn’t have a good test suite to start but once we added some utilities to easily call our various endpoints (and internal API client if you will) and make assertions about the coverage came quickly. Popular frameworks like Laravel offer such test utilities out of the box now. That combined with static analysis tools like psalm make it so we can fearlessly move past major upgrades. One thing I was surprised at was just how much crap PHP allowed with just a notice (not even a warning for a long time). A lot of that stuff still works (although over time some notices have progressed to warnings or errors gradually). We have our test suite convert any notices or warnings to exceptions and fail the test case.
- acdha 3y ago
  > The real key to minimize the pain was writing effective integration tests with high coverage I think this makes it really hard to do comparisons: I’ve done Python 2 to 3 migrations which took an hour or two because the code had tests and was well-maintained, and PHP migrations which were painful slogs without tests and sloppy code (“is this ignored error new or something we should have fixed in the 2000s?”). Most developers don’t have enough data points to say whether the experience they had was due to the language or the culture.
  - charrondev 3y ago
    I’m not familiar enough with the python transition to say much. I can think of a few things that the PHP developers did that helped make the transition easier: - multibyte aware string functions were implemented as a separate (and optional) extension with separately named functions (prefixed with mb) and there was a popular community polyfill from the Symfony project (and is for many new language functions). - Weird sloppy behaviours (like performing array access on a Boolean, or trying to access a property on null, and many more than would silently just turn into null/false) had lengthy deprecation periods and if you had error logging turned on you could clean these up relatively easily even without a big test suite.
    - acdha 3y ago
      > multibyte aware string functions were implemented as a separate (and optional) extension with separately named functions (prefixed with mb) Python had a different take on this with some interesting psychology: you had a new string type which had to explicitly be converted (i.e. concatenating a Unicode string with a byte string causes an exception), which had a stark divide. Projects which had previously handled Unicode correctly converted almost trivially, but the projects which had been sloppy were a morass trying to figure out where Unicode was desirable and where you really needed raw bytes. Almost all of the code I saw where this was a problem didn’t handle Unicode properly but the developers _hated_ the idea of the language forcing them to fix those bugs.

doctoboggan 3y ago

The big and obvious difference is that all the GIL vs no-GIL stuff happens in the background and your average python dev can just ignore it if they want to. The interpreter will note if you have C extensions that don't opt in to no-GIL and then will give you the GIL version. This is _very_ different to the 2-to-3 transition where absolutely every single person, even those who couldn't care less, had to change their code if they wanted to use python 3.

quickthrower2 3y ago
But you need to pick your horse. In 5 years time, Python will either be GIL or no GIL, and it is hard to tell which. It might be a setting (which might be more ideal). If you assume nogil, you need to choose dependencies that support that. You may need to trade off: eschew dependencies that aren't looking like they will be nogil compatible by the deadline. You are stuck on Python 3.18 maintenance branch or whatever, rather than the 3.19 (in reality .. 4.0) version. Or choose gil then you can use everything. But is there a prisoners dilemma - everyone picks gil, uses whatever dependencies, library maintainers assuming this don't bother to add nogil support, and then the decision becomes to stick to gil, which if you suspect will happen makes you reason even harder not to support nogil.
- doctoboggan 3y ago
  I don’t really understand this. Unless I am missing something you should always pick the “no GIL” version as that will work with or without a GIL. Thread safe No GIL code would be totally fine to run on python compiled with the GIL with zero modifications. Because of this I don’t expect there to be multiple versions of any library. Once a library does the (admittedly heavy) lift to no GIL it will just be the main version of that library going forward.
  - quickthrower2 3y ago
    Each library maintainer (probably mostly volunteers) has to decide whether to put effort into making their code thread safe. Clearly it won't be 100% of libraries that "upgrade". Then on top of that, they know their effort might be for nothing if the decision is made to keep Python GIL-only all along (one of the possible 3 outcomes at the end of the 5 years: ["gil", "nogil", "both supported").

threatripper 3y ago

I was assuming that no-GIL will only be enabled if all imported libraries support it. That means that they are marked as no-GIL ready and otherwise the import would throw an exception. Not sure how it is implemented now but that sounded very reasonable to me. The no-GIL compatible code would start with the core libraries and then expand from that. Using legacy libraries just means that you have to revert back to GIL-mode. Any no-GIL enabled library should 100% still function in GIL-mode, so I don't expect the Python 2->3 transition situation to repeat.

rtpg 3y ago

> what about the enormous long-tail of abandoned extensions which still work today, etc. I mean there they're talking about keeping GIL in (and I imagine that will be the case for many many years) so those would still keep working. The fear is if some libraries just drop GIL-ful support, but there too I am hopeful for that not to be the case.