9 ms·

That's the exact attitude that lead to decades of pain with the 2 to 3 transition and libraries though. There was zero plan for how to help libraries migrate f

That's the exact attitude that lead to decades of pain with the 2 to 3 transition and libraries though. There was zero plan for how to help libraries migrate from python 2 to 3, or more importantly how does one library support _both_ python 2 and 3 from one codebase as their users take time to switch their python interpretor. The attitude of "just turn off GIL if you don't need it, just use libraries that support turning it off" means library authors will be asked to provide versions of their library that do and don't support non-GIL mode. That's a big burden to dump on library maintainers.

miraculixx 3y ago
Indeed. The burden is 2x at least. They already estimate it's going to take 5+ years, so that's 10x the cost of a current library. At least. If the burden becomes too big, at some point people may find it is the easier option to switch to a different ecosystem. I hope not.
- JonChesterfield 3y ago
  I hope it'll be lua. I fear it'll be javascript.
  - anyoneamous 3y ago
    I hope it'll be Nim, I fear it'll be Julia.
  - concordDance 3y ago
    Lua? Really? That's by far the most unpleasant language I've coded in.
    - JonChesterfield 3y ago
      Lexical scope, first class functions, native coroutines, compiler available at runtime. Trivially extensible. It's semantically really close to lisp. Yep, lua's one of my favourites. The front end syntax isn't what I'd like but whatever, I can still see the AST through it with a little effort.
    - DonHopkins 3y ago
      Obviously there are a LOT of languages you've never coded in.
      - concordDance 3y ago
        Yes, I've never coded in brainfuck. But of the languages people actually use its been the most painful.
  - imtringued 3y ago
    I am not using luajit for various reasons but the problem with the regular Lua interpreter is that the lua ffi backport from luajit is not included by default and the third party ones are broken on ARM.
paulddraper 3y ago
> how does one library support _both_ python 2 and 3 from one codebase How does one library support both GIL and no-GIL? Easy, it supports no-GIL, so it supports both. Done.
- qbasic_forever 3y ago
  You're saying, "just rewrite and re-release all your dependencies, it's easy!" which was exactly what happened disastrously with python 2 to 3.
  - wokwokwok 3y ago
    You can down vote this til the cows come home but it’s historically what happened. This “that’s not how it went” stuff going down is quite just blatant historical revisionism. Maybe it’s a good idea? Maybe it’s not? …but anyone down voting “this reminds me of the Python 2/3” fiasco has no idea what they’re talking about.
    - qbasic_forever 3y ago
      It's good if there's a benefit for everyone, like python 3 fixing the terrible Unicode story. It's not clear non-GIL will even be a net performance improvement for most people--you are effectively moving syncronization from the core runtime to each and every library and program at the edge. Writing safe code at that level doesn't come for free, your basic program will be slower (and likely buggier) if every call into a library is now doing its own little bespoke GIL instead of relying on python's global one like now.
      - wokwokwok 3y ago
        I’m not gonna argue that point; but it seems massively disingenuous to down vote someone who complains “but now I have to rewrite my library because some people might use it in non-GIL mode”. That’s not whining; it’s just an observation that the committee making these decisions gives zero ducks about the impact this will have for anyone other than the handful of vested parties involved in making the decisions. Pypi has what, 500k projects on it? Many abandoned. Whom exactly is going to update those? Or do packages get an automatic “doesn’t work with no-GIL” unless the author explicitly opts to enable it? Or do we live in a future where any package, with any dependency may or may not have undefined behaviour in no-GIL mode? Like, sure… it’s a good change for many people… once all the hard work is done by the community. Does that remind you of anything? Mmm.
MarkMarine 3y ago
I’m no fan of Java, but in comparison with python, Java’s focus on extreme backwards compatibility and their ability to actually execute on this promise year after year stands in stark relief with how python has handled the same challenges. I have low confidence, despite their claims this won’t be python 4, that this will actually be executed well. Looking forward to having homebrew deliver python@3.25_GIL and python@3.25_no_GIL with each flipping package I install.
- qbasic_forever 3y ago
  Yep, and you'll have pip3-gil and pip3-nogil binaries because each permutation of python has separate and incompatible site-packages folders and libraries. It could get really ugly.
  - kelipso 3y ago
    Lol the correct statement is it will get really ugly. Pretty much no question it will be a bigger mess than 2 to 3 transition. Here's hoping my subfield will move to a different language in the meanwhile because I don't want to deal with this shit again.
    - MarkMarine 3y ago
      All your hopes of GIL free code ruined by a GIL requiring left pad dep buried deep in the dependency tree.
      - baq 3y ago
        Fortunately Python’s package management is so horrible left pad is not a problem.
  - TylerE 3y ago
    Is the situation in rust, where the answer is apparently to vendor the world, much better? Don't many of the big rust libraries still depend on nightly, too?
xmonkee 3y ago
no-GIL libs will work with the GIL on surely? So you won’t have to support both?
- qbasic_forever 3y ago
  Where will you get no-GIL libraries, especially in the early days? Just yell at the maintainers of core libs like flask and requests until they use their volunteer and spare time to implement incredibly complex and tricky locking semantics all over their codebases, AND test it all with both GIL and non-GIL interpretors at scale to suss out race conditions? That just happens for free and overnight because a lot of people are plus one mashing on GitHub issues I guess?
  - kzrdude 3y ago
    Flask is pure python, isn't it?
    - qbasic_forever 3y ago
      Even pure python code could have race conditions with the GIL disabled. Stuff like accessing and modifying a dictionary item in python code is assumed and currently guaranteed to be atomic because of the GIL. Remove the GIL and decades of assumptions break.
      - kzrdude 3y ago
        I think that's a misconception? At least the way how I understood this issue. Things like `map[k] = v` would be atomic both before and after nogil, and things like `map[k] += 1` are not atomic even with GIL, the read and store can be split from each other.
      - samus 3y ago
        The GIL doesn't serialize the execution of threads, but ensures only one executes at a time. Therefore, the race condition issues you describe already exist even with the GIL. The GIL is only there to ensure that the interpreter doesn't corrupt Python objects and its internal data structures, which in the best case leads to a crash.
smashed 3y ago
As I understand it, as a library author you either do absolutely nothing and your library will be marked as requiring GIL by default. Nothing to do, you keep on working with that good old GIL and nothing changes. Or you make the extra effort of being thread safe and you can declare your library as not requiring the GIL. Now if a user script mixes your GIL free lib with an older lib that has not been updated, well, too bad for them. Even with your hard work, the code will still operate like before, everything gets the GIL treatment. Normal python devs will need to track down which pesky dependency of their script is causing the GIL slowdown. Kinda sucks but at least nothing breaks. It's a sensitive, opt-in, and safe way forward. Hard to argue against it, really..
- miraculixx 3y ago
  Where can I read up on this planned way of working? This is not what the PEP says
  - LegionMammal978 3y ago
    It's in the "Py_mod_gil Slot" section [0] in the PEP: > In --disable-gil builds, when loading an extension, CPython will check for a new PEP 489-style Py_mod_gil slot. If the slot is set to Py_mod_gil_not_used, then extension loading proceeds as normal. If the slot is not set, the interpreter pauses all threads and enables the GIL before continuing. Additionally, the interpreter will issue a visible warning naming the extension, that the GIL was enabled (and why) and the steps the user can take to override it. [0] https://peps.python.org/pep-0703/#py-mod-gil-slot https://peps.python.org/pep-0703/#py-mod-gil-slot
    - miraculixx 3y ago
      Thanks. Glad to know now!
    - miraculixx 3y ago
      Here's what's puzzling me about this: Why then is there even a need for a seperate nogil build? If it is like this says, wouldn't it be easier to just make the standard build switch to gil or nogil automatically (or honor the users choice). The fact that the SC thinks having two versions suggests there is more complexity involved than this section of the PEP leads readers to believe.

miraculixx 3y ago

Indeed. The burden is 2x at least. They already estimate it's going to take 5+ years, so that's 10x the cost of a current library. At least. If the burden becomes too big, at some point people may find it is the easier option to switch to a different ecosystem. I hope not.

JonChesterfield 3y ago
I hope it'll be lua. I fear it'll be javascript.
- anyoneamous 3y ago
  I hope it'll be Nim, I fear it'll be Julia.
- concordDance 3y ago
  Lua? Really? That's by far the most unpleasant language I've coded in.
  - JonChesterfield 3y ago
    Lexical scope, first class functions, native coroutines, compiler available at runtime. Trivially extensible. It's semantically really close to lisp. Yep, lua's one of my favourites. The front end syntax isn't what I'd like but whatever, I can still see the AST through it with a little effort.
  - DonHopkins 3y ago
    Obviously there are a LOT of languages you've never coded in.
    - concordDance 3y ago
      Yes, I've never coded in brainfuck. But of the languages people actually use its been the most painful.
- imtringued 3y ago
  I am not using luajit for various reasons but the problem with the regular Lua interpreter is that the lua ffi backport from luajit is not included by default and the third party ones are broken on ARM.

paulddraper 3y ago

> how does one library support _both_ python 2 and 3 from one codebase How does one library support both GIL and no-GIL? Easy, it supports no-GIL, so it supports both. Done.

qbasic_forever 3y ago
You're saying, "just rewrite and re-release all your dependencies, it's easy!" which was exactly what happened disastrously with python 2 to 3.
- wokwokwok 3y ago
  You can down vote this til the cows come home but it’s historically what happened. This “that’s not how it went” stuff going down is quite just blatant historical revisionism. Maybe it’s a good idea? Maybe it’s not? …but anyone down voting “this reminds me of the Python 2/3” fiasco has no idea what they’re talking about.
  - qbasic_forever 3y ago
    It's good if there's a benefit for everyone, like python 3 fixing the terrible Unicode story. It's not clear non-GIL will even be a net performance improvement for most people--you are effectively moving syncronization from the core runtime to each and every library and program at the edge. Writing safe code at that level doesn't come for free, your basic program will be slower (and likely buggier) if every call into a library is now doing its own little bespoke GIL instead of relying on python's global one like now.
    - wokwokwok 3y ago
      I’m not gonna argue that point; but it seems massively disingenuous to down vote someone who complains “but now I have to rewrite my library because some people might use it in non-GIL mode”. That’s not whining; it’s just an observation that the committee making these decisions gives zero ducks about the impact this will have for anyone other than the handful of vested parties involved in making the decisions. Pypi has what, 500k projects on it? Many abandoned. Whom exactly is going to update those? Or do packages get an automatic “doesn’t work with no-GIL” unless the author explicitly opts to enable it? Or do we live in a future where any package, with any dependency may or may not have undefined behaviour in no-GIL mode? Like, sure… it’s a good change for many people… once all the hard work is done by the community. Does that remind you of anything? Mmm.

MarkMarine 3y ago

I’m no fan of Java, but in comparison with python, Java’s focus on extreme backwards compatibility and their ability to actually execute on this promise year after year stands in stark relief with how python has handled the same challenges. I have low confidence, despite their claims this won’t be python 4, that this will actually be executed well. Looking forward to having homebrew deliver python@3.25_GIL and python@3.25_no_GIL with each flipping package I install.

qbasic_forever 3y ago
Yep, and you'll have pip3-gil and pip3-nogil binaries because each permutation of python has separate and incompatible site-packages folders and libraries. It could get really ugly.
- kelipso 3y ago
  Lol the correct statement is it will get really ugly. Pretty much no question it will be a bigger mess than 2 to 3 transition. Here's hoping my subfield will move to a different language in the meanwhile because I don't want to deal with this shit again.
  - MarkMarine 3y ago
    All your hopes of GIL free code ruined by a GIL requiring left pad dep buried deep in the dependency tree.
    - baq 3y ago
      Fortunately Python’s package management is so horrible left pad is not a problem.
- TylerE 3y ago
  Is the situation in rust, where the answer is apparently to vendor the world, much better? Don't many of the big rust libraries still depend on nightly, too?

xmonkee 3y ago

no-GIL libs will work with the GIL on surely? So you won’t have to support both?

qbasic_forever 3y ago
Where will you get no-GIL libraries, especially in the early days? Just yell at the maintainers of core libs like flask and requests until they use their volunteer and spare time to implement incredibly complex and tricky locking semantics all over their codebases, AND test it all with both GIL and non-GIL interpretors at scale to suss out race conditions? That just happens for free and overnight because a lot of people are plus one mashing on GitHub issues I guess?
- kzrdude 3y ago
  Flask is pure python, isn't it?
  - qbasic_forever 3y ago
    Even pure python code could have race conditions with the GIL disabled. Stuff like accessing and modifying a dictionary item in python code is assumed and currently guaranteed to be atomic because of the GIL. Remove the GIL and decades of assumptions break.
    - kzrdude 3y ago
      I think that's a misconception? At least the way how I understood this issue. Things like `map[k] = v` would be atomic both before and after nogil, and things like `map[k] += 1` are not atomic even with GIL, the read and store can be split from each other.
    - samus 3y ago
      The GIL doesn't serialize the execution of threads, but ensures only one executes at a time. Therefore, the race condition issues you describe already exist even with the GIL. The GIL is only there to ensure that the interpreter doesn't corrupt Python objects and its internal data structures, which in the best case leads to a crash.

smashed 3y ago

As I understand it, as a library author you either do absolutely nothing and your library will be marked as requiring GIL by default. Nothing to do, you keep on working with that good old GIL and nothing changes. Or you make the extra effort of being thread safe and you can declare your library as not requiring the GIL. Now if a user script mixes your GIL free lib with an older lib that has not been updated, well, too bad for them. Even with your hard work, the code will still operate like before, everything gets the GIL treatment. Normal python devs will need to track down which pesky dependency of their script is causing the GIL slowdown. Kinda sucks but at least nothing breaks. It's a sensitive, opt-in, and safe way forward. Hard to argue against it, really..

miraculixx 3y ago
Where can I read up on this planned way of working? This is not what the PEP says
- LegionMammal978 3y ago
  It's in the "Py_mod_gil Slot" section [0] in the PEP: > In --disable-gil builds, when loading an extension, CPython will check for a new PEP 489-style Py_mod_gil slot. If the slot is set to Py_mod_gil_not_used, then extension loading proceeds as normal. If the slot is not set, the interpreter pauses all threads and enables the GIL before continuing. Additionally, the interpreter will issue a visible warning naming the extension, that the GIL was enabled (and why) and the steps the user can take to override it. [0] https://peps.python.org/pep-0703/#py-mod-gil-slot https://peps.python.org/pep-0703/#py-mod-gil-slot
  - miraculixx 3y ago
    Thanks. Glad to know now!
  - miraculixx 3y ago
    Here's what's puzzling me about this: Why then is there even a need for a seperate nogil build? If it is like this says, wouldn't it be easier to just make the standard build switch to gil or nogil automatically (or honor the users choice). The fact that the SC thinks having two versions suggests there is more complexity involved than this section of the PEP leads readers to believe.