4 ms·

As I understand it, as a library author you either do absolutely nothing and your library will be marked as requiring GIL by default. Nothing to do, you keep on

As I understand it, as a library author you either do absolutely nothing and your library will be marked as requiring GIL by default. Nothing to do, you keep on working with that good old GIL and nothing changes. Or you make the extra effort of being thread safe and you can declare your library as not requiring the GIL. Now if a user script mixes your GIL free lib with an older lib that has not been updated, well, too bad for them. Even with your hard work, the code will still operate like before, everything gets the GIL treatment. Normal python devs will need to track down which pesky dependency of their script is causing the GIL slowdown. Kinda sucks but at least nothing breaks. It's a sensitive, opt-in, and safe way forward. Hard to argue against it, really..

miraculixx 3y ago
Where can I read up on this planned way of working? This is not what the PEP says
- LegionMammal978 3y ago
  It's in the "Py_mod_gil Slot" section [0] in the PEP: > In --disable-gil builds, when loading an extension, CPython will check for a new PEP 489-style Py_mod_gil slot. If the slot is set to Py_mod_gil_not_used, then extension loading proceeds as normal. If the slot is not set, the interpreter pauses all threads and enables the GIL before continuing. Additionally, the interpreter will issue a visible warning naming the extension, that the GIL was enabled (and why) and the steps the user can take to override it. [0] https://peps.python.org/pep-0703/#py-mod-gil-slot https://peps.python.org/pep-0703/#py-mod-gil-slot
  - miraculixx 3y ago
    Thanks. Glad to know now!
  - miraculixx 3y ago
    Here's what's puzzling me about this: Why then is there even a need for a seperate nogil build? If it is like this says, wouldn't it be easier to just make the standard build switch to gil or nogil automatically (or honor the users choice). The fact that the SC thinks having two versions suggests there is more complexity involved than this section of the PEP leads readers to believe.
    - lozenge 3y ago
      You have to recompile your C extensions for the nogil build. Consider an application where one thread is calling xs.append from a C extension and another is calling xs.pop and xs[-1] on the same list object xs. In the nogil build these operations need to use a fine-grained lock on xs, and in the gil build these are thread-safe due to each thread holding the GIL when it does these operations. On top of that, some of the list and dictionary operations are available to extensions as C macros to avoid the overhead of a C function call. However, it looks like the nogil build will be able to run in "GIL mode" for maximum compatibility, including switching to GIL mode partway through execution, but I'm expecting this to be slower than running the gil build in GIL mode.
      - Dylan16807 3y ago
        > However, it looks like the nogil build will be able to run in "GIL mode" for maximum compatibility, including switching to GIL mode partway through execution Yes, that's what they were asking about. Why have two versions for a mode switch. Everything you explained before that is irrelevant, I'm afraid. > but I'm expecting this to be slower than running the gil build in GIL mode. That could be the answer to their question, but that's not definitive enough.
        lozenge 3y ago
        1. The "stable ABI" is broken on nogil, the selling point of the stable ABI was "add this C preprocessor flag and your extension will work on all future CPython versions, after paying a speed penalty". This is useful for eg closed source extensions. If the nogil build was the only build available, these extensions would require recompilation. 2. There are a lot of users that want fast Python and a lot of effort was put in to optimise it. I think the core team wouldn't want to release and expect everybody to use a version which regresses in performance, for a feature (nogil) which most won't be able to use due to using C extensions which haven't had nogil-supporting code changes.
        Dylan16807 3y ago

miraculixx 3y ago

Where can I read up on this planned way of working? This is not what the PEP says

LegionMammal978 3y ago
It's in the "Py_mod_gil Slot" section [0] in the PEP: > In --disable-gil builds, when loading an extension, CPython will check for a new PEP 489-style Py_mod_gil slot. If the slot is set to Py_mod_gil_not_used, then extension loading proceeds as normal. If the slot is not set, the interpreter pauses all threads and enables the GIL before continuing. Additionally, the interpreter will issue a visible warning naming the extension, that the GIL was enabled (and why) and the steps the user can take to override it. [0] https://peps.python.org/pep-0703/#py-mod-gil-slot https://peps.python.org/pep-0703/#py-mod-gil-slot
- miraculixx 3y ago
  Thanks. Glad to know now!
- miraculixx 3y ago
  Here's what's puzzling me about this: Why then is there even a need for a seperate nogil build? If it is like this says, wouldn't it be easier to just make the standard build switch to gil or nogil automatically (or honor the users choice). The fact that the SC thinks having two versions suggests there is more complexity involved than this section of the PEP leads readers to believe.
  - lozenge 3y ago
    You have to recompile your C extensions for the nogil build. Consider an application where one thread is calling xs.append from a C extension and another is calling xs.pop and xs[-1] on the same list object xs. In the nogil build these operations need to use a fine-grained lock on xs, and in the gil build these are thread-safe due to each thread holding the GIL when it does these operations. On top of that, some of the list and dictionary operations are available to extensions as C macros to avoid the overhead of a C function call. However, it looks like the nogil build will be able to run in "GIL mode" for maximum compatibility, including switching to GIL mode partway through execution, but I'm expecting this to be slower than running the gil build in GIL mode.
    - Dylan16807 3y ago
      > However, it looks like the nogil build will be able to run in "GIL mode" for maximum compatibility, including switching to GIL mode partway through execution Yes, that's what they were asking about. Why have two versions for a mode switch. Everything you explained before that is irrelevant, I'm afraid. > but I'm expecting this to be slower than running the gil build in GIL mode. That could be the answer to their question, but that's not definitive enough.
      - lozenge 3y ago
        1. The "stable ABI" is broken on nogil, the selling point of the stable ABI was "add this C preprocessor flag and your extension will work on all future CPython versions, after paying a speed penalty". This is useful for eg closed source extensions. If the nogil build was the only build available, these extensions would require recompilation. 2. There are a lot of users that want fast Python and a lot of effort was put in to optimise it. I think the core team wouldn't want to release and expect everybody to use a version which regresses in performance, for a feature (nogil) which most won't be able to use due to using C extensions which haven't had nogil-supporting code changes.
        Dylan16807 3y ago