7 ms·

> existing C-API extensions will minimally need to be rebuilt and may also require other changes. This vastly understates the work involved. Most C extensions

> existing C-API extensions will minimally need to be rebuilt and may also require other changes. This vastly understates the work involved. Most C extensions and embeddings will require major structural changes or even rewrites. These things are everywhere and are a major reason for python's popularity. A typical financial instition, for example, will have a whole bunch of them, with a morass of python code built on top. Many of these companies took ages to transition to python 3 (and many still have pockets that haven't!). For them to remove GIL reliance from their C extensions is a much bigger ask than migrating to 3 was, and their response to being asked to do it is likely to be blunt. The fact that only a small minority of developers involved in python directly use the C API (and fewer understand the consequences of GIL removal for them) means the issue tends to be overlooked in discussions like this, but it's the reason a GILectomy will be worse than 2 to 3. In practice it's likely to end up being either a fork or a mode you have to switch on/off, both of which would be miserable for everyone. The GIL is part of python's success story. It made it easy to write extensions, and the extension ecosystem made the language popular. Every language doesn't have to converge to the same endpoint. Different tools are suited to different jobs. Let it be.

deleted 3y ago
[deleted]
floomk 3y ago
It would be helpful and interesting if you could explain why the impact is so huge
- gary_0 3y ago
  They're talking about C code that blindly relies on implicit concurrency guarantees that are now conditional. Isn't that enough said?
  - mrlyx 3y ago
    Not at all. The GIL was explicitly part of the C-API! No one used the guarantees accidentally, including CPython itself. The GIL was one of the reasons why the C module ecosytem took off.
    - gary_0 3y ago
      That's not the implicitness I'm talking about. While it sounds like loading old C modules will just re-enable the GIL, the problem is that they will never be updated to not rely on the old Python concurrency model. All that C code was written implicitly assuming that certain blocks of code were surrounded by a GIL. It could be a real headache for any hoped-for transition to nogil Python if lots of GIL-reliant C code is floating around where there's little hope of updating it without having to worry about subtle bugs popping up. And even if the conversion was risk-free (which I doubt), many organizations will still not want to dig into their legacy C codebases and make significant changes.
    - lanstin 3y ago
      And the probability of a given piece of C code that was written for a thread safe environment working when called from multiple threads at once is pretty low for anything not 100% purely functional.
      - CJefferson 3y ago
        Honestly, that’s not my biggest worry. My bigger worry is that now Python variables your extension is reading can be changed by another thread in real time. Before if you held the GIL, you knew nothing would change while you poked some Python datastructures.
klyrs 3y ago
> > This will be an opt-in feature by building the interpreter using `--disable-gil`. The PEP is currently targeted at Python 3.13. > In practice it's likely to end up being either a fork or a mode you have to switch on/off, both of which would be miserable for everyone. Yep. That's the plan.
- faitswulff 3y ago
  > Yep. That's the plan. The misery, you mean?
  - klyrs 3y ago
    True Pythonistas love misery, almost as much as we love kvetching. Without the misery, we'd have nothing to talk about. Don't get me started on typing.
  - lyu07282 3y ago
    Have you seen PIP, conda, asyncio from 3.4 to 3.8, the standard library documentation, the great 2 to 3 migration, type "annotations" and mypy, the variable scoping "rules" or the multiprocessing module? We python users are more than ready for whatever pain we have to deal with in the future, we are used to misery we live it every day.
gjulianm 3y ago
I’m not sure that most will require rewrites. First, if the extension isn’t being used in a multithreaded environment, nothing should change. Yes, it isn’t thread safe, but it doesn’t really matter in that context. And given how bad GIL-Python works with threads, I doubt the majority of extensions are written for multithreaded applications. And the ones that are written for multithreaded are probably already releasing the GIL for long running computations, so they should be written with at least a little bit of thread safety in mind. And in the worst case it shouldn’t be too difficult to hack the code to include a GIL-like lock that needs to be held by library users in order to ensure thread safety without really changing the architecture of the extension.
- bjourne 3y ago
  There is no way of knowing whether a Python module is used in multithreaded environments or not. The point is that C extensions that previously worked fine in multithreaded environments probably will not work fine in multithreaded environments with the GIL disabled.
  - schmeii 3y ago
    That is not true. If an extension doesn't advertise that it is thread safe, the GIL get activated at runtime so it will work exactly like it does now.
- CJefferson 3y ago
  In my personal experience almost every large Python program is a little multithreaded. For example, if you need to handle the external programs input, stdout and errout, most of the examples you find by googling use threads for this.
  - norir 3y ago
    I'm sure you know this, but as a general psa: threads are the lowest common denominator way of doing this kind of thing. Where possible, it is nicer to use epoll/kqueue/whatever the windows equivalent is. For a library though I totally get how annoying it would be to interface with the platform event system on each os. Indeed, if there isn't already a python library that can handle subprocesses for you on a single thread then it might be something worth building (I don't use python personally so I don't know the ecosystem).
cpgxiii 3y ago
> Most C extensions and embeddings will require major structural changes or even rewrites. A lot of those important "extensions" are actually C and C++ libraries designed for parallelism in their native use which have been made available in Python via bindings (e.g. Pytorch). The cores of these libraries are fine, only the binding layers (often automatically/semi-automatically generated) may need to be updated. I suspect only the C-API extensions designed from scratch to only be extensions are going to have major problems with a no-GIL world. > Every language doesn't have to converge to the same endpoint. Different tools are suited to different jobs. People have decided they want to use Python frontends. Plenty of alternative languages could have won out, which would have provided native threading, faster runtimes etc, but they didn't; we are stuck with Python. The continued existence of the GIL is incredibly restrictive on how you can leverage parallelism both in Python itself and in extensions. Only in the simple cases can you make the Python<->extension boundary clean: the second you want to be able to call Python from extension code (e.g. via callbacks or inheritance) the GIL stands in your way.
- 0cf8612b2e1e 3y ago
  For every popular, well-engineered extension, I suspect there are a dozen hacked-together ones which will break the moment the GIL guarantees disappear.
  - cpgxiii 3y ago
    I'm sure there are a lot of hacky extensions out there (particularly hiding in proprietary codebases), but letting the entire future of Python be held hostage to the poor SWE choices of third parties who almost certainly contribute nothing back is not a sustainable path.
    - 0cf8612b2e1e 3y ago
      But it is a programming language. You do not break backwards compatibility lightly. Nobody has ever chosen Python for its runtime performance. Unfortunately, sometimes you have to accept the technical debt cannot be escaped.
bsder 3y ago
> This vastly understates the work involved. Most C extensions and embeddings will require major structural changes or even rewrites. Why? Once you have a threaded Python, you can allocate one thread to run with a GIL and the other threads without it. Old extensions can access one thread with the old GIL API and new extensions can access all the things with the GIL-less API. People who want the performance will rewrite their extension. People who don't, won't.
- tsukikage 3y ago
  That's not quite good enough - people use threads to e.g. permit multiple blocking i/o operations to be in flight at once, which works perfectly well right now. So you're going to want to support multiple GIL threads if you want to be able to support existing code safely. Sadly, you can't just trivially bolt on thread safety to code that was not designed with it in mind. However, if you go the other way and introduce e.g. an @nogil decorator, similar to that seen in some cpython alternatives, people will have a straightforward path to opt in incrementally as they fix and verify the critical parts of their code, while preserving known working behaviour elsewhere without having to throw entire complex systems over the wall at once.
- schmeii 3y ago
  I don't think that is possible to activate the GIL only for one thread, it has to be activated process-wide. But, if you want free-threading but you have to use an old extension that is not GIL-less, then you still can, by garding all access to this extension with a lock and setting the PYTHONGIL variable to 0. But, you will need to compile the extension with --disable-gil or to obtain that from somewhere, if the extension author doesn't provide it, as the ABI of "standard" python and "--disable-gil" python is different. How --disable-gil builds of python and of extensions will be distributed is not yet totally clear. I don't think pypi has already a clear idea of how to handle those extension, for instance.
jtokoph 3y ago
Could there be a shim between the two that emulates the original behavior?
- schmeii 3y ago
  There already is such a mechanism in the pep. Basically there will be two build mode. The standard build that you get at python.org will work exactly as now (so with a Gil). The disable-gil build (which will most likely be available through conda or similar) will have the option to run either with the GIL or without. It will automatically detect extension that are not compatible and switch to Gil mode but you can override this with an environment variable.
dataflow 3y ago
> A typical financial institution Could you elaborate on how the Python ecosystem will suffer if these financial institutions get left behind? What sorts of contributions come from these users in particular?
- paulddraper 3y ago
  Python will lose marketshare/mindshare. StackOverflow answers, side projects, fanboys
rmbyrro 3y ago
Did you read the part that you'll have to opt in and build the interpreter with --disable-gil? If it doesn't work for you, use the GIL enabled. What's the problem of letting others disable it?
- wokwokwok 3y ago
  Come now, how many people compile python to use it? How many people will be surprised / upset that they have to do that once `--disable-gil` is mainlined into the default builds on python.org? How long before 'build it yourself and use --disable-gil' becomes 'we've enabled --disable-gil by default for Great Victory for Most Users'? I think it's very very naive to think that the flag will remain an obscure 'use if you want to build python yourself' if the PEP is successful. > The global interpreter lock will remain the default for CPython builds and python.org downloads. Is a joke. It means, for now. For the work in this PEP. I mean, you've literally got the folk from numpy saying: > Coordinating on APIs and design decisions to control parallelism is still a major amount of work, and one of the harder challenges across the PyData ecosystem. It would have looked a lot different (better, easier) without a GIL. That isn't 'for a few people who are vaguely interested in obscure technical stuff'; this is the entire python data ecosystem, saying "this will be the default in the future because the GIL is a massive pain in the ass". > If it doesn't work for you, use the GIL enabled. Sure, it can be opt-out; but you're fooling yourself if you think this is going to be 'opt-in'. It will be the default once it's completed, sooner or later, imo. ...and when the 'default' build of python breaks c-extensions, that is a breaking change, even if technically you can build your own copy of python with different compile flags.
  - AlphaSite 3y ago
    It will probably be something like: brew install python@3.12-mp apt install python3.12-mp Etc. I don’t think anyone actually needs to compile it themselves, unless they really want to.
    - wokwokwok 3y ago
      Long term? I doubt it. Who’s going build those packages? Two major parallel incompatible implementations of Python, maintained at the same time, duplicating all efforts at testing, dev, build, release train, etc. while both are maintained. Does it sound familiar? Even ignoring the obvious parallels to Python 3000, there’s a limited amount of time people will be bothered maintaining both.
dzsekijo 3y ago
However, these days AI assisted tools can be involved in the task.
js2 3y ago
> Most C extensions and embeddings will require major structural changes or even rewrites. Sam claims otherwise: "Most C API extensions don’t require any changes, and for those that do require changes, the changes are small. For example, for “nogil” Python I’m providing binary wheels for ~35 extensions that are slow or difficult to build from source and only about seven projects required code changes (PyTorch, pybind11, Cython, numpy, scikit-learn, viztracer, pyo3). For four of those projects, the code changes have already been contributed and adopted upstream. For comparison, many of those same projects also frequently require changes to support minor releases of CPython." https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional/22606/44 https://discuss.python.org/t/pep-703-making-the-global-inter... In the following talk, he shows the scikit-learn changes that were needed: https://youtu.be/9OOJcTp8dqE?t=1958 https://youtu.be/9OOJcTp8dqE?t=1958 I'd also like to point out that the GIL does not inherently make C extensions thread safe. It only protects the interpreter itself. For example, calling in to the C API can already release the GIL as I explained in reply to /u/bjourne here: https://news.ycombinator.com/item?id=36649769 https://news.ycombinator.com/item?id=36649769
semi 3y ago
would a fork be so bad? maybe I'm off but I feel like most people have no real need for GILless and would leave it off if they were even aware it existed. Those who would need it the most are the most equipped to handle the cost of adoption.

deleted 3y ago

[deleted]

floomk 3y ago

It would be helpful and interesting if you could explain why the impact is so huge

gary_0 3y ago
They're talking about C code that blindly relies on implicit concurrency guarantees that are now conditional. Isn't that enough said?
- mrlyx 3y ago
  Not at all. The GIL was explicitly part of the C-API! No one used the guarantees accidentally, including CPython itself. The GIL was one of the reasons why the C module ecosytem took off.
  - gary_0 3y ago
    That's not the implicitness I'm talking about. While it sounds like loading old C modules will just re-enable the GIL, the problem is that they will never be updated to not rely on the old Python concurrency model. All that C code was written implicitly assuming that certain blocks of code were surrounded by a GIL. It could be a real headache for any hoped-for transition to nogil Python if lots of GIL-reliant C code is floating around where there's little hope of updating it without having to worry about subtle bugs popping up. And even if the conversion was risk-free (which I doubt), many organizations will still not want to dig into their legacy C codebases and make significant changes.
  - lanstin 3y ago
    And the probability of a given piece of C code that was written for a thread safe environment working when called from multiple threads at once is pretty low for anything not 100% purely functional.
    - CJefferson 3y ago
      Honestly, that’s not my biggest worry. My bigger worry is that now Python variables your extension is reading can be changed by another thread in real time. Before if you held the GIL, you knew nothing would change while you poked some Python datastructures.

klyrs 3y ago

> > This will be an opt-in feature by building the interpreter using `--disable-gil`. The PEP is currently targeted at Python 3.13. > In practice it's likely to end up being either a fork or a mode you have to switch on/off, both of which would be miserable for everyone. Yep. That's the plan.

faitswulff 3y ago
> Yep. That's the plan. The misery, you mean?
- klyrs 3y ago
  True Pythonistas love misery, almost as much as we love kvetching. Without the misery, we'd have nothing to talk about. Don't get me started on typing.
- lyu07282 3y ago
  Have you seen PIP, conda, asyncio from 3.4 to 3.8, the standard library documentation, the great 2 to 3 migration, type "annotations" and mypy, the variable scoping "rules" or the multiprocessing module? We python users are more than ready for whatever pain we have to deal with in the future, we are used to misery we live it every day.

gjulianm 3y ago

I’m not sure that most will require rewrites. First, if the extension isn’t being used in a multithreaded environment, nothing should change. Yes, it isn’t thread safe, but it doesn’t really matter in that context. And given how bad GIL-Python works with threads, I doubt the majority of extensions are written for multithreaded applications. And the ones that are written for multithreaded are probably already releasing the GIL for long running computations, so they should be written with at least a little bit of thread safety in mind. And in the worst case it shouldn’t be too difficult to hack the code to include a GIL-like lock that needs to be held by library users in order to ensure thread safety without really changing the architecture of the extension.

bjourne 3y ago
There is no way of knowing whether a Python module is used in multithreaded environments or not. The point is that C extensions that previously worked fine in multithreaded environments probably will not work fine in multithreaded environments with the GIL disabled.
- schmeii 3y ago
  That is not true. If an extension doesn't advertise that it is thread safe, the GIL get activated at runtime so it will work exactly like it does now.
CJefferson 3y ago
In my personal experience almost every large Python program is a little multithreaded. For example, if you need to handle the external programs input, stdout and errout, most of the examples you find by googling use threads for this.
- norir 3y ago
  I'm sure you know this, but as a general psa: threads are the lowest common denominator way of doing this kind of thing. Where possible, it is nicer to use epoll/kqueue/whatever the windows equivalent is. For a library though I totally get how annoying it would be to interface with the platform event system on each os. Indeed, if there isn't already a python library that can handle subprocesses for you on a single thread then it might be something worth building (I don't use python personally so I don't know the ecosystem).

cpgxiii 3y ago

> Most C extensions and embeddings will require major structural changes or even rewrites. A lot of those important "extensions" are actually C and C++ libraries designed for parallelism in their native use which have been made available in Python via bindings (e.g. Pytorch). The cores of these libraries are fine, only the binding layers (often automatically/semi-automatically generated) may need to be updated. I suspect only the C-API extensions designed from scratch to only be extensions are going to have major problems with a no-GIL world. > Every language doesn't have to converge to the same endpoint. Different tools are suited to different jobs. People have decided they want to use Python frontends. Plenty of alternative languages could have won out, which would have provided native threading, faster runtimes etc, but they didn't; we are stuck with Python. The continued existence of the GIL is incredibly restrictive on how you can leverage parallelism both in Python itself and in extensions. Only in the simple cases can you make the Python<->extension boundary clean: the second you want to be able to call Python from extension code (e.g. via callbacks or inheritance) the GIL stands in your way.

0cf8612b2e1e 3y ago
For every popular, well-engineered extension, I suspect there are a dozen hacked-together ones which will break the moment the GIL guarantees disappear.
- cpgxiii 3y ago
  I'm sure there are a lot of hacky extensions out there (particularly hiding in proprietary codebases), but letting the entire future of Python be held hostage to the poor SWE choices of third parties who almost certainly contribute nothing back is not a sustainable path.
  - 0cf8612b2e1e 3y ago
    But it is a programming language. You do not break backwards compatibility lightly. Nobody has ever chosen Python for its runtime performance. Unfortunately, sometimes you have to accept the technical debt cannot be escaped.

bsder 3y ago

> This vastly understates the work involved. Most C extensions and embeddings will require major structural changes or even rewrites. Why? Once you have a threaded Python, you can allocate one thread to run with a GIL and the other threads without it. Old extensions can access one thread with the old GIL API and new extensions can access all the things with the GIL-less API. People who want the performance will rewrite their extension. People who don't, won't.

tsukikage 3y ago
That's not quite good enough - people use threads to e.g. permit multiple blocking i/o operations to be in flight at once, which works perfectly well right now. So you're going to want to support multiple GIL threads if you want to be able to support existing code safely. Sadly, you can't just trivially bolt on thread safety to code that was not designed with it in mind. However, if you go the other way and introduce e.g. an @nogil decorator, similar to that seen in some cpython alternatives, people will have a straightforward path to opt in incrementally as they fix and verify the critical parts of their code, while preserving known working behaviour elsewhere without having to throw entire complex systems over the wall at once.
schmeii 3y ago
I don't think that is possible to activate the GIL only for one thread, it has to be activated process-wide. But, if you want free-threading but you have to use an old extension that is not GIL-less, then you still can, by garding all access to this extension with a lock and setting the PYTHONGIL variable to 0. But, you will need to compile the extension with --disable-gil or to obtain that from somewhere, if the extension author doesn't provide it, as the ABI of "standard" python and "--disable-gil" python is different. How --disable-gil builds of python and of extensions will be distributed is not yet totally clear. I don't think pypi has already a clear idea of how to handle those extension, for instance.

jtokoph 3y ago

Could there be a shim between the two that emulates the original behavior?

schmeii 3y ago
There already is such a mechanism in the pep. Basically there will be two build mode. The standard build that you get at python.org will work exactly as now (so with a Gil). The disable-gil build (which will most likely be available through conda or similar) will have the option to run either with the GIL or without. It will automatically detect extension that are not compatible and switch to Gil mode but you can override this with an environment variable.

dataflow 3y ago

> A typical financial institution Could you elaborate on how the Python ecosystem will suffer if these financial institutions get left behind? What sorts of contributions come from these users in particular?

paulddraper 3y ago
Python will lose marketshare/mindshare. StackOverflow answers, side projects, fanboys

rmbyrro 3y ago

Did you read the part that you'll have to opt in and build the interpreter with --disable-gil? If it doesn't work for you, use the GIL enabled. What's the problem of letting others disable it?

wokwokwok 3y ago
Come now, how many people compile python to use it? How many people will be surprised / upset that they have to do that once `--disable-gil` is mainlined into the default builds on python.org? How long before 'build it yourself and use --disable-gil' becomes 'we've enabled --disable-gil by default for Great Victory for Most Users'? I think it's very very naive to think that the flag will remain an obscure 'use if you want to build python yourself' if the PEP is successful. > The global interpreter lock will remain the default for CPython builds and python.org downloads. Is a joke. It means, for now. For the work in this PEP. I mean, you've literally got the folk from numpy saying: > Coordinating on APIs and design decisions to control parallelism is still a major amount of work, and one of the harder challenges across the PyData ecosystem. It would have looked a lot different (better, easier) without a GIL. That isn't 'for a few people who are vaguely interested in obscure technical stuff'; this is the entire python data ecosystem, saying "this will be the default in the future because the GIL is a massive pain in the ass". > If it doesn't work for you, use the GIL enabled. Sure, it can be opt-out; but you're fooling yourself if you think this is going to be 'opt-in'. It will be the default once it's completed, sooner or later, imo. ...and when the 'default' build of python breaks c-extensions, that is a breaking change, even if technically you can build your own copy of python with different compile flags.
- AlphaSite 3y ago
  It will probably be something like: brew install python@3.12-mp apt install python3.12-mp Etc. I don’t think anyone actually needs to compile it themselves, unless they really want to.
  - wokwokwok 3y ago
    Long term? I doubt it. Who’s going build those packages? Two major parallel incompatible implementations of Python, maintained at the same time, duplicating all efforts at testing, dev, build, release train, etc. while both are maintained. Does it sound familiar? Even ignoring the obvious parallels to Python 3000, there’s a limited amount of time people will be bothered maintaining both.

dzsekijo 3y ago

However, these days AI assisted tools can be involved in the task.

js2 3y ago

> Most C extensions and embeddings will require major structural changes or even rewrites. Sam claims otherwise: "Most C API extensions don’t require any changes, and for those that do require changes, the changes are small. For example, for “nogil” Python I’m providing binary wheels for ~35 extensions that are slow or difficult to build from source and only about seven projects required code changes (PyTorch, pybind11, Cython, numpy, scikit-learn, viztracer, pyo3). For four of those projects, the code changes have already been contributed and adopted upstream. For comparison, many of those same projects also frequently require changes to support minor releases of CPython." https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional/22606/44 https://discuss.python.org/t/pep-703-making-the-global-inter... In the following talk, he shows the scikit-learn changes that were needed: https://youtu.be/9OOJcTp8dqE?t=1958 https://youtu.be/9OOJcTp8dqE?t=1958 I'd also like to point out that the GIL does not inherently make C extensions thread safe. It only protects the interpreter itself. For example, calling in to the C API can already release the GIL as I explained in reply to /u/bjourne here: https://news.ycombinator.com/item?id=36649769 https://news.ycombinator.com/item?id=36649769

semi 3y ago

would a fork be so bad? maybe I'm off but I feel like most people have no real need for GILless and would leave it off if they were even aware it existed. Those who would need it the most are the most equipped to handle the cost of adoption.