4 ms·

> Most C extensions and embeddings will require major structural changes or even rewrites. A lot of those important "extensions" are actually C and C++ librari

> Most C extensions and embeddings will require major structural changes or even rewrites. A lot of those important "extensions" are actually C and C++ libraries designed for parallelism in their native use which have been made available in Python via bindings (e.g. Pytorch). The cores of these libraries are fine, only the binding layers (often automatically/semi-automatically generated) may need to be updated. I suspect only the C-API extensions designed from scratch to only be extensions are going to have major problems with a no-GIL world. > Every language doesn't have to converge to the same endpoint. Different tools are suited to different jobs. People have decided they want to use Python frontends. Plenty of alternative languages could have won out, which would have provided native threading, faster runtimes etc, but they didn't; we are stuck with Python. The continued existence of the GIL is incredibly restrictive on how you can leverage parallelism both in Python itself and in extensions. Only in the simple cases can you make the Python<->extension boundary clean: the second you want to be able to call Python from extension code (e.g. via callbacks or inheritance) the GIL stands in your way.

0cf8612b2e1e 3y ago
For every popular, well-engineered extension, I suspect there are a dozen hacked-together ones which will break the moment the GIL guarantees disappear.
- cpgxiii 3y ago
  I'm sure there are a lot of hacky extensions out there (particularly hiding in proprietary codebases), but letting the entire future of Python be held hostage to the poor SWE choices of third parties who almost certainly contribute nothing back is not a sustainable path.
  - 0cf8612b2e1e 3y ago
    But it is a programming language. You do not break backwards compatibility lightly. Nobody has ever chosen Python for its runtime performance. Unfortunately, sometimes you have to accept the technical debt cannot be escaped.
    - TillE 3y ago
      Stuff breaks in software all the time, it's really not a huge deal. You either update your code or stick to an older version.
    - cpgxiii 3y ago
      > Nobody has ever chosen Python for its runtime performance. No, they choose it for the ease of using its performant extensions. And those extensions are fundamentally limited in performance by the existence of the GIL. And the authors of those extensions (and their employers) are behind the work to get rid of it. There are three groups here: 1. "Pure" Python users, whose code makes little/no use of extensions. GIL removal is an immediate win for these uses. 2. "Good" extension users/authors, whose code will support and benefit from no-GIL. GIL removal is an immediate win for these uses. 3. "Bad" extension users/authors, whose code doesn't/can't support no-GIL. GIL removal probably doesn't break existing code, but makes new uses potentially unsafe. Maintaining the technical debt of the GIL indefinitely so that group 3 never needs to address its own technical debt is not a good tradeoff for groups 1 and 2 which actively want to move the language forwards.
      - 0cf8612b2e1e 3y ago
        Don’t break user space. The above arguments could apply to any breaking changes that a host wants to implement. Those who don’t directly consume the change, those who can adapt, and everyone who is fine with the status quo. Python is free to do as it pleases, but this breaking change is going to result in a lot of churn. > GIL removal is an immediate win for these uses. As a minor aside, the GIL free Python version was actually a performance regression, somewhere between 5-10% impact. https://peps.python.org/pep-0703/#performance https://peps.python.org/pep-0703/#performance . So, an immediate loss for nearly everyone.
      - omnicognate 3y ago
        No, this is false. Relying on the GIL isn't bad practice in a python extension. It's something you have no choice but to do. The GIL is the thread safety guarantee given to the extension by the python intepreter. It's the contract you have and you have no choice but to code to it. Removal of the GIL requires an alternative contract representing a finer-grained set of thread-safety guarantees for people to code against. Programmers who coded against the contract that existed rather than somehow divining the future and coding against a different contract that hadn't been invented yet (while also managing to make it compatible with the existing one) weren't doing something wrong. But besides being incorrect, the idea that extensions that break on removal of the GIL are "bad" is irrelevant. The question is what the ecosystem will bear. The people who own the codebases I'm talking about are not pushing for GIL removal. They have large, working codebases they're using in production and the cost of redesigning large chunks of them would be real. GILectomy is being pushed by people with varied motives but they certainly don't speak for everyone and their case isn't going to be made successfully by casting aspersions on those with different priorities.

0cf8612b2e1e 3y ago

For every popular, well-engineered extension, I suspect there are a dozen hacked-together ones which will break the moment the GIL guarantees disappear.

cpgxiii 3y ago
I'm sure there are a lot of hacky extensions out there (particularly hiding in proprietary codebases), but letting the entire future of Python be held hostage to the poor SWE choices of third parties who almost certainly contribute nothing back is not a sustainable path.
- 0cf8612b2e1e 3y ago
  But it is a programming language. You do not break backwards compatibility lightly. Nobody has ever chosen Python for its runtime performance. Unfortunately, sometimes you have to accept the technical debt cannot be escaped.
  - TillE 3y ago
    Stuff breaks in software all the time, it's really not a huge deal. You either update your code or stick to an older version.
  - cpgxiii 3y ago
    > Nobody has ever chosen Python for its runtime performance. No, they choose it for the ease of using its performant extensions. And those extensions are fundamentally limited in performance by the existence of the GIL. And the authors of those extensions (and their employers) are behind the work to get rid of it. There are three groups here: 1. "Pure" Python users, whose code makes little/no use of extensions. GIL removal is an immediate win for these uses. 2. "Good" extension users/authors, whose code will support and benefit from no-GIL. GIL removal is an immediate win for these uses. 3. "Bad" extension users/authors, whose code doesn't/can't support no-GIL. GIL removal probably doesn't break existing code, but makes new uses potentially unsafe. Maintaining the technical debt of the GIL indefinitely so that group 3 never needs to address its own technical debt is not a good tradeoff for groups 1 and 2 which actively want to move the language forwards.
    - 0cf8612b2e1e 3y ago
      Don’t break user space. The above arguments could apply to any breaking changes that a host wants to implement. Those who don’t directly consume the change, those who can adapt, and everyone who is fine with the status quo. Python is free to do as it pleases, but this breaking change is going to result in a lot of churn. > GIL removal is an immediate win for these uses. As a minor aside, the GIL free Python version was actually a performance regression, somewhere between 5-10% impact. https://peps.python.org/pep-0703/#performance https://peps.python.org/pep-0703/#performance . So, an immediate loss for nearly everyone.
    - omnicognate 3y ago
      No, this is false. Relying on the GIL isn't bad practice in a python extension. It's something you have no choice but to do. The GIL is the thread safety guarantee given to the extension by the python intepreter. It's the contract you have and you have no choice but to code to it. Removal of the GIL requires an alternative contract representing a finer-grained set of thread-safety guarantees for people to code against. Programmers who coded against the contract that existed rather than somehow divining the future and coding against a different contract that hadn't been invented yet (while also managing to make it compatible with the existing one) weren't doing something wrong. But besides being incorrect, the idea that extensions that break on removal of the GIL are "bad" is irrelevant. The question is what the ecosystem will bear. The people who own the codebases I'm talking about are not pushing for GIL removal. They have large, working codebases they're using in production and the cost of redesigning large chunks of them would be real. GILectomy is being pushed by people with varied motives but they certainly don't speak for everyone and their case isn't going to be made successfully by casting aspersions on those with different priorities.