5 ms·

The article is well written, and a good history of the all ordeal. But please note it gives more weight to the "against the GIL" side of the story, and all that

The article is well written, and a good history of the all ordeal. But please note it gives more weight to the "against the GIL" side of the story, and all that can go wrong. It doesn't highlight enough the other side of the coin: - Sam's work is very high quality, and he brought with the no-gil some unrelated perf improvements so that people don't feel like they loose perf too much. - Sam played the open source game perfectly, and is incredibly patient given what he is bringing on the able and how slow and flaccid the steering council reaction was (without the community pushing on it, it would still be collecting dust). - Sub interpreters have yet to demonstrate any usefulness at all in Python. In fact, any serious metrics at all. This is the first attempt to be that well defined, and measured. - The community feedback shown a great interest in this particular project. - The steering council did conclude "We intend to accept PEP 703, although we’re still working on the acceptance details." I'm not a no-gil enthusiast. I would be fine with have it never be removed, and I think we should try sub-interpreter first. But what's fair is fair.

TylerE 3y ago
What I’d liked to see is the performance patches without the no-Gil part.
- Znafon 3y ago
  The performance patches have already been merged and there is ongoing working in the Faster CPython project.
- formerly_proven 3y ago
  You're already using a bunch of those in recent Python versions.
Znafon 3y ago
> I would be fine with have it never be removed, and I think we should try sub-interpreter first. There is a lot of work going on with sub-interpreters and the per-interpreter GIL is shipped in Python 3.12. The results are very impressive: https://lwn.net/SubscriberLink/941090/8bcb029dbf548f26/ https://lwn.net/SubscriberLink/941090/8bcb029dbf548f26/, as good as one could have hoped I think. It seems to me like the work on sub-interpreters will continue in parallel ;) to the work on free-threading. Sub-interpreters and no-GIL have different use-cases though.
- ed_blackburn 3y ago
  I'd be interested in introducing appropriate abstractions (ABI) and in a module for parallelism. Whereby one can swap out different implementations for sub-interpreters, green threads, GIL or even hardware or cloud vendor-optimised implementations. Not that dissimilar to Project Loom in Java.
- jnwatson 3y ago
  Once no-GIL is integrated, what is the use case for sub-interpreters?
  - deleted 3y ago
    [deleted]
  - kodablah 3y ago
    I have a use case for sandboxing (without a secure-sandboxing requirement) that it would help solve. But at last read, they discourage/disallow sharing modules across the subinterpreters and reloading all modules in each subinterpreter is not acceptable for my use case.
- scruple 3y ago
  That's very interesting, thank you for the link. I've used Python since the 00s and I don't ever recall having come across sub-interpreters...
  - Znafon 3y ago
    They were always usable using the C-API but not exposed to the Python library (and until 3.12 you were blocked by the shared GIL so there was little point anyway). Now that the work from Eric Snow has been merged, you can use https://pypi.org/project/interpreters-3-12/ https://pypi.org/project/interpreters-3-12/ to create one from Python code.
- BiteCode_dev 3y ago
  For now there is nothing impressive with it. The perfs they show off don't include data sharing, which is very primitive. Which means there is nothing subinterpreters do that can't be done with multiprocessing. They show progress, and it's good. But I would wait until we can see gunicorn spawning WSGI interpreters workers and get similar performances to the setup with regular workers to get enthusiastic.
ilyt 3y ago
SO, why would people be against removal of GIL ?
- throw16180339 3y ago
  Here are some possible reasons. They use unmaintained C extensions that won't be updated. They maintain C extensions that would be complex or painful to update. They're concerned about fragmenting the ecosystem into GIL and No GIL. They think that it will make single-threaded Python programs slower. They have no interest in multithreaded Python and don't think the additional complexity is justified.
  - wheelerof4te 3y ago
    > They use unmaintained C extensions that won't be updated. Well, that's on them. > They're concerned about fragmenting the ecosystem into GIL and No GIL. Same way we have fragmentated ecosystem into ASYNC vs SYNC now? /s > They think that it will make single-threaded Python programs slower. Python is already horribly slow. A little bit slower single thread programs shouldn't be a problem. But the inability to run multi-threaded programs effectively in this day and age is very much a problem.
- zzzeek 3y ago
  I haven't fully confirmed this but my understanding is the proposal forks the ABI into two versions. which IIUC means any package with native extensions needs to release 2x as many wheel files (not to mention if I need to have two versions of the interpreter installed to build them, not sure though). seems like a mess
seanp2k2 3y ago
PEP582 - “Python local packages directory” is my personal bugbear when it comes to Python annoyance: https://peps.python.org/pep-0582/ https://peps.python.org/pep-0582/ I believe PDM still supports it. I just hate virtualenvs for builds and deployments and wish Python could just do what JS does. It’s been proven that it can. https://discuss.python.org/t/pep-582-python-local-packages-directory/963/ https://discuss.python.org/t/pep-582-python-local-packages-d... Is the discussion thread. Also frustrating that “exactly one correct way to do something” seems to be one of the justifications thrown around for rejecting this when I’ve never found that to be true in Python.
- pnt12 3y ago
  Yeah, python is great but I wish package management was more standardized and simpler.

TylerE 3y ago

What I’d liked to see is the performance patches without the no-Gil part.

Znafon 3y ago
The performance patches have already been merged and there is ongoing working in the Faster CPython project.
formerly_proven 3y ago
You're already using a bunch of those in recent Python versions.

Znafon 3y ago

> I would be fine with have it never be removed, and I think we should try sub-interpreter first. There is a lot of work going on with sub-interpreters and the per-interpreter GIL is shipped in Python 3.12. The results are very impressive: https://lwn.net/SubscriberLink/941090/8bcb029dbf548f26/ https://lwn.net/SubscriberLink/941090/8bcb029dbf548f26/, as good as one could have hoped I think. It seems to me like the work on sub-interpreters will continue in parallel ;) to the work on free-threading. Sub-interpreters and no-GIL have different use-cases though.

ed_blackburn 3y ago
I'd be interested in introducing appropriate abstractions (ABI) and in a module for parallelism. Whereby one can swap out different implementations for sub-interpreters, green threads, GIL or even hardware or cloud vendor-optimised implementations. Not that dissimilar to Project Loom in Java.
jnwatson 3y ago
Once no-GIL is integrated, what is the use case for sub-interpreters?
- deleted 3y ago
  [deleted]
- kodablah 3y ago
  I have a use case for sandboxing (without a secure-sandboxing requirement) that it would help solve. But at last read, they discourage/disallow sharing modules across the subinterpreters and reloading all modules in each subinterpreter is not acceptable for my use case.
scruple 3y ago
That's very interesting, thank you for the link. I've used Python since the 00s and I don't ever recall having come across sub-interpreters...
- Znafon 3y ago
  They were always usable using the C-API but not exposed to the Python library (and until 3.12 you were blocked by the shared GIL so there was little point anyway). Now that the work from Eric Snow has been merged, you can use https://pypi.org/project/interpreters-3-12/ https://pypi.org/project/interpreters-3-12/ to create one from Python code.
BiteCode_dev 3y ago
For now there is nothing impressive with it. The perfs they show off don't include data sharing, which is very primitive. Which means there is nothing subinterpreters do that can't be done with multiprocessing. They show progress, and it's good. But I would wait until we can see gunicorn spawning WSGI interpreters workers and get similar performances to the setup with regular workers to get enthusiastic.

ilyt 3y ago

SO, why would people be against removal of GIL ?

throw16180339 3y ago
Here are some possible reasons. They use unmaintained C extensions that won't be updated. They maintain C extensions that would be complex or painful to update. They're concerned about fragmenting the ecosystem into GIL and No GIL. They think that it will make single-threaded Python programs slower. They have no interest in multithreaded Python and don't think the additional complexity is justified.
- wheelerof4te 3y ago
  > They use unmaintained C extensions that won't be updated. Well, that's on them. > They're concerned about fragmenting the ecosystem into GIL and No GIL. Same way we have fragmentated ecosystem into ASYNC vs SYNC now? /s > They think that it will make single-threaded Python programs slower. Python is already horribly slow. A little bit slower single thread programs shouldn't be a problem. But the inability to run multi-threaded programs effectively in this day and age is very much a problem.
zzzeek 3y ago
I haven't fully confirmed this but my understanding is the proposal forks the ABI into two versions. which IIUC means any package with native extensions needs to release 2x as many wheel files (not to mention if I need to have two versions of the interpreter installed to build them, not sure though). seems like a mess

seanp2k2 3y ago

PEP582 - “Python local packages directory” is my personal bugbear when it comes to Python annoyance: https://peps.python.org/pep-0582/ https://peps.python.org/pep-0582/ I believe PDM still supports it. I just hate virtualenvs for builds and deployments and wish Python could just do what JS does. It’s been proven that it can. https://discuss.python.org/t/pep-582-python-local-packages-directory/963/ https://discuss.python.org/t/pep-582-python-local-packages-d... Is the discussion thread. Also frustrating that “exactly one correct way to do something” seems to be one of the justifications thrown around for rejecting this when I’ve never found that to be true in Python.

pnt12 3y ago
Yeah, python is great but I wish package management was more standardized and simpler.