5 ms·

PEP 684 was accepted – Per-interpreter GIL in Python 3.12

75 points by bratao 4y ago

roflyear 4y ago
This is really cool and I'm excited about learning this new concurrency model, which isn't something I've ever said before or expected to say. Most languages I used, python included, haven't done concurrency in a way that added enough without taking so much away where I have been inclined to use it by default. Maybe this is a change from that for python.
jeffreygoesto 4y ago
Nudging closer to the Beam, eh?
nas 4y ago
This is great and was a huge effort by Eric Snow and collaborators to make it happen. To be clear, this is not "PEP 703 – Making the Global Interpreter Lock Optional in CPython", by Sam Gross. PEP 703 is more ambitious and add quite a bit of complexity to the CPython implementation. PEP 684 mostly cleans things up and encapsulates global state better. The most distruptive change is to allow immortal objects. I hope PEP 703 can go in too, even with the downsides (complexity, bit slower single threaded performance).
- westurner 4y ago
  "PEP 703 – Making the Global Interpreter Lock Optional in CPython" (2023) https://github.com/python/peps/blob/main/pep-0703.rst https://github.com/python/peps/blob/main/pep-0703.rst https://peps.python.org/pep-0703/ https://peps.python.org/pep-0703/ colesbury/nogil https://github.com/colesbury/nogil https://github.com/colesbury/nogil : docker run -it nogil/python docker run -it nogil/python-cuda
- viraptor 4y ago
  Are the immortal objects different than what happens with gc.freeze() ?
  - SethTro 4y ago
    Moderately different; Particularly with respect to how to performance. If I read this correctly immortal objects stop changing refcount which prevents cache line invalidation while `gc.freeze()` just stops cleaning up objects with zero refcount.
    - viraptor 4y ago
      Oh, that would be amazing for quite a few things in web apps if exposed properly. Love it.
fulafel 4y ago
So after this and some follow-on work, you can switch from multiprocess message passing parallelism to same-process message passing multi-interpreter parallelism? Which would decrease message passing overhead at cost of crash/memory safety isolation.
- bratao 4y ago
  Yes, that what I´m excited about. A ThreadPoolExecutor like implementation that effectively use multiple CPUs.
  - fulafel 4y ago
    ThreadPoolExecutor seems to be a concept from Java land which is pretty different, you can share objects between Java threads.
    - fulafel 4y ago
      Replying to myself, there's also a identically named class in Python nowadays. But the point remains, you won't get that type of shared-memory parallel multithreading from this. (Which is good I think as I'm a fan of Erlang style message passing parallelism)
dragonwriter 4y ago
Now we just need PEP 554 so this can actually be used from Python code.
- jwilk 4y ago
  https://peps.python.org/pep-0554/ https://peps.python.org/pep-0554/ "Multiple Interpreters in the Stdlib"

75 points by bratao 4y ago

This is really cool and I'm excited about learning this new concurrency model, which isn't something I've ever said before or expected to say. Most languages I used, python included, haven't done concurrency in a way that added enough without taking so much away where I have been inclined to use it by default. Maybe this is a change from that for python.

jeffreygoesto 4y ago

Nudging closer to the Beam, eh?

nas 4y ago

This is great and was a huge effort by Eric Snow and collaborators to make it happen. To be clear, this is not "PEP 703 – Making the Global Interpreter Lock Optional in CPython", by Sam Gross. PEP 703 is more ambitious and add quite a bit of complexity to the CPython implementation. PEP 684 mostly cleans things up and encapsulates global state better. The most distruptive change is to allow immortal objects. I hope PEP 703 can go in too, even with the downsides (complexity, bit slower single threaded performance).

westurner 4y ago
"PEP 703 – Making the Global Interpreter Lock Optional in CPython" (2023) https://github.com/python/peps/blob/main/pep-0703.rst https://github.com/python/peps/blob/main/pep-0703.rst https://peps.python.org/pep-0703/ https://peps.python.org/pep-0703/ colesbury/nogil https://github.com/colesbury/nogil https://github.com/colesbury/nogil : docker run -it nogil/python docker run -it nogil/python-cuda
viraptor 4y ago
Are the immortal objects different than what happens with gc.freeze() ?
- SethTro 4y ago
  Moderately different; Particularly with respect to how to performance. If I read this correctly immortal objects stop changing refcount which prevents cache line invalidation while `gc.freeze()` just stops cleaning up objects with zero refcount.
  - viraptor 4y ago
    Oh, that would be amazing for quite a few things in web apps if exposed properly. Love it.

fulafel 4y ago

So after this and some follow-on work, you can switch from multiprocess message passing parallelism to same-process message passing multi-interpreter parallelism? Which would decrease message passing overhead at cost of crash/memory safety isolation.

bratao 4y ago
Yes, that what I´m excited about. A ThreadPoolExecutor like implementation that effectively use multiple CPUs.
- fulafel 4y ago
  ThreadPoolExecutor seems to be a concept from Java land which is pretty different, you can share objects between Java threads.
  - fulafel 4y ago
    Replying to myself, there's also a identically named class in Python nowadays. But the point remains, you won't get that type of shared-memory parallel multithreading from this. (Which is good I think as I'm a fan of Erlang style message passing parallelism)

dragonwriter 4y ago

Now we just need PEP 554 so this can actually be used from Python code.

jwilk 4y ago
https://peps.python.org/pep-0554/ https://peps.python.org/pep-0554/ "Multiple Interpreters in the Stdlib"