5 ms·

Holy shit, actual multithreading. Never thought I'd see the day, good on the Python team for making such great improvements.

skitter 3y ago
Eh, you still can't pass Python objects between them. This just means that if you run multiple instances of Python, you can do so in the same process. If you use the C api.
jjgreen 3y ago
Steady now ... The features of Per-Interpreter GIL are - for now - only available using C-API, so there's no direct interface for Python developers. Such interface is expected to come with PEP 554, which - if accepted - is supposed to land in Python 3.13, until then we will have to hack our way to the sub-interpreter implementation. https://martinheinz.dev/blog/97 https://martinheinz.dev/blog/97
- takeda 3y ago
  If gets approved https://peps.python.org/pep-0703/ https://peps.python.org/pep-0703/ would be much better. Yes, it might require code and package changes, but if people see that having python without GIL is an option if some less used features aren't used they likely would jump on it.
qprofyeh 3y ago
Ergonomic multi-core multithreading is already solved, for me at least, with ProcessPoolExecutor.map(). I wonder what advantages this direction brings to the table.
- sitkack 3y ago
  You will now be able to, in 3.12 run multiple CPython VMs inside the same process w/o having them share state. Previously CPython VM state was spread over a bunch of global variables so you had one GIL per process embedding Python. For your usecase, eventually that ProcessPoolExecutor.map will be GillFreeThreadPoolExecutor.map and all the cross process serialization shenanigans will go away.
  - ptx 3y ago
    But instead of cross-process serialization we get cross-interpreter serialization, so serialization shenanigans don't go away. There is support in the newest pickle protocol for using a shared buffer to transfer data more efficiently, but that would work in multiprocessing [0] just as well as in subinterpreters (and currently isn't implemented in either one). [0] https://github.com/python/cpython/issues/89467 https://github.com/python/cpython/issues/89467
    - sitkack 3y ago
      I agree in the short term, those problems aren't entirely solved, but it does provide for an eventual serialization free way to communicate between threads. Queues, immutable records, atomic refcounts, a global object heap for shared items. There are lots of way forward here that don't involve a full SERDES round trip.
- Spivak 3y ago
  Not needing multiple processes and not having to use slow IPC with serialization for them to talk to one another.
- jtkasdfw 3y ago
  That's not multithreading > is already solved Has been for decades outside python

skitter 3y ago

Eh, you still can't pass Python objects between them. This just means that if you run multiple instances of Python, you can do so in the same process. If you use the C api.

jjgreen 3y ago

Steady now ... The features of Per-Interpreter GIL are - for now - only available using C-API, so there's no direct interface for Python developers. Such interface is expected to come with PEP 554, which - if accepted - is supposed to land in Python 3.13, until then we will have to hack our way to the sub-interpreter implementation. https://martinheinz.dev/blog/97 https://martinheinz.dev/blog/97

takeda 3y ago
If gets approved https://peps.python.org/pep-0703/ https://peps.python.org/pep-0703/ would be much better. Yes, it might require code and package changes, but if people see that having python without GIL is an option if some less used features aren't used they likely would jump on it.

qprofyeh 3y ago

Ergonomic multi-core multithreading is already solved, for me at least, with ProcessPoolExecutor.map(). I wonder what advantages this direction brings to the table.

sitkack 3y ago
You will now be able to, in 3.12 run multiple CPython VMs inside the same process w/o having them share state. Previously CPython VM state was spread over a bunch of global variables so you had one GIL per process embedding Python. For your usecase, eventually that ProcessPoolExecutor.map will be GillFreeThreadPoolExecutor.map and all the cross process serialization shenanigans will go away.
- ptx 3y ago
  But instead of cross-process serialization we get cross-interpreter serialization, so serialization shenanigans don't go away. There is support in the newest pickle protocol for using a shared buffer to transfer data more efficiently, but that would work in multiprocessing [0] just as well as in subinterpreters (and currently isn't implemented in either one). [0] https://github.com/python/cpython/issues/89467 https://github.com/python/cpython/issues/89467
  - sitkack 3y ago
    I agree in the short term, those problems aren't entirely solved, but it does provide for an eventual serialization free way to communicate between threads. Queues, immutable records, atomic refcounts, a global object heap for shared items. There are lots of way forward here that don't involve a full SERDES round trip.
Spivak 3y ago
Not needing multiple processes and not having to use slow IPC with serialization for them to talk to one another.
jtkasdfw 3y ago
That's not multithreading > is already solved Has been for decades outside python