9 ms·

Can someone give a good argument of why subinterpreters are an interesting or useful solution to concurrency in Python? It seems like all the drawbacks of multi

Can someone give a good argument of why subinterpreters are an interesting or useful solution to concurrency in Python? It seems like all the drawbacks of multiprocessing (high memory use, slow serialized object copying) with little benefit and higher complexity for the user. The nogil effort seems like such a better solution, that even if it breaks the C interface, subinterpreters aren't worth considering.

bratao 3y ago
I understand that sub-interpreter can perform better on object copying as it is sharing the same memory. But yeah, nogil looks like the correct way.
samus 3y ago
Easy interop with C is one of the core selling points of Python. It's not just about performance - it's about Python being able to be a glue language that can interface with any legacy or system library written in C or being accessible via FFI. A LOT of systems exist that take advantage of this, and changes to the C interface impact all of these. Unless the porting story is thought out extraordinary well, it would be another disaster like Python 2->3. Since this affects C code, it won't be anything but tricky.
- gyrovagueGeist 3y ago
  From reading the nogil repo and related PEP, C ABI breaking does not seem to be the worst problem. An updated Cython, Swig, etc seems like it would be enough in most cases to get something running. More extreme cases might only have ~dozen LoC changes to replace the GIL Ensure/Release patterns. The hidden really hard problem is that extension modules may have been written relying on GIL behavior for thread safety... these may be undocumented and unmaintained. Even so I hope the community decides it is worth it. A glue language with actual MT support would be much more useful.
  - samus 3y ago
    Indeed, thanks for clarifying that. I actually had the extensions in mind when I wrote my comment, but got somehow distracted by the C interop aspect. Extensions indeed have a more intimate relation with the interpreter and (directly or indirectly) rely on the GIL and its associated semantics.
celeritascelery 3y ago
> Can someone give a good argument of why subinterpreters are an interesting or useful solution to concurrency in Python? I will give it a shot. Subinterpreters are better than multiple processes because: - they have significantly less memory overhead - they can move objects much faster between subinterpreters because they don’t need to serialize through a format like JSON - since they are all in the same process you can implement things like atomics or channels easily. Subinterpreter are better then no-Gil because: - they make the code easier to reason about and debug relative to raw multi-theading - they don’t negatively impact single threaded (basically all existing) python code performance - they don’t require any changes to the C interface, preventing a fractured ecosystem - they can’t have data races
- Spivak 3y ago
  > - they don’t negatively impact single threaded (basically all existing) python code performance I think this deserves an extra callout because even your multi-threaded Python programs are effectively single threaded and benefiting from the performance gain.
- deleted 3y ago
  [deleted]
- spacechild1 3y ago
  > - they can move objects much faster between subinterpreters because they don’t need to serialize through a format like JSON Why do you think that you would need to serialize to JSON? Pipes and sockets can deal with binary data just fine. With shared memory, there wouldn't be any difference at all. > - since they are all in the same process you can implement things like atomics or channels easily. This is also possible with shared memory. AFAICT the advantage of subinterpreters over subprocesses are: - lower memory overhead - faster creation/destruction time - ability to share global data (with subprocesses the data would either need to be duplicated or live in shared memory)
  - celeritascelery 3y ago
    Sure, you could do something like that. But a shared memory segment python with a stable binary object format doesn't exist (and isn't even being worked on). Comparing the proposed PEP 554 solution to a non-existent theoretical solution isn't very useful. But you do bring up some good points for ways you could achieve similar goals without the need to make the interpreters thread safe.
nhumrich 3y ago
Sub interpreters are still faster than processes. With them you could do continuation message passing which other modern languages, such as go, use for multi-threading. Also, for cases such as web apps or data science training, you don't need to share memory between threads, and a sub interpreter uses a lot less resources than a full python process. > Even if it breaks the c interface Than most of your python packages wouldn't work. A python that isn't backwards compatible? Ya, that has been tried once before, and was a disaster. If you want a non-backwards compatible gil-less python, it already exists. You can find versions of nogil online.
globular-toast 3y ago
Minor nitpick: Python can perfectly well do concurrency. What you mean is parallel execution (multi-threading).
- gyrovagueGeist 3y ago
  Point taken! Thanks, I always mess this up when talking colloquially.
bsder 3y ago
Pure uninformed speculation follows ... I suspect sub-interpreters are a punt and a feint. My guess is that there will likely be exactly 2 sub-interpreters in most Python code. One which talks to an old C API with a GIL and one which talks to a new C API without a GIL. It's going to be a lot easier to manage handing objects between two Python sub-interpreters than to manage handing objects between two incompatible ABIs.
- kzrdude 3y ago
  Here's the recent subinterpreters talk, it has information about it https://www.youtube.com/watch?v=3ywZjnjeAO4 https://www.youtube.com/watch?v=3ywZjnjeAO4

bratao 3y ago

I understand that sub-interpreter can perform better on object copying as it is sharing the same memory. But yeah, nogil looks like the correct way.

samus 3y ago

Easy interop with C is one of the core selling points of Python. It's not just about performance - it's about Python being able to be a glue language that can interface with any legacy or system library written in C or being accessible via FFI. A LOT of systems exist that take advantage of this, and changes to the C interface impact all of these. Unless the porting story is thought out extraordinary well, it would be another disaster like Python 2->3. Since this affects C code, it won't be anything but tricky.

gyrovagueGeist 3y ago
From reading the nogil repo and related PEP, C ABI breaking does not seem to be the worst problem. An updated Cython, Swig, etc seems like it would be enough in most cases to get something running. More extreme cases might only have ~dozen LoC changes to replace the GIL Ensure/Release patterns. The hidden really hard problem is that extension modules may have been written relying on GIL behavior for thread safety... these may be undocumented and unmaintained. Even so I hope the community decides it is worth it. A glue language with actual MT support would be much more useful.
- samus 3y ago
  Indeed, thanks for clarifying that. I actually had the extensions in mind when I wrote my comment, but got somehow distracted by the C interop aspect. Extensions indeed have a more intimate relation with the interpreter and (directly or indirectly) rely on the GIL and its associated semantics.

celeritascelery 3y ago

> Can someone give a good argument of why subinterpreters are an interesting or useful solution to concurrency in Python? I will give it a shot. Subinterpreters are better than multiple processes because: - they have significantly less memory overhead - they can move objects much faster between subinterpreters because they don’t need to serialize through a format like JSON - since they are all in the same process you can implement things like atomics or channels easily. Subinterpreter are better then no-Gil because: - they make the code easier to reason about and debug relative to raw multi-theading - they don’t negatively impact single threaded (basically all existing) python code performance - they don’t require any changes to the C interface, preventing a fractured ecosystem - they can’t have data races

Spivak 3y ago
> - they don’t negatively impact single threaded (basically all existing) python code performance I think this deserves an extra callout because even your multi-threaded Python programs are effectively single threaded and benefiting from the performance gain.
deleted 3y ago
[deleted]
spacechild1 3y ago
> - they can move objects much faster between subinterpreters because they don’t need to serialize through a format like JSON Why do you think that you would need to serialize to JSON? Pipes and sockets can deal with binary data just fine. With shared memory, there wouldn't be any difference at all. > - since they are all in the same process you can implement things like atomics or channels easily. This is also possible with shared memory. AFAICT the advantage of subinterpreters over subprocesses are: - lower memory overhead - faster creation/destruction time - ability to share global data (with subprocesses the data would either need to be duplicated or live in shared memory)
- celeritascelery 3y ago
  Sure, you could do something like that. But a shared memory segment python with a stable binary object format doesn't exist (and isn't even being worked on). Comparing the proposed PEP 554 solution to a non-existent theoretical solution isn't very useful. But you do bring up some good points for ways you could achieve similar goals without the need to make the interpreters thread safe.

nhumrich 3y ago

Sub interpreters are still faster than processes. With them you could do continuation message passing which other modern languages, such as go, use for multi-threading. Also, for cases such as web apps or data science training, you don't need to share memory between threads, and a sub interpreter uses a lot less resources than a full python process. > Even if it breaks the c interface Than most of your python packages wouldn't work. A python that isn't backwards compatible? Ya, that has been tried once before, and was a disaster. If you want a non-backwards compatible gil-less python, it already exists. You can find versions of nogil online.

globular-toast 3y ago

Minor nitpick: Python can perfectly well do concurrency. What you mean is parallel execution (multi-threading).

gyrovagueGeist 3y ago
Point taken! Thanks, I always mess this up when talking colloquially.

bsder 3y ago

Pure uninformed speculation follows ... I suspect sub-interpreters are a punt and a feint. My guess is that there will likely be exactly 2 sub-interpreters in most Python code. One which talks to an old C API with a GIL and one which talks to a new C API without a GIL. It's going to be a lot easier to manage handing objects between two Python sub-interpreters than to manage handing objects between two incompatible ABIs.

kzrdude 3y ago
Here's the recent subinterpreters talk, it has information about it https://www.youtube.com/watch?v=3ywZjnjeAO4 https://www.youtube.com/watch?v=3ywZjnjeAO4