4 ms·

Not for pure python code; but there's massive advantages for mixed C(++) and Python: I can now have multiple sub interpreters running concurrently and accessing

by ynik 3y ago

Not for pure python code; but there's massive advantages for mixed C(++) and Python: I can now have multiple sub interpreters running concurrently and accessing the same shared state in a thread-safe C++ library. Previously this required rewriting the whole C++ library to support either pickling (multiplying the total memory consumption by the number of cores), or support allocating everything in shared memory (which means normal C++ types like `std::string` are unusable, need to switch e.g. to boost::interprocess). Now is sufficient to pickle a pointer to a C++ object as an integer, and it'll still be a valid pointer in the other subinterpreter.

ledauphin 3y ago
this is the _first_ place I've seen anyone clearly articulate an actual use case for subinterpreters. Thank you!
veber-alex 3y ago
But why? There is no GIL in C/C++/Rust/Zig/Whatever, just use threads.
- nutate 3y ago
  Let's say you wanted to run some python code that was written and test it against some C/C++/Rust for accuracy of some sort (numerical, lexicographical, etc). In the old way you would have to fire up multiple processes to do that (like OS level processes) but now you can have your multithreaded compiled code running in threads and your multi-GIL'd interpreted code running all in one process and comparing their results in the `main` of your C/C++/Rust. That's a contrived example, but the issue was that a single GIL isn't threadsafe in and of itself. So if you're using these compiled languages as sort of python runners you couldn't multithread python interpreter execution and guarantee the code working. Also as the above comment stated, you could do hacks, but you'd double your memory allocation by needing a python and C/C++/Rust representation for everything that went back and forth.
- ynik 3y ago
  Because in our case, all threads would be running a mixture of Python and C++. Our core data structure is implemented in C++, large (often >10 GB), and can be shared across threads (thread safe, usage is mostly read-only). But we have lots of analysis algorithms accessing that data structure, most implemented in Python. We tried releasing the GIL for every tiny call to C++, but that approach can barely keep two cores busy due to constant fighting over the GIL. (there's no good "inner loop" that could avoid touching the GIL within the loop body) Rewriting most/all of the Python analyses in a different (GIL-free language) is a no-go, the analyses have accumulated over the years and now there's more than a thousand of them. It would consume all our development resources for the next ~5 years. In retrospect I can say that choosing Python for these was major mistake, but it's one that cannot be fixed without a company-killing rewrite :( We actually invested several months of developer time in allocating our core data structure in shared memory, allowing us to parallelize with multiprocessing. But there's still a whole bunch of ancillary data structures written in C++ that are not so easy to put in shared memory, so all analyses touching those are limited to a single process, which by Amdahl's law immediately starting dominating our execution time.
nutate 3y ago
Same w/ Rust and Python, this is really neat because now each thread could have a GIL without doing exactly what you said. The pyO3 commit to allow subinterpreters was merged 21 days ago, so this might "just work" today: https://github.com/PyO3/pyo3/pull/3446 https://github.com/PyO3/pyo3/pull/3446
ddorian43 3y ago
> I can now have multiple sub interpreters running concurrently and accessing the same shared state in a thread-safe C++ library. Can sub-interpreters be used to share a db/http/etc connection pool across multiple processes?
elashri 3y ago
Wouldn't you also gain some overhead reduction too? Multiprocessing will have more overhead for spawning new processes vs signle process that will contain the sup-interprets? or I am missing something? At least in theory this what would happen.
- spacechild1 3y ago
  Threads are certainly faster to create, use less memory and allow for faster context switching.

by ynik 3y ago

ledauphin 3y ago

this is the _first_ place I've seen anyone clearly articulate an actual use case for subinterpreters. Thank you!

veber-alex 3y ago

But why? There is no GIL in C/C++/Rust/Zig/Whatever, just use threads.

nutate 3y ago
Let's say you wanted to run some python code that was written and test it against some C/C++/Rust for accuracy of some sort (numerical, lexicographical, etc). In the old way you would have to fire up multiple processes to do that (like OS level processes) but now you can have your multithreaded compiled code running in threads and your multi-GIL'd interpreted code running all in one process and comparing their results in the `main` of your C/C++/Rust. That's a contrived example, but the issue was that a single GIL isn't threadsafe in and of itself. So if you're using these compiled languages as sort of python runners you couldn't multithread python interpreter execution and guarantee the code working. Also as the above comment stated, you could do hacks, but you'd double your memory allocation by needing a python and C/C++/Rust representation for everything that went back and forth.
ynik 3y ago
Because in our case, all threads would be running a mixture of Python and C++. Our core data structure is implemented in C++, large (often >10 GB), and can be shared across threads (thread safe, usage is mostly read-only). But we have lots of analysis algorithms accessing that data structure, most implemented in Python. We tried releasing the GIL for every tiny call to C++, but that approach can barely keep two cores busy due to constant fighting over the GIL. (there's no good "inner loop" that could avoid touching the GIL within the loop body) Rewriting most/all of the Python analyses in a different (GIL-free language) is a no-go, the analyses have accumulated over the years and now there's more than a thousand of them. It would consume all our development resources for the next ~5 years. In retrospect I can say that choosing Python for these was major mistake, but it's one that cannot be fixed without a company-killing rewrite :( We actually invested several months of developer time in allocating our core data structure in shared memory, allowing us to parallelize with multiprocessing. But there's still a whole bunch of ancillary data structures written in C++ that are not so easy to put in shared memory, so all analyses touching those are limited to a single process, which by Amdahl's law immediately starting dominating our execution time.

nutate 3y ago

Same w/ Rust and Python, this is really neat because now each thread could have a GIL without doing exactly what you said. The pyO3 commit to allow subinterpreters was merged 21 days ago, so this might "just work" today: https://github.com/PyO3/pyo3/pull/3446 https://github.com/PyO3/pyo3/pull/3446

ddorian43 3y ago

> I can now have multiple sub interpreters running concurrently and accessing the same shared state in a thread-safe C++ library. Can sub-interpreters be used to share a db/http/etc connection pool across multiple processes?

elashri 3y ago

Wouldn't you also gain some overhead reduction too? Multiprocessing will have more overhead for spawning new processes vs signle process that will contain the sup-interprets? or I am missing something? At least in theory this what would happen.

spacechild1 3y ago
Threads are certainly faster to create, use less memory and allow for faster context switching.