4 ms·
Yeah, it's got some of the same weaknesses as multiprocessing (and several new ones). Conceivably you could provide an API for handing off objects to the other
by loeg 8y ago
Yeah, it's got some of the same weaknesses as multiprocessing (and several new ones). Conceivably you could provide an API for handing off objects to the other interpreter without copying. I'm imagining an API like:
my_foo = interpreterX.pass_object(my_foo)
(The assignment being required to delete the originating reference from the source interpreter.) The interface would be obligated to check that there are no references that escape to the current interpreter and then my_foo and all referenced objects could be handed off to the other interpreter in whole.
I don't have any intuitions for if that would be cheaper than copying or not, and getting it right is certainly more difficult than serialization. (Because of the complexity, it's not worth having if it isn't cheaper.)
- lsorber 8y agoWhat in your view are the weaknesses of subinterpreters compared to multiprocessing?
- pas 8y agoIf they live in the same process (address space), then it's a lot harder to correctly manage concurrent memory access (at both low level and high). It's not a theoretical problem, just a very likely pragmatic observation. (Meaning, we'll see bugs in both the interpreter and the code using this feature.) Plus if one of the threads crashes, the whole process aborts. (Sure, the interpreter can handle a lot of faults gracefully, but not all.)
- loeg 8y agoThe main weakness is the one pas described in the sibling comment. Historically, Python C modules, inside and outside stdlib, have more or less safely been able to assume there was one global interpreter. This means they may have global state (apparently even some stdlib C libraries assume this). With multiple interpreters in the same process address space, any global state in C modules conflicts. I'm not claiming any originality here — the author of the article recognizes this problem and describes it, starting with: > Because CPython has been implemented with a single interpreter for so long, many parts of the code base use the “Runtime State” instead of the “Interpreter State”, so if PEP554 were to be merged in it’s current form there would still be many issues.