3 ms·

If all global state is made thread safe and, then whether threads are subinterpreters or a single interpreter is conceptually irrelevant and probably easier to

If all global state is made thread safe and, then whether threads are subinterpreters or a single interpreter is conceptually irrelevant and probably easier to implement.

kjeetgill 4y ago
It really really depends on what you mean by global state here. If you mean the global state within the interpreter that's one thing. Preserving the global state of your application is another. But a weird "global state" (really more a global property) is the semantics between concurrent pieces of code and the expectations about things like setting variables, possibly interleavings etc. The nice part of different interpreters isn't just getting around the gil, and maintaining similar isolation, but it's almost like a Terms and Services agreement: I opened this can of worms and it's my responsibility to learn what the consequences are.
forrestthewoods 4y ago
> conceptually irrelevant and probably easier to implement. Well, it depends on how it’s implemented. If “made thread safe” means constantly grabbing locks around large blocks of data then the end result is concurrency (hopefully!) but not parallelism. Meaning you might only have one thread active at a time in practice. Wrapping the universe in a mutex is thread safe. But it’s not a good solution.
- isthisthingon99 4y ago
  I presume they know what they are doing and won't be doing a big world mutex :-)
int_19h 4y ago
It is not conceptually irrelevant. With threads, you can create a Python object on one, store it into a global (or some shared state), and use that object from a different thread. You can't do that with subinterpreters tho.
- samsquire 4y ago
  Couldn't you transfer ownership between subinterpreters with shared memory outside the subinterpreter? Associate some shared memory with each subinterpreter (the same array or map)
  - int_19h 4y ago
    It's not a question of having a mechanism to transfer data. Sure, you can easily use a static global in a native module to easily transfer a reference across subinterpreter boundaries. But the moment you try to increment refcount for the referenced object, things already break, because you're going to be using the wrong (subinterpreter-global) lock.
    - samsquire 4y ago
      Oh I was thinking from a native python object perspective. You could have a rule that the refcount must be 1 when sending an object between subinterpreters. In other words, you cannot use an object that was .send() to another subinterpreter. Then you invalidate the reference in that subinterpreter when it calls send to the other subinterpreter which is transferred by assignment. Can transfer any amount of data with zero copies.
      - int_19h 4y ago
        Even a completely empty object will contain a reference to its type, which is itself an object. How will you marshal that? Bear in mind that each subinterpreter has its own copy of type objects, and there isn't even a guarantee that those types match even if their names do.
        samsquire 4y ago
        It sounds it is a problem but I feel it can be solved with engineering and mathematics. Not saying it would be easy though. Couldn't you separate the storage of the refcounts from the objects and use a map to get at them? As for the identities between types being different. To create an subinterpreter that can marshall between subinterpreters without copying the data structures requires a different data structure that is safe to use from any interpreter. We need to decouple the book keeping data structures from the underlying data. We can regenerate book keeping data structures during a .send or .receive Maintaining identity equivalence is an interesting problem. I think it's a more fundamental problem that probably has mathematical solutions. If we think of objects as being a vector in mathematical space. We have pointers between vectors in this memory space. For a data structure to be position independent. We need some way of intending references to be global. But we don't want to introduce a layer of indirection on the reference of object relationships. That would be slower. Could use an atomic counter to ensure that identifiers are globally unique. Don't want to serialize the access to global types. It sounds to me it is a many-to-many to many-to-many problem. Which even databases don't solve very well.

If all global state is made thread safe and, then whether threads are subinterpreters or a single interpreter is conceptually irrelevant and probably easier to implement.

kjeetgill 4y ago

It really really depends on what you mean by global state here. If you mean the global state within the interpreter that's one thing. Preserving the global state of your application is another. But a weird "global state" (really more a global property) is the semantics between concurrent pieces of code and the expectations about things like setting variables, possibly interleavings etc. The nice part of different interpreters isn't just getting around the gil, and maintaining similar isolation, but it's almost like a Terms and Services agreement: I opened this can of worms and it's my responsibility to learn what the consequences are.

forrestthewoods 4y ago

> conceptually irrelevant and probably easier to implement. Well, it depends on how it’s implemented. If “made thread safe” means constantly grabbing locks around large blocks of data then the end result is concurrency (hopefully!) but not parallelism. Meaning you might only have one thread active at a time in practice. Wrapping the universe in a mutex is thread safe. But it’s not a good solution.

isthisthingon99 4y ago
I presume they know what they are doing and won't be doing a big world mutex :-)

int_19h 4y ago

It is not conceptually irrelevant. With threads, you can create a Python object on one, store it into a global (or some shared state), and use that object from a different thread. You can't do that with subinterpreters tho.

samsquire 4y ago
Couldn't you transfer ownership between subinterpreters with shared memory outside the subinterpreter? Associate some shared memory with each subinterpreter (the same array or map)
- int_19h 4y ago
  It's not a question of having a mechanism to transfer data. Sure, you can easily use a static global in a native module to easily transfer a reference across subinterpreter boundaries. But the moment you try to increment refcount for the referenced object, things already break, because you're going to be using the wrong (subinterpreter-global) lock.
  - samsquire 4y ago
    Oh I was thinking from a native python object perspective. You could have a rule that the refcount must be 1 when sending an object between subinterpreters. In other words, you cannot use an object that was .send() to another subinterpreter. Then you invalidate the reference in that subinterpreter when it calls send to the other subinterpreter which is transferred by assignment. Can transfer any amount of data with zero copies.
    - int_19h 4y ago
      Even a completely empty object will contain a reference to its type, which is itself an object. How will you marshal that? Bear in mind that each subinterpreter has its own copy of type objects, and there isn't even a guarantee that those types match even if their names do.
      - samsquire 4y ago
        It sounds it is a problem but I feel it can be solved with engineering and mathematics. Not saying it would be easy though. Couldn't you separate the storage of the refcounts from the objects and use a map to get at them? As for the identities between types being different. To create an subinterpreter that can marshall between subinterpreters without copying the data structures requires a different data structure that is safe to use from any interpreter. We need to decouple the book keeping data structures from the underlying data. We can regenerate book keeping data structures during a .send or .receive Maintaining identity equivalence is an interesting problem. I think it's a more fundamental problem that probably has mathematical solutions. If we think of objects as being a vector in mathematical space. We have pointers between vectors in this memory space. For a data structure to be position independent. We need some way of intending references to be global. But we don't want to introduce a layer of indirection on the reference of object relationships. That would be slower. Could use an atomic counter to ensure that identifiers are globally unique. Don't want to serialize the access to global types. It sounds to me it is a many-to-many to many-to-many problem. Which even databases don't solve very well.