3 ms·

> - they can move objects much faster between subinterpreters because they don’t need to serialize through a format like JSON Why do you think that you would n

> - they can move objects much faster between subinterpreters because they don’t need to serialize through a format like JSON Why do you think that you would need to serialize to JSON? Pipes and sockets can deal with binary data just fine. With shared memory, there wouldn't be any difference at all. > - since they are all in the same process you can implement things like atomics or channels easily. This is also possible with shared memory. AFAICT the advantage of subinterpreters over subprocesses are: - lower memory overhead - faster creation/destruction time - ability to share global data (with subprocesses the data would either need to be duplicated or live in shared memory)

celeritascelery 3y ago
Sure, you could do something like that. But a shared memory segment python with a stable binary object format doesn't exist (and isn't even being worked on). Comparing the proposed PEP 554 solution to a non-existent theoretical solution isn't very useful. But you do bring up some good points for ways you could achieve similar goals without the need to make the interpreters thread safe.
- spacechild1 3y ago
  > But a shared memory segment python with a stable binary object format doesn't exist There is multiprocessing.Queue (https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue https://docs.python.org/3/library/multiprocessing.html#multi...). I don't know if it uses shared memory, or rather sockets or pipes, but this is just an implementation detail. My point is that there is no fundamental difference between isolated interpreters and processes when it comes to data sharing. Either way, you need a (binary) serialization format and some thread/process-safe queue. I would have naively assumed that you could repurpose multiprocessing.Queue for passing data between multiple interpreters; you would just need to replace the underlying communication mechanism (sockets, pipes, shared memory, whatever it is) with a queue + mutex. But then again, I'm not familiar with the actual code base. If there are any complications that I didn't take into acccount, I would be curious to hear about them. Interestingly, the PEP authors currently don't propose an API for exchanging data and instead suggest using raw pipes: > https://peps.python.org/pep-0554/#api-for-sharing-data https://peps.python.org/pep-0554/#api-for-sharing-data Of course, this is just a temporary hack. It would be ridiculous to use actual pipes for sharing data within the same process...

celeritascelery 3y ago

Sure, you could do something like that. But a shared memory segment python with a stable binary object format doesn't exist (and isn't even being worked on). Comparing the proposed PEP 554 solution to a non-existent theoretical solution isn't very useful. But you do bring up some good points for ways you could achieve similar goals without the need to make the interpreters thread safe.

spacechild1 3y ago
> But a shared memory segment python with a stable binary object format doesn't exist There is multiprocessing.Queue (https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue https://docs.python.org/3/library/multiprocessing.html#multi...). I don't know if it uses shared memory, or rather sockets or pipes, but this is just an implementation detail. My point is that there is no fundamental difference between isolated interpreters and processes when it comes to data sharing. Either way, you need a (binary) serialization format and some thread/process-safe queue. I would have naively assumed that you could repurpose multiprocessing.Queue for passing data between multiple interpreters; you would just need to replace the underlying communication mechanism (sockets, pipes, shared memory, whatever it is) with a queue + mutex. But then again, I'm not familiar with the actual code base. If there are any complications that I didn't take into acccount, I would be curious to hear about them. Interestingly, the PEP authors currently don't propose an API for exchanging data and instead suggest using raw pipes: > https://peps.python.org/pep-0554/#api-for-sharing-data https://peps.python.org/pep-0554/#api-for-sharing-data Of course, this is just a temporary hack. It would be ridiculous to use actual pipes for sharing data within the same process...