4 ms·

Sure, you could do something like that. But a shared memory segment python with a stable binary object format doesn't exist (and isn't even being worked on). Co

Sure, you could do something like that. But a shared memory segment python with a stable binary object format doesn't exist (and isn't even being worked on). Comparing the proposed PEP 554 solution to a non-existent theoretical solution isn't very useful. But you do bring up some good points for ways you could achieve similar goals without the need to make the interpreters thread safe.

spacechild1 3y ago
> But a shared memory segment python with a stable binary object format doesn't exist There is multiprocessing.Queue (https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue https://docs.python.org/3/library/multiprocessing.html#multi...). I don't know if it uses shared memory, or rather sockets or pipes, but this is just an implementation detail. My point is that there is no fundamental difference between isolated interpreters and processes when it comes to data sharing. Either way, you need a (binary) serialization format and some thread/process-safe queue. I would have naively assumed that you could repurpose multiprocessing.Queue for passing data between multiple interpreters; you would just need to replace the underlying communication mechanism (sockets, pipes, shared memory, whatever it is) with a queue + mutex. But then again, I'm not familiar with the actual code base. If there are any complications that I didn't take into acccount, I would be curious to hear about them. Interestingly, the PEP authors currently don't propose an API for exchanging data and instead suggest using raw pipes: > https://peps.python.org/pep-0554/#api-for-sharing-data https://peps.python.org/pep-0554/#api-for-sharing-data Of course, this is just a temporary hack. It would be ridiculous to use actual pipes for sharing data within the same process...

4 ms·

Sure, you could do something like that. But a shared memory segment python with a stable binary object format doesn't exist (and isn't even being worked on). Co

by celeritascelery 3y ago

spacechild1 3y ago
> But a shared memory segment python with a stable binary object format doesn't exist There is multiprocessing.Queue (https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue https://docs.python.org/3/library/multiprocessing.html#multi...). I don't know if it uses shared memory, or rather sockets or pipes, but this is just an implementation detail. My point is that there is no fundamental difference between isolated interpreters and processes when it comes to data sharing. Either way, you need a (binary) serialization format and some thread/process-safe queue. I would have naively assumed that you could repurpose multiprocessing.Queue for passing data between multiple interpreters; you would just need to replace the underlying communication mechanism (sockets, pipes, shared memory, whatever it is) with a queue + mutex. But then again, I'm not familiar with the actual code base. If there are any complications that I didn't take into acccount, I would be curious to hear about them. Interestingly, the PEP authors currently don't propose an API for exchanging data and instead suggest using raw pipes: > https://peps.python.org/pep-0554/#api-for-sharing-data https://peps.python.org/pep-0554/#api-for-sharing-data Of course, this is just a temporary hack. It would be ridiculous to use actual pipes for sharing data within the same process...