3 ms·

Perfomance,I suppose? Spawning a process is expensive.

chrstphrhrt 6y ago
This makes sense. Combine it with asyncio and you have a full Node-like runtime without relying on Uvicorn and the like (even though clustering is still good for scaling).
- pdonis 6y ago
  If you're using asyncio, I don't see what having a subinterpreter gains you. Asyncio by itself is already a Node-like runtime without relying on third-party libraries.
  - chrstphrhrt 6y ago
    True true, my comment was not clear. With asyncio it's still a single threaded event loop (that can be clustered like Node or any Python processes as has always been done). With sub-interpreters, I wonder how different it is from normal multithreading like in JVM. Need to read up on this idea. I can't see how it wouldn't require the same locking and mutexes or message passing if multiple interpreters are to work on logically related data. So my immediate thoughts were about leveraging it as a replacement for multithreading and event loop clustering by treating the interpreters as lightweight processes and having them communicate over some kind of protocol. Like how the BEAM does built-in process supervision. The "supervisor" process itself is async in the way it coordinates the tree.
    - chrstphrhrt 6y ago
      Aaand I finally read the article and PEP and there's no escape from the GIL :( Damn wishful thinking.
xioxox 6y ago
I understood that on Linux using a process was similar in speed to a thread, though they're much slower on Windows. Has that changed? For my python multi core code I like using fork with sending objects over a socket with pickle. It gives more control than multiprocessing. It works pretty well. One downside of fork with python is that the reference counters are scattered about, leading to big COW memory churn, however. Big numpy arrays should be shared, however.
- BiteCode_dev 6y ago
  Spwaning the process is not the only problem, communicating between them is also expensive.

This makes sense. Combine it with asyncio and you have a full Node-like runtime without relying on Uvicorn and the like (even though clustering is still good for scaling).

pdonis 6y ago
If you're using asyncio, I don't see what having a subinterpreter gains you. Asyncio by itself is already a Node-like runtime without relying on third-party libraries.
- chrstphrhrt 6y ago
  True true, my comment was not clear. With asyncio it's still a single threaded event loop (that can be clustered like Node or any Python processes as has always been done). With sub-interpreters, I wonder how different it is from normal multithreading like in JVM. Need to read up on this idea. I can't see how it wouldn't require the same locking and mutexes or message passing if multiple interpreters are to work on logically related data. So my immediate thoughts were about leveraging it as a replacement for multithreading and event loop clustering by treating the interpreters as lightweight processes and having them communicate over some kind of protocol. Like how the BEAM does built-in process supervision. The "supervisor" process itself is async in the way it coordinates the tree.
  - chrstphrhrt 6y ago
    Aaand I finally read the article and PEP and there's no escape from the GIL :( Damn wishful thinking.

xioxox 6y ago

I understood that on Linux using a process was similar in speed to a thread, though they're much slower on Windows. Has that changed? For my python multi core code I like using fork with sending objects over a socket with pickle. It gives more control than multiprocessing. It works pretty well. One downside of fork with python is that the reference counters are scattered about, leading to big COW memory churn, however. Big numpy arrays should be shared, however.

BiteCode_dev 6y ago
Spwaning the process is not the only problem, communicating between them is also expensive.