4 ms·
I don't see the benefit of a subinterpreter compared to a subprocess. I wish python didn't have a GIL, but I don't see what problems this would solve for me tha
by shanemhansen 6y ago
I don't see the benefit of a subinterpreter compared to a subprocess. I wish python didn't have a GIL, but I don't see what problems this would solve for me that multiprocessing doesn't.
- deleted 6y ago[deleted]
- cheez 6y agoYeah... super confusing.
- joelthelion 6y agoPerfomance,I suppose? Spawning a process is expensive.
- chrstphrhrt 6y agoThis makes sense. Combine it with asyncio and you have a full Node-like runtime without relying on Uvicorn and the like (even though clustering is still good for scaling).
- pdonis 6y agoIf you're using asyncio, I don't see what having a subinterpreter gains you. Asyncio by itself is already a Node-like runtime without relying on third-party libraries.
- chrstphrhrt 6y agoTrue true, my comment was not clear. With asyncio it's still a single threaded event loop (that can be clustered like Node or any Python processes as has always been done). With sub-interpreters, I wonder how different it is from normal multithreading like in JVM. Need to read up on this idea. I can't see how it wouldn't require the same locking and mutexes or message passing if multiple interpreters are to work on logically related data. So my immediate thoughts were about leveraging it as a replacement for multithreading and event loop clustering by treating the interpreters as lightweight processes and having them communicate over some kind of protocol. Like how the BEAM does built-in process supervision. The "supervisor" process itself is async in the way it coordinates the tree.
- chrstphrhrt 6y agoAaand I finally read the article and PEP and there's no escape from the GIL :( Damn wishful thinking.
- xioxox 6y agoI understood that on Linux using a process was similar in speed to a thread, though they're much slower on Windows. Has that changed? For my python multi core code I like using fork with sending objects over a socket with pickle. It gives more control than multiprocessing. It works pretty well. One downside of fork with python is that the reference counters are scattered about, leading to big COW memory churn, however. Big numpy arrays should be shared, however.
- BiteCode_dev 6y agoSpwaning the process is not the only problem, communicating between them is also expensive.
- jonahbenton 6y agoGetting to a go-like lightweight concurrency model in the language. Different abstraction than multi process. Py subinterpreter will never be as cheap as a goroutine stack, so you wont be able to run millions of pyroutines. But thousands is a reasonable and useful goal.
- pansa2 6y agoIt's interesting to compare with Go, which uses goroutines for all concurrency. Python has threading, multiprocessing, async-await and now subinterpreters. Remind me which language is designed to have "one obvious way to do it"...
- acdha 6y agoThose are all different kinds of “it” with differences which affect your application design. If you want a stronger criticism, pick multiprocessing/threading and concurrent, and the long migration trajectory for deprecating the older ones.
- pansa2 6y ago> Those are all different kinds of “it” with differences which affect your application design. "It" here is concurrency, and Go proves that its possible for a language to have "one way to do it". Goroutines are often seen as an alternative to async/await, but they're also a better parallelism solution than all of Python's threading, multiprocessing and subinterpreters. > If you want a stronger criticism, pick multiprocessing/threading and concurrent, and the long migration trajectory for deprecating the older ones. I haven't heard anything about deprecating multiprocessing/threading. Is that the long-term plan for subinterpreters?
- jonahbenton 6y agoYeah, in his later years I think Guido stopped having the energy to keep saying no. Not a criticism. He deserves nothing but plaudits. One of best earned retirements in all of CS history, IMO.
- thelazydogsback 6y agoI never understood why Stackless Python didn't become the de-facto implementation.