3 ms·
The Nick Coghlan quote seems a bit out of context - sure, there are all these things it does that multiprocessing doesn't do, but multiprocessing allows concurr
by geofft 6y ago
The Nick Coghlan quote seems a bit out of context - sure, there are all these things it does that multiprocessing doesn't do, but multiprocessing allows concurrency at all and this doesn't. The comparison makes sense if it's talking about no-shared-state concurrency, but this PEP is quite clear that it's only proposing no-shared-state ... single-threaded operation.
The rest of the section is sparse on what you actually do with it. It says that it "has the potential to be a powerful tool" but not what you do with the power. It says it's about "enabling the fundamental capability of multiple isolated interpreters" but it's not clear what the capability brings you. The only thing with some details is the first sentence - "Running code in multiple interpreters provides a useful level of isolation within the same process." But what is the level? It sounds like it gives you isolation for most things but not all, which is about as useful as a face mask with a breathing hole cut out in it.
It'd help to see a concrete answer of something you can build with this that you can't build without it (or perhaps not as easily/performantly/reliably/etc.). The Ceph PR, where they do something similar themselves (https://github.com/ceph/ceph/pull/14971 https://github.com/ceph/ceph/pull/14971), gives a good answer: "Notably, with this change, it's possible to have more than one mgr module use cherrypy." (But it's still not clear to me why the mgr can't just use multiple Python interpreters... ceph-mgr is kind of an amalgam of various useful services for your Ceph cluster, and it's never been clear to me why it has to be "the" manager with some modules instead of various independent services you can turn off/on as you need.)
- pdonis 6y ago> multiprocessing allows concurrency at all and this doesn't I do not think multiprocessing will meet the concurrency needs of many potential applications. For one thing, I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen. To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child. That seems awfully kludgey to me. For another, if you have multiple concurrent tasks that have to share data, doing it within a single process between multiple threads is a lot easier than doing it between multiple processes. Every other major language besides Python allows the former; only Python has the GIL which makes the former basically useless.
- montecarl 6y agoThis is nothing but message passing. In the high performance computing world, MPI is the standard scheme for running massively parallel calculations. While Python's pickle may not be the highest performance serialization scheme, it is pretty fast and transparently handles most objects you can throw at it. It certainly does not cover all the same use cases as threads and as you point out shared data it is much easier to use theads.
- loeg 6y ago> To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child. FWIW, the subinterpreter threading scheme will also use a similar message passing construct to pass values between interpreters. As the sibling comment mentions, it's just a message passing scheme. That said, > I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen. I have as well, and completely agree.
- stuaxo 6y agoCame here just to agree.
- sicromoft 6y ago> Every other major language besides Python allows the former As noted in a reply to your other comment, this is not true -- Ruby has the equivalent of Python's GIL.