4 ms·
> multiprocessing allows concurrency at all and this doesn't I do not think multiprocessing will meet the concurrency needs of many potential applications. Fo
by pdonis 7y ago
> multiprocessing allows concurrency at all and this doesn't
I do not think multiprocessing will meet the concurrency needs of many potential applications.
For one thing, I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen. To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child. That seems awfully kludgey to me.
For another, if you have multiple concurrent tasks that have to share data, doing it within a single process between multiple threads is a lot easier than doing it between multiple processes. Every other major language besides Python allows the former; only Python has the GIL which makes the former basically useless.
- montecarl 7y agoThis is nothing but message passing. In the high performance computing world, MPI is the standard scheme for running massively parallel calculations. While Python's pickle may not be the highest performance serialization scheme, it is pretty fast and transparently handles most objects you can throw at it. It certainly does not cover all the same use cases as threads and as you point out shared data it is much easier to use theads.
- loeg 7y ago> To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child. FWIW, the subinterpreter threading scheme will also use a similar message passing construct to pass values between interpreters. As the sibling comment mentions, it's just a message passing scheme. That said, > I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen. I have as well, and completely agree.
- stuaxo 7y agoCame here just to agree.
- sicromoft 7y ago> Every other major language besides Python allows the former As noted in a reply to your other comment, this is not true -- Ruby has the equivalent of Python's GIL.
- jashmatthews 7y agoOnly CRuby has a GIL. JRuby is totally viable for production use and doesn’t have a GIL.