4 ms·

I think the Rationale section tries to explain the benefits? https://www.python.org/dev/peps/pep-0554/#rationale https://www.python.org/dev/peps/pep-0554/#ratio

I think the Rationale section tries to explain the benefits? https://www.python.org/dev/peps/pep-0554/#rationale https://www.python.org/dev/peps/pep-0554/#rationale What I don't get is how they expect C extensions to adapt. I don't expect widespread support for this for like... a decade?

geofft 6y ago
The Nick Coghlan quote seems a bit out of context - sure, there are all these things it does that multiprocessing doesn't do, but multiprocessing allows concurrency at all and this doesn't. The comparison makes sense if it's talking about no-shared-state concurrency, but this PEP is quite clear that it's only proposing no-shared-state ... single-threaded operation. The rest of the section is sparse on what you actually do with it. It says that it "has the potential to be a powerful tool" but not what you do with the power. It says it's about "enabling the fundamental capability of multiple isolated interpreters" but it's not clear what the capability brings you. The only thing with some details is the first sentence - "Running code in multiple interpreters provides a useful level of isolation within the same process." But what is the level? It sounds like it gives you isolation for most things but not all, which is about as useful as a face mask with a breathing hole cut out in it. It'd help to see a concrete answer of something you can build with this that you can't build without it (or perhaps not as easily/performantly/reliably/etc.). The Ceph PR, where they do something similar themselves (https://github.com/ceph/ceph/pull/14971 https://github.com/ceph/ceph/pull/14971), gives a good answer: "Notably, with this change, it's possible to have more than one mgr module use cherrypy." (But it's still not clear to me why the mgr can't just use multiple Python interpreters... ceph-mgr is kind of an amalgam of various useful services for your Ceph cluster, and it's never been clear to me why it has to be "the" manager with some modules instead of various independent services you can turn off/on as you need.)
- pdonis 6y ago
  > multiprocessing allows concurrency at all and this doesn't I do not think multiprocessing will meet the concurrency needs of many potential applications. For one thing, I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen. To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child. That seems awfully kludgey to me. For another, if you have multiple concurrent tasks that have to share data, doing it within a single process between multiple threads is a lot easier than doing it between multiple processes. Every other major language besides Python allows the former; only Python has the GIL which makes the former basically useless.
  - montecarl 6y ago
    This is nothing but message passing. In the high performance computing world, MPI is the standard scheme for running massively parallel calculations. While Python's pickle may not be the highest performance serialization scheme, it is pretty fast and transparently handles most objects you can throw at it. It certainly does not cover all the same use cases as threads and as you point out shared data it is much easier to use theads.
    - Nursie 6y ago
      I ended up having to switch to dill instead of the default pickle a while back in one of my projects, because pickling lambdas is apparently not possible by default. But in general it's pretty useful.
      - stuaxo 6y ago
        Same here. Everything seemed to be working fine until it didn't.
  - loeg 6y ago
    > To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child. FWIW, the subinterpreter threading scheme will also use a similar message passing construct to pass values between interpreters. As the sibling comment mentions, it's just a message passing scheme. That said, > I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen. I have as well, and completely agree.
    - stuaxo 6y ago
      Came here just to agree.
  - sicromoft 6y ago
    > Every other major language besides Python allows the former As noted in a reply to your other comment, this is not true -- Ruby has the equivalent of Python's GIL.
BiteCode_dev 6y ago
Yes, that's why the work has to start ASAP. There is currently a project to abstract the C-API with HPy: https://speakerdeck.com/antocuni/hpy-a-future-proof-way-of-extending-python https://speakerdeck.com/antocuni/hpy-a-future-proof-way-of-e... The idea is to (very slowly) remove access to direct CPython internals and give a handle to important low-level behavior through a standardized API hidding implementation details. It's a lot of work, and going to affect a huge part of the ecosystem, but I'm hopeful this is something the extensions authors will actually want to be part of because it promises to solve a lot of problems: - on the long run, this will make maintaining a c-extension much easier: more stability no matter the evolution of upstream, more certainty. - there will be a clear way to do things, which means better documentation and conventions, potentially libs to do common things. Right now creating extensions is more a set of receipes people found out to work. - because of this, scrutiny will be given to common patterns, which will be abstracted and optimized, and given the security of an API contract - extensions using this API will work out of the box, or faster if they already did, on alternative python implementations like pypy Of course, the end goal is to free the python core devs to experiment with breaking changes at the c level, such as changing the memory management model, and eventually maybe remove the GIL or allow for a JIT.

geofft 6y ago

The Nick Coghlan quote seems a bit out of context - sure, there are all these things it does that multiprocessing doesn't do, but multiprocessing allows concurrency at all and this doesn't. The comparison makes sense if it's talking about no-shared-state concurrency, but this PEP is quite clear that it's only proposing no-shared-state ... single-threaded operation. The rest of the section is sparse on what you actually do with it. It says that it "has the potential to be a powerful tool" but not what you do with the power. It says it's about "enabling the fundamental capability of multiple isolated interpreters" but it's not clear what the capability brings you. The only thing with some details is the first sentence - "Running code in multiple interpreters provides a useful level of isolation within the same process." But what is the level? It sounds like it gives you isolation for most things but not all, which is about as useful as a face mask with a breathing hole cut out in it. It'd help to see a concrete answer of something you can build with this that you can't build without it (or perhaps not as easily/performantly/reliably/etc.). The Ceph PR, where they do something similar themselves (https://github.com/ceph/ceph/pull/14971 https://github.com/ceph/ceph/pull/14971), gives a good answer: "Notably, with this change, it's possible to have more than one mgr module use cherrypy." (But it's still not clear to me why the mgr can't just use multiple Python interpreters... ceph-mgr is kind of an amalgam of various useful services for your Ceph cluster, and it's never been clear to me why it has to be "the" manager with some modules instead of various independent services you can turn off/on as you need.)

pdonis 6y ago
> multiprocessing allows concurrency at all and this doesn't I do not think multiprocessing will meet the concurrency needs of many potential applications. For one thing, I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen. To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child. That seems awfully kludgey to me. For another, if you have multiple concurrent tasks that have to share data, doing it within a single process between multiple threads is a lot easier than doing it between multiple processes. Every other major language besides Python allows the former; only Python has the GIL which makes the former basically useless.
- montecarl 6y ago
  This is nothing but message passing. In the high performance computing world, MPI is the standard scheme for running massively parallel calculations. While Python's pickle may not be the highest performance serialization scheme, it is pretty fast and transparently handles most objects you can throw at it. It certainly does not cover all the same use cases as threads and as you point out shared data it is much easier to use theads.
  - Nursie 6y ago
    I ended up having to switch to dill instead of the default pickle a while back in one of my projects, because pickling lambdas is apparently not possible by default. But in general it's pretty useful.
    - stuaxo 6y ago
      Same here. Everything seemed to be working fine until it didn't.
- loeg 6y ago
  > To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child. FWIW, the subinterpreter threading scheme will also use a similar message passing construct to pass values between interpreters. As the sibling comment mentions, it's just a message passing scheme. That said, > I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen. I have as well, and completely agree.
  - stuaxo 6y ago
    Came here just to agree.
- sicromoft 6y ago
  > Every other major language besides Python allows the former As noted in a reply to your other comment, this is not true -- Ruby has the equivalent of Python's GIL.

BiteCode_dev 6y ago

Yes, that's why the work has to start ASAP. There is currently a project to abstract the C-API with HPy: https://speakerdeck.com/antocuni/hpy-a-future-proof-way-of-extending-python https://speakerdeck.com/antocuni/hpy-a-future-proof-way-of-e... The idea is to (very slowly) remove access to direct CPython internals and give a handle to important low-level behavior through a standardized API hidding implementation details. It's a lot of work, and going to affect a huge part of the ecosystem, but I'm hopeful this is something the extensions authors will actually want to be part of because it promises to solve a lot of problems: - on the long run, this will make maintaining a c-extension much easier: more stability no matter the evolution of upstream, more certainty. - there will be a clear way to do things, which means better documentation and conventions, potentially libs to do common things. Right now creating extensions is more a set of receipes people found out to work. - because of this, scrutiny will be given to common patterns, which will be abstracted and optimized, and given the security of an API contract - extensions using this API will work out of the box, or faster if they already did, on alternative python implementations like pypy Of course, the end goal is to free the python core devs to experiment with breaking changes at the c level, such as changing the memory management model, and eventually maybe remove the GIL or allow for a JIT.