4 ms·

From the PEP: https://www.python.org/dev/peps/pep-0554/ https://www.python.org/dev/peps/pep-0554/ > A common misconception is that this PEP also includes a pro

by geofft 6y ago

From the PEP: https://www.python.org/dev/peps/pep-0554/ https://www.python.org/dev/peps/pep-0554/ > A common misconception is that this PEP also includes a promise that subinterpreters will no longer share the GIL. When that is clarified, the next question is "what is the point?". This is already answered at length in this PEP. Just to be clear, the value lies in: * increase exposure of the existing feature, which helps improve the code health of the entire CPython runtime * expose the (mostly) isolated execution of subinterpreters * preparation for per-interpreter GIL * encourage experimentation I think I'll ask the followup question - what is the point of those? Why should we increase exposure of an existing feature we know is not fully baked and we know will cause problems with NumPy/SciPy? How will the exposure improve the code health of CPython and who will do the improvement? What is the advantage in exposing isolated execution of subinterpreters? In what way does exposing this feature help prepare for a per-interpreter GIL? What experiments are being encouraged specifically?

mehrdadn 6y ago
I think the Rationale section tries to explain the benefits? https://www.python.org/dev/peps/pep-0554/#rationale https://www.python.org/dev/peps/pep-0554/#rationale What I don't get is how they expect C extensions to adapt. I don't expect widespread support for this for like... a decade?
- geofft 6y ago
  The Nick Coghlan quote seems a bit out of context - sure, there are all these things it does that multiprocessing doesn't do, but multiprocessing allows concurrency at all and this doesn't. The comparison makes sense if it's talking about no-shared-state concurrency, but this PEP is quite clear that it's only proposing no-shared-state ... single-threaded operation. The rest of the section is sparse on what you actually do with it. It says that it "has the potential to be a powerful tool" but not what you do with the power. It says it's about "enabling the fundamental capability of multiple isolated interpreters" but it's not clear what the capability brings you. The only thing with some details is the first sentence - "Running code in multiple interpreters provides a useful level of isolation within the same process." But what is the level? It sounds like it gives you isolation for most things but not all, which is about as useful as a face mask with a breathing hole cut out in it. It'd help to see a concrete answer of something you can build with this that you can't build without it (or perhaps not as easily/performantly/reliably/etc.). The Ceph PR, where they do something similar themselves (https://github.com/ceph/ceph/pull/14971 https://github.com/ceph/ceph/pull/14971), gives a good answer: "Notably, with this change, it's possible to have more than one mgr module use cherrypy." (But it's still not clear to me why the mgr can't just use multiple Python interpreters... ceph-mgr is kind of an amalgam of various useful services for your Ceph cluster, and it's never been clear to me why it has to be "the" manager with some modules instead of various independent services you can turn off/on as you need.)
  - pdonis 6y ago
    > multiprocessing allows concurrency at all and this doesn't I do not think multiprocessing will meet the concurrency needs of many potential applications. For one thing, I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen. To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child. That seems awfully kludgey to me. For another, if you have multiple concurrent tasks that have to share data, doing it within a single process between multiple threads is a lot easier than doing it between multiple processes. Every other major language besides Python allows the former; only Python has the GIL which makes the former basically useless.
    - montecarl 6y ago
      This is nothing but message passing. In the high performance computing world, MPI is the standard scheme for running massively parallel calculations. While Python's pickle may not be the highest performance serialization scheme, it is pretty fast and transparently handles most objects you can throw at it. It certainly does not cover all the same use cases as threads and as you point out shared data it is much easier to use theads.
      - Nursie 6y ago
        I ended up having to switch to dill instead of the default pickle a while back in one of my projects, because pickling lambdas is apparently not possible by default. But in general it's pretty useful.
        stuaxo 6y ago
        Same here. Everything seemed to be working fine until it didn't.
    - loeg 6y ago
      > To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child. FWIW, the subinterpreter threading scheme will also use a similar message passing construct to pass values between interpreters. As the sibling comment mentions, it's just a message passing scheme. That said, > I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen. I have as well, and completely agree.
- BiteCode_dev 6y ago
  Yes, that's why the work has to start ASAP. There is currently a project to abstract the C-API with HPy: https://speakerdeck.com/antocuni/hpy-a-future-proof-way-of-extending-python https://speakerdeck.com/antocuni/hpy-a-future-proof-way-of-e... The idea is to (very slowly) remove access to direct CPython internals and give a handle to important low-level behavior through a standardized API hidding implementation details. It's a lot of work, and going to affect a huge part of the ecosystem, but I'm hopeful this is something the extensions authors will actually want to be part of because it promises to solve a lot of problems: - on the long run, this will make maintaining a c-extension much easier: more stability no matter the evolution of upstream, more certainty. - there will be a clear way to do things, which means better documentation and conventions, potentially libs to do common things. Right now creating extensions is more a set of receipes people found out to work. - because of this, scrutiny will be given to common patterns, which will be abstracted and optimized, and given the security of an API contract - extensions using this API will work out of the box, or faster if they already did, on alternative python implementations like pypy Of course, the end goal is to free the python core devs to experiment with breaking changes at the c level, such as changing the memory management model, and eventually maybe remove the GIL or allow for a JIT.
pdonis 6y ago
My take on this is that subinterpreters by themselves don't excite me that much, but the possibility of having subinterpreters running each in their own thread that do not have to share a single GIL is a huge potential win for Python. No other popular language has the huge restriction on multi-threaded code that Python has with the GIL. I have been hoping for years that the Python dev team would recognize that as an issue and try to do something about it.
- pansa2 6y ago
  > No other popular language has the huge restriction on multi-threaded code that Python has with the GIL. Is Ruby’s (MRI’s) GVL equivalent to Python’s GIL?
  - sicromoft 6y ago
    It is.
scoot_718 6y ago
* increase exposure of the existing feature, which helps improve the code health of the entire CPython runtime So masturbation * expose the (mostly) isolated execution of subinterpreters masturbation * preparation for per-interpreter GIL What they're asking for anyway * encourage experimentation more masturbation.

by geofft 6y ago

mehrdadn 6y ago

I think the Rationale section tries to explain the benefits? https://www.python.org/dev/peps/pep-0554/#rationale https://www.python.org/dev/peps/pep-0554/#rationale What I don't get is how they expect C extensions to adapt. I don't expect widespread support for this for like... a decade?

geofft 6y ago
The Nick Coghlan quote seems a bit out of context - sure, there are all these things it does that multiprocessing doesn't do, but multiprocessing allows concurrency at all and this doesn't. The comparison makes sense if it's talking about no-shared-state concurrency, but this PEP is quite clear that it's only proposing no-shared-state ... single-threaded operation. The rest of the section is sparse on what you actually do with it. It says that it "has the potential to be a powerful tool" but not what you do with the power. It says it's about "enabling the fundamental capability of multiple isolated interpreters" but it's not clear what the capability brings you. The only thing with some details is the first sentence - "Running code in multiple interpreters provides a useful level of isolation within the same process." But what is the level? It sounds like it gives you isolation for most things but not all, which is about as useful as a face mask with a breathing hole cut out in it. It'd help to see a concrete answer of something you can build with this that you can't build without it (or perhaps not as easily/performantly/reliably/etc.). The Ceph PR, where they do something similar themselves (https://github.com/ceph/ceph/pull/14971 https://github.com/ceph/ceph/pull/14971), gives a good answer: "Notably, with this change, it's possible to have more than one mgr module use cherrypy." (But it's still not clear to me why the mgr can't just use multiple Python interpreters... ceph-mgr is kind of an amalgam of various useful services for your Ceph cluster, and it's never been clear to me why it has to be "the" manager with some modules instead of various independent services you can turn off/on as you need.)
- pdonis 6y ago
  > multiprocessing allows concurrency at all and this doesn't I do not think multiprocessing will meet the concurrency needs of many potential applications. For one thing, I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen. To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child. That seems awfully kludgey to me. For another, if you have multiple concurrent tasks that have to share data, doing it within a single process between multiple threads is a lot easier than doing it between multiple processes. Every other major language besides Python allows the former; only Python has the GIL which makes the former basically useless.
  - montecarl 6y ago
    This is nothing but message passing. In the high performance computing world, MPI is the standard scheme for running massively parallel calculations. While Python's pickle may not be the highest performance serialization scheme, it is pretty fast and transparently handles most objects you can throw at it. It certainly does not cover all the same use cases as threads and as you point out shared data it is much easier to use theads.
    - Nursie 6y ago
      I ended up having to switch to dill instead of the default pickle a while back in one of my projects, because pickling lambdas is apparently not possible by default. But in general it's pretty useful.
      - stuaxo 6y ago
        Same here. Everything seemed to be working fine until it didn't.
  - loeg 6y ago
    > To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child. FWIW, the subinterpreter threading scheme will also use a similar message passing construct to pass values between interpreters. As the sibling comment mentions, it's just a message passing scheme. That said, > I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen. I have as well, and completely agree.
BiteCode_dev 6y ago
Yes, that's why the work has to start ASAP. There is currently a project to abstract the C-API with HPy: https://speakerdeck.com/antocuni/hpy-a-future-proof-way-of-extending-python https://speakerdeck.com/antocuni/hpy-a-future-proof-way-of-e... The idea is to (very slowly) remove access to direct CPython internals and give a handle to important low-level behavior through a standardized API hidding implementation details. It's a lot of work, and going to affect a huge part of the ecosystem, but I'm hopeful this is something the extensions authors will actually want to be part of because it promises to solve a lot of problems: - on the long run, this will make maintaining a c-extension much easier: more stability no matter the evolution of upstream, more certainty. - there will be a clear way to do things, which means better documentation and conventions, potentially libs to do common things. Right now creating extensions is more a set of receipes people found out to work. - because of this, scrutiny will be given to common patterns, which will be abstracted and optimized, and given the security of an API contract - extensions using this API will work out of the box, or faster if they already did, on alternative python implementations like pypy Of course, the end goal is to free the python core devs to experiment with breaking changes at the c level, such as changing the memory management model, and eventually maybe remove the GIL or allow for a JIT.

pdonis 6y ago

My take on this is that subinterpreters by themselves don't excite me that much, but the possibility of having subinterpreters running each in their own thread that do not have to share a single GIL is a huge potential win for Python. No other popular language has the huge restriction on multi-threaded code that Python has with the GIL. I have been hoping for years that the Python dev team would recognize that as an issue and try to do something about it.

pansa2 6y ago
> No other popular language has the huge restriction on multi-threaded code that Python has with the GIL. Is Ruby’s (MRI’s) GVL equivalent to Python’s GIL?
- sicromoft 6y ago
  It is.

scoot_718 6y ago

* increase exposure of the existing feature, which helps improve the code health of the entire CPython runtime So masturbation * expose the (mostly) isolated execution of subinterpreters masturbation * preparation for per-interpreter GIL What they're asking for anyway * encourage experimentation more masturbation.