4 ms·

> they can move objects much faster between subinterpreters because they don’t need to serialize through a format like JSON That would be a huge advantage, but

by ptx 3y ago

> they can move objects much faster between subinterpreters because they don’t need to serialize through a format like JSON That would be a huge advantage, but it's not there yet. According to PEP 554 [1] the only mechanism for sharing data is sending bytes through OS pipes, which is exactly the same as for multiprocessing and requires the same sort of serialization. [1] https://peps.python.org/pep-0554/#api-for-sharing-data https://peps.python.org/pep-0554/#api-for-sharing-data

semiquaver 3y ago
Is the overhead of pickle eg as used in multiprocessing.Pipe() actually a limiting factor in most circumstances? https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Pipe https://docs.python.org/3/library/multiprocessing.html#multi... In a message passing system there’s always going to need to be some form of serialization. I’ll wager that pickle is fast and flexible enough for most cases and for those that aren’t, using something like flatbuffers or capn proto in shared memory wouldn’t be too much of a lift to integrate. Although all of that has long been possible in a multiple-process architecture, so I’m also curious to know if there are any real advantages to subinterpreters. From this message [1] linked to from the PEP it sounds like the author once thought that object sharing was a possibility, but if it’s not there seem to be no real benefits over multiprocessing and one big downside (the GIL). Contrast with ruby’s Ractor system [2], which is similar to the subinterpreter concept but allows true parallelism within a single process by giving each ractor its own interpreter lock, along with a system for marking an object as immutable so it can be shared among ractors. [1] https://mail.python.org/pipermail/python-ideas/2017-September/047122.html https://mail.python.org/pipermail/python-ideas/2017-Septembe... [2] https://github.com/ruby/ruby/blob/master/doc/ractor.md https://github.com/ruby/ruby/blob/master/doc/ractor.md
- gyrovagueGeist 3y ago
  In my experience, yes, pickle is painfully slow and can't be used for anything real. It's fantastic for prototyping (especially https://mpi4py.readthedocs.io/en/stable/reference/mpi4py.MPI.Pickle.html https://mpi4py.readthedocs.io/en/stable/reference/mpi4py.MPI...) but it will be your bottleneck. But I work in more computational science so I understand my constraints are different than most folks.
  - aragilar 3y ago
    Which pickle version did you use (usually the default is relatively old)?
    - ptx 3y ago
      Hmm. Apparently version 5 of the pickle protocol (which is faster) isn't yet used by the multiprocessing module [1] even though it was mentioned as part of the rationale in PEP 574 [2]. [1] https://github.com/python/cpython/issues/89467 https://github.com/python/cpython/issues/89467 [2] https://peps.python.org/pep-0574/ https://peps.python.org/pep-0574/

by ptx 3y ago

semiquaver 3y ago

Is the overhead of pickle eg as used in multiprocessing.Pipe() actually a limiting factor in most circumstances? https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Pipe https://docs.python.org/3/library/multiprocessing.html#multi... In a message passing system there’s always going to need to be some form of serialization. I’ll wager that pickle is fast and flexible enough for most cases and for those that aren’t, using something like flatbuffers or capn proto in shared memory wouldn’t be too much of a lift to integrate. Although all of that has long been possible in a multiple-process architecture, so I’m also curious to know if there are any real advantages to subinterpreters. From this message [1] linked to from the PEP it sounds like the author once thought that object sharing was a possibility, but if it’s not there seem to be no real benefits over multiprocessing and one big downside (the GIL). Contrast with ruby’s Ractor system [2], which is similar to the subinterpreter concept but allows true parallelism within a single process by giving each ractor its own interpreter lock, along with a system for marking an object as immutable so it can be shared among ractors. [1] https://mail.python.org/pipermail/python-ideas/2017-September/047122.html https://mail.python.org/pipermail/python-ideas/2017-Septembe... [2] https://github.com/ruby/ruby/blob/master/doc/ractor.md https://github.com/ruby/ruby/blob/master/doc/ractor.md

gyrovagueGeist 3y ago
In my experience, yes, pickle is painfully slow and can't be used for anything real. It's fantastic for prototyping (especially https://mpi4py.readthedocs.io/en/stable/reference/mpi4py.MPI.Pickle.html https://mpi4py.readthedocs.io/en/stable/reference/mpi4py.MPI...) but it will be your bottleneck. But I work in more computational science so I understand my constraints are different than most folks.
- aragilar 3y ago
  Which pickle version did you use (usually the default is relatively old)?
  - ptx 3y ago
    Hmm. Apparently version 5 of the pickle protocol (which is faster) isn't yet used by the multiprocessing module [1] even though it was mentioned as part of the rationale in PEP 574 [2]. [1] https://github.com/python/cpython/issues/89467 https://github.com/python/cpython/issues/89467 [2] https://peps.python.org/pep-0574/ https://peps.python.org/pep-0574/