4 ms·

PEP 703 mentions the multiprocessing module which I used recently with great success. It is essentially message passing parallelism, i.e. the actor model. I wou

by hgs3 3y ago

PEP 703 mentions the multiprocessing module which I used recently with great success. It is essentially message passing parallelism, i.e. the actor model. I would think even with the GIL removed message passing is still the best approach for a higher-level language, like Python, to handle parallelism as it is conceptually simple and lock free. The primary disadvantage of the multiprocessing module is it uses separate processes which means my resource heavy Python extension needs to be duplicated multiple times in memory (at least on Windows). I think the move to supporting threads means the module could leverage them instead of separate processes, avoiding the need to duplicate C extensions in memory, and thereby reducing the overall memory pressure on a system.

packetlost 3y ago
I agree that message passing can be a really good choice. The problem is the overhead of passing large messages can be really substantial. In general, you'll need to store messages in memory 4x per message (at least temporarily): once on each end for the Python representation and the serialized "on the wire" representation. There's also the issue with `pickle` generally being a horrible serialization format and being the default. There are ways around this, Python has a rather rudimentary and archaic shared memory subsystem[0] that can make this much more efficient. On the other hand, the Python multiproccessing sync primitives actually work over a network with almost no code changes, so you can horizontally scale with little effort... but then you're paying the serialization and network overhead, so ymmv. [0]: https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem...
shepardrtc 3y ago
Multiprocessing is great. Very easy to use and with the Manager(), passing data is easy enough, though very "slow" relatively speaking. It has to serialize the data, send it to the Manager process, then send to all the processes (I believe), and then deserialize it. Perhaps this nogil update can speed that up?
tga_d 3y ago
I have no experience with Windows, but at least on Linux, if the forking is handled intelligently (namely, late enough that identical assets don't need to be re-written), duplicated memory should be handled by Copy on Write.
- jmuhlich 3y ago
  CPython still needs to update reference counts after the fork even if your python code itself doesn't modify any data, making Copy on Write less helpful than it would first appear. Monolithic memory buffers like numpy arrays can be shared efficiently via CoW, but rich data structures with many PyObjects suffer from the refcounting issue.
- sanxiyn 3y ago
  Have you tried it? You are right in general, but even if you don't write anything in Python, CPython runtime constantly writes to heap, so everything is copied. You need specific support from CPython runtime to exploit copy-on-write. This is documented: https://docs.python.org/3/library/gc.html#gc.freeze https://docs.python.org/3/library/gc.html#gc.freeze For details, see "Copy-on-write friendly Python garbage collection" by Instagram Engineering: https://instagram-engineering.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf https://instagram-engineering.com/copy-on-write-friendly-pyt...
  - tga_d 3y ago
    I guess when GP said "heavy ... C extensions" I was thinking code pages from libraries and structures managed by C, yes python objects will require additional consideration.
- gjulianm 3y ago
  Multiprocessing in Python isn't exactly like forking on other languages. There's no shared Python memory, it's just a new interpreter instance, and then you have to use sharing mechanisms (shared arrays/values, pipes, queues) to transfer data explicitly.
  - 3y ago
gjulianm 3y ago
My experience with the multiprocessing module is exactly opposite. It's been a few years since I last used it seriously (I still use Python a lot) mainly because of how bad it was. The main issue is precisely communication between processes: Python uses the pickle module to serialize/deserialize data, which not only adds quite a bit of performance cost, but it's very hard to debug when for some reason some object is not supported by pickle (and it's a common occurrence).

by hgs3 3y ago

packetlost 3y ago

I agree that message passing can be a really good choice. The problem is the overhead of passing large messages can be really substantial. In general, you'll need to store messages in memory 4x per message (at least temporarily): once on each end for the Python representation and the serialized "on the wire" representation. There's also the issue with `pickle` generally being a horrible serialization format and being the default. There are ways around this, Python has a rather rudimentary and archaic shared memory subsystem[0] that can make this much more efficient. On the other hand, the Python multiproccessing sync primitives actually work over a network with almost no code changes, so you can horizontally scale with little effort... but then you're paying the serialization and network overhead, so ymmv. [0]: https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem...

shepardrtc 3y ago

Multiprocessing is great. Very easy to use and with the Manager(), passing data is easy enough, though very "slow" relatively speaking. It has to serialize the data, send it to the Manager process, then send to all the processes (I believe), and then deserialize it. Perhaps this nogil update can speed that up?

tga_d 3y ago

I have no experience with Windows, but at least on Linux, if the forking is handled intelligently (namely, late enough that identical assets don't need to be re-written), duplicated memory should be handled by Copy on Write.

jmuhlich 3y ago
CPython still needs to update reference counts after the fork even if your python code itself doesn't modify any data, making Copy on Write less helpful than it would first appear. Monolithic memory buffers like numpy arrays can be shared efficiently via CoW, but rich data structures with many PyObjects suffer from the refcounting issue.
sanxiyn 3y ago
Have you tried it? You are right in general, but even if you don't write anything in Python, CPython runtime constantly writes to heap, so everything is copied. You need specific support from CPython runtime to exploit copy-on-write. This is documented: https://docs.python.org/3/library/gc.html#gc.freeze https://docs.python.org/3/library/gc.html#gc.freeze For details, see "Copy-on-write friendly Python garbage collection" by Instagram Engineering: https://instagram-engineering.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf https://instagram-engineering.com/copy-on-write-friendly-pyt...
- tga_d 3y ago
  I guess when GP said "heavy ... C extensions" I was thinking code pages from libraries and structures managed by C, yes python objects will require additional consideration.
gjulianm 3y ago
Multiprocessing in Python isn't exactly like forking on other languages. There's no shared Python memory, it's just a new interpreter instance, and then you have to use sharing mechanisms (shared arrays/values, pipes, queues) to transfer data explicitly.
- 3y ago

gjulianm 3y ago

My experience with the multiprocessing module is exactly opposite. It's been a few years since I last used it seriously (I still use Python a lot) mainly because of how bad it was. The main issue is precisely communication between processes: Python uses the pickle module to serialize/deserialize data, which not only adds quite a bit of performance cost, but it's very hard to debug when for some reason some object is not supported by pickle (and it's a common occurrence).