I agree that message passing can be a really good choice. The problem is the overhead of passing large messages can be really substantial. In general, you'll ne

I agree that message passing can be a really good choice. The problem is the overhead of passing large messages can be really substantial. In general, you'll need to store messages in memory 4x per message (at least temporarily): once on each end for the Python representation and the serialized "on the wire" representation. There's also the issue with `pickle` generally being a horrible serialization format and being the default. There are ways around this, Python has a rather rudimentary and archaic shared memory subsystem[0] that can make this much more efficient. On the other hand, the Python multiproccessing sync primitives actually work over a network with almost no code changes, so you can horizontally scale with little effort... but then you're paying the serialization and network overhead, so ymmv. [0]: https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem...