4 ms·

To me, the headline feature for Python 3.8 is shared memory for multiprocessing (contributed by Davin Potts). Some kinds of data can be passed back and forth b

To me, the headline feature for Python 3.8 is shared memory for multiprocessing (contributed by Davin Potts). Some kinds of data can be passed back and forth between processes with near zero overhead (no pickling, sockets, or unpickling). This significantly improves Python's story for taking advantage of multiple cores.

aportnoy 7y ago
I’ve been waiting for this for a very long time. Thank you for mentioning this. Would this work with e.g. large NumPy arrays? (and this is Raymond Hettinger himself, wow)
- solarist 7y ago
  An alternative you may want is Dask.
  - aportnoy 7y ago
    Dask doesn’t support shared memory without pickling because Python doesn’t.
aidos 7y ago
Oh no way. That has huge potential. What are the limitations?
acqq 7y ago
For us who didn't follow: "multiprocessing.shared_memory — Provides shared memory for direct access across processes" https://docs.python.org/3.9/library/multiprocessing.shared_memory.html https://docs.python.org/3.9/library/multiprocessing.shared_m... And it has the example which "demonstrates a practical use of the SharedMemory class with NumPy arrays, accessing the same numpy.ndarray from two distinct Python shells." Also, SharedMemory "Creates a new shared memory block or attaches to an existing shared memory block. Each shared memory block is assigned a unique name. In this way, one process can create a shared memory block with a particular name and a different process can attach to that same shared memory block using that same name. As a resource for sharing data across processes, shared memory blocks may outlive the original process that created them. When one process no longer needs access to a shared memory block that might still be needed by other processes, the close() method should be called. When a shared memory block is no longer needed by any process, the unlink() method should be called to ensure proper cleanup." Really nice.
amelius 7y ago
> no pickling, sockets, or unpickling But still copying? If not, then how does it interoperate with garbage collection?
- fulafel 7y ago
  It works with the memoryview/buffer interface, so you can have eg a Numpy array backed by a sharedmemory object attached to a named SM segment. So it's not for containing normal Python dicts, strings etc that are individually tracked by GC. https://docs.python.org/3.8/library/multiprocessing.shared_memory.html https://docs.python.org/3.8/library/multiprocessing.shared_m...
gigatexal 7y ago
Agreed this is huge.
agent008t 7y ago
Isn't that already the case? I thought that when you use multiprocessing in Python, a new process gets forked, and while each new process has separate virtual memory, that virtual memory points to the same physical location until the process tries to write to it (i.e. copy-on-write)?
- jashmatthews 7y ago
  That's true but running VMs mutate their heaps, both managed and malloced. CoW also only works from parent to child. You can't share mutable memory this way. Empty space in internal pages gets used allocating new objects, refence counts updated or GC flags get flipped etc, and it just takes one write in each 4kb page to trigger a whole page copy. It doesn't take long before a busy web worker etc will cause a huge chunk of the memory to be copied into the child. There are definitely ways to make it much more effective like this work by Instagram that went into Python 3.7: https://instagram-engineering.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf https://instagram-engineering.com/copy-on-write-friendly-pyt...
- amelius 7y ago
  Yes, the problem is sharing data between parent and child after the parent process has been forked.
- sametmax 7y ago
  If you have 4 cores, you may want to spaw 4 children, then share stuff between them. Not just top-down. E.G: live settings, cached values, white/black lists, etc
- paulddraper 7y ago
  Yes, sharing pre-fork data is as old as fork(). Sharing post-fork data is where it gets interesting.
quietbritishjim 7y ago
It looks like this will make efficient data transfer much more convenient, but it's worth noting this had always been possible with some manual effort. Python has had `mmap` support at least as long ago as Python 2.7, which works fine for zero-overhead transfer of data. With mmap you have to specify a file name (actually a file number), but so long as you set the length to zero before you close it there's no reason any data would get written to disk. On Unix you can even unlink the file before you start writing it if you wish, or create it with the tempfile module and never give it a file name at all (although this makes it harder to open in other processes as they can't then just mmap by file name). The mmap object satisfies the buffer protocol so you can create numpy arrays that directly reference the bytes in it. The memory-mapped data can be shared between processes regardless of whether they use the multiprocessing module or even whether they're all written in Python. https://docs.python.org/3.7/library/mmap.html https://docs.python.org/3.7/library/mmap.html
- mncharity 7y ago
  Also on linux is sysv_ipc.SharedMemory.

aportnoy 7y ago

I’ve been waiting for this for a very long time. Thank you for mentioning this. Would this work with e.g. large NumPy arrays? (and this is Raymond Hettinger himself, wow)

solarist 7y ago
An alternative you may want is Dask.
- aportnoy 7y ago
  Dask doesn’t support shared memory without pickling because Python doesn’t.

aidos 7y ago

Oh no way. That has huge potential. What are the limitations?

acqq 7y ago

For us who didn't follow: "multiprocessing.shared_memory — Provides shared memory for direct access across processes" https://docs.python.org/3.9/library/multiprocessing.shared_memory.html https://docs.python.org/3.9/library/multiprocessing.shared_m... And it has the example which "demonstrates a practical use of the SharedMemory class with NumPy arrays, accessing the same numpy.ndarray from two distinct Python shells." Also, SharedMemory "Creates a new shared memory block or attaches to an existing shared memory block. Each shared memory block is assigned a unique name. In this way, one process can create a shared memory block with a particular name and a different process can attach to that same shared memory block using that same name. As a resource for sharing data across processes, shared memory blocks may outlive the original process that created them. When one process no longer needs access to a shared memory block that might still be needed by other processes, the close() method should be called. When a shared memory block is no longer needed by any process, the unlink() method should be called to ensure proper cleanup." Really nice.

amelius 7y ago

> no pickling, sockets, or unpickling But still copying? If not, then how does it interoperate with garbage collection?

fulafel 7y ago
It works with the memoryview/buffer interface, so you can have eg a Numpy array backed by a sharedmemory object attached to a named SM segment. So it's not for containing normal Python dicts, strings etc that are individually tracked by GC. https://docs.python.org/3.8/library/multiprocessing.shared_memory.html https://docs.python.org/3.8/library/multiprocessing.shared_m...

gigatexal 7y ago

Agreed this is huge.

agent008t 7y ago

Isn't that already the case? I thought that when you use multiprocessing in Python, a new process gets forked, and while each new process has separate virtual memory, that virtual memory points to the same physical location until the process tries to write to it (i.e. copy-on-write)?

jashmatthews 7y ago
That's true but running VMs mutate their heaps, both managed and malloced. CoW also only works from parent to child. You can't share mutable memory this way. Empty space in internal pages gets used allocating new objects, refence counts updated or GC flags get flipped etc, and it just takes one write in each 4kb page to trigger a whole page copy. It doesn't take long before a busy web worker etc will cause a huge chunk of the memory to be copied into the child. There are definitely ways to make it much more effective like this work by Instagram that went into Python 3.7: https://instagram-engineering.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf https://instagram-engineering.com/copy-on-write-friendly-pyt...
amelius 7y ago
Yes, the problem is sharing data between parent and child after the parent process has been forked.
sametmax 7y ago
If you have 4 cores, you may want to spaw 4 children, then share stuff between them. Not just top-down. E.G: live settings, cached values, white/black lists, etc
paulddraper 7y ago
Yes, sharing pre-fork data is as old as fork(). Sharing post-fork data is where it gets interesting.

quietbritishjim 7y ago

It looks like this will make efficient data transfer much more convenient, but it's worth noting this had always been possible with some manual effort. Python has had `mmap` support at least as long ago as Python 2.7, which works fine for zero-overhead transfer of data. With mmap you have to specify a file name (actually a file number), but so long as you set the length to zero before you close it there's no reason any data would get written to disk. On Unix you can even unlink the file before you start writing it if you wish, or create it with the tempfile module and never give it a file name at all (although this makes it harder to open in other processes as they can't then just mmap by file name). The mmap object satisfies the buffer protocol so you can create numpy arrays that directly reference the bytes in it. The memory-mapped data can be shared between processes regardless of whether they use the multiprocessing module or even whether they're all written in Python. https://docs.python.org/3.7/library/mmap.html https://docs.python.org/3.7/library/mmap.html

mncharity 7y ago
Also on linux is sysv_ipc.SharedMemory.