3 ms·
That's like claiming the ability to share JS ArrayBuffer's between workers supplants the need for the ability to share high level objects like a Map. What I'm
by goldenkey 5y ago
That's like claiming the ability to share JS ArrayBuffer's between workers supplants the need for the ability to share high level objects like a Map.
What I'm getting at is that JS had the exact same problem, and it offers the same ability for sharing blocks of raw memory. But the same pitfalls for any actual structured object.
Instead, we have to serialize/unserialize (creating a copy) from a separate worker, process, or disk source, it's performance is shite for objects of large size.
Message passing is not ideal for large pieces of data. Shared memory is hundreds of times faster.
- BiteCode_dev 5y agoYes. SharedMemory can share list (https://docs.python.org/3/library/multiprocessing.shared_memory.html#multiprocessing.shared_memory.ShareableList https://docs.python.org/3/library/multiprocessing.shared_mem...) but that's it. That's why diskscache is better as an alternative for complex objects: https://pypi.org/project/diskcache/ https://pypi.org/project/diskcache/ It's not as good as a simple shared variable, but it's surprisingly performant, since it uses memory-mapped files.
- goldenkey 5y agoI see what you mean. Does diskcache still have to do deserialization of the mapped data? Nested objects (large disparate heap pointer graphs) would suffer performance penalties.
- BiteCode_dev 5y agoIt does yes, although pickle is pretty fast, it's still a penalty.