4 ms·
Depends on what you're doing. Shoving a numpy array with 10^7 elements over the wire isn't going to scale nicely at all. There are still a lot of computing scen
by freyrs3 14y ago
Depends on what you're doing. Shoving a numpy array with 10^7 elements over the wire isn't going to scale nicely at all. There are still a lot of computing scenarios having shared memory is the best approach.
- jandrewrogers 14y agoYou do not need threads for shared memory. I know of quite a few high-performance software systems that use process isolation and yet leverage shared memory via mechanisms like mmap() for IPC.
- seunosewa 14y agoGood point, but shared memory via mmap isn't very Pythonic since you have to manage the shared memory manually, you can't allocate Python objects on the shared region, etc.
- freyrs3 14y agoYes, but if you're doing HPC idiomatic Python comes second to raw performance.
- d0mine 14y agoYou don't need to use mmap explicitly. multiprocessing.Array and numpy arrays can share the same memory that can be used from different processes e.g., http://stackoverflow.com/a/9849971 http://stackoverflow.com/a/9849971
- exDM69 14y agoA context switch from one thread to another is less expensive than from one process to another. And there are other advantages to running in a shared memory address space. It tends to make some things complicated, but if getting the most out of your CPUs is the goal, threads are the way to go.
- d0mine 14y agoOn some systems (Linux) there is no difference once threads/processes are initialized. Simple benchmark confirms it (see my comment about mmap).