We do the same doing some terabyte-scale data processing. We try and load all of the shared state in to a read-only structure ahead of time, then fork off the w

We do the same doing some terabyte-scale data processing. We try and load all of the shared state in to a read-only structure ahead of time, then fork off the workers. This way, copy-on-write memory architectures get the advantage of only persisting the data once, yet we get multiple processes working from a very very large shared set of data, all without having to do any sort of IPC or locking. The only locks we used are around streams used for a very basic sort of IPC used to communicate to shared stream controllers like logfiles - these do perform very poorly compared to 'proper' parallelism but they make up for a very tiny fraction of the total workload. We've found that this will beat a 'better' parallelism even if we need to do a lot of extra or redundant work due to the lack of IPC or reduction due to the no-overhead approach, although this will of course be highly dependent on the problem at hand.