3 ms·
If you want to limit yourself to a single machine: isn't that what files are for? Sharing large data between processes?
by e12e 9y ago
If you want to limit yourself to a single machine: isn't that what files are for? Sharing large data between processes?
- amelius 9y agoSometimes you want to set up an efficient processing pipeline on a single machine, e.g. when writing a database engine. In that case, you don't want to serialize all communication between processes. Of course, other processes (not related to the efficient pipeline) can live on different machines.
- zbentley 9y agoWho said anything about serializing? To coordinate multiple processes via a file-backed interface, you still might have to wait for the disk (unless you're using /dev/shm), but you don't necessarily have to serialize/deserialize data. Most people tend to confuse the two. You can cast bytes from a memory-mapped file just fine, if you trust the file/other writers, or are willing to sacrifice some safety. There are also a lot of serialization systems that are simple/fast enough as to basically be structs, e.g. flatbuffers: https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/. Even those still don't avoid that pesky copy, though; for that you'll have to interact with raw memory and hope it's in the right layout for your purposes.