3 ms·
Does not sound odd, but in your case you will store results on disk, rather than in memory? That means that there might be some overhead in (de-)serialization a
by leethargo 8y ago
Does not sound odd, but in your case you will store results on disk, rather than in memory? That means that there might be some overhead in (de-)serialization and also some restrictions on what kind of data can be stored as an intermediate result?
- jacobush 8y agoProbably but it's not a given. It depends on the tools. The tools might work with memory mapped data structures directly.
- AboutTheWhisles 8y agoYou can mitigate the reading from disk by memory mapping the file. I don't think serialization is a problem either. It is possible to make data structures that always exist as a single span of memory, so that they don't have a separate serialized representation. Most data structures are better served like this anyway, since prefetching and linear memory access is a huge speedup over pointer chasing. Building a data structure out of many separate heap allocations will also slow it down.
- gnulinux 8y agoYou can memory map and store binary, in-memory data (as long as you don't use pointers). So there'll be some overhead serializing (removing pointers) but ime it's not noticeable since most serialization tasks can be done in milliseconds (even if the dataset is Terabytes) whereas most algorithmic tasks are in the order of seconds, minutes or hours.