3 ms·
The act of serializing data structures and reloading them can be a guard against long-term corruption. If you only use mmap, there's a risk of some corner of t
by barrkel 13y ago
The act of serializing data structures and reloading them can be a guard against long-term corruption.
If you only use mmap, there's a risk of some corner of the object graph getting subtly wrong owing to a bug in one version of the software, and never getting repaired.
Versioning of data structures is also a problem.
I'd leave this pattern for use cases for which copying of memory on load has a measurable impact on performance, and where required integrity of data isn't very high - ideally, the data should be able to be regenerated from primary sources.
One case I know of where this technique was used was the precompiled headers file for the Borland C++ compiler. There, it makes sense on both counts.
- pm215 13y agoThe "just mmap" pattern has also got portability issues if you might move the data file between systems which could have different endianness, pointer size, alignment requirements, etc. (Serialization done properly avoids this because you can swap to a consistent host-independent on-the-wire representation as part of the process.) This is also less of an issue for the cases you suggest where you can regenerate the data from a primary source -- you can just treat the host-dependent data file as a cache that can be blown away if moving to a different machine.
- mikeash 13y agoIsn't the potential for long-term corruption purely related to not rewriting the entire file from scratch every time you make changes? Even without mmap, any data store where you make incremental changes is potentially vulnerable to this sort of thing. One obvious example would be filesystems. They're basically specialized databases which treat your whole disk as a single gigantic file, and of course there's a long, proud history of programs used to repair corruption in them due to bugs or other problematic events. To me, that's an argument for not using formats with incremental formats if you can get away with rewriting the file each time, but once you have enough data to where you can't afford a total rewrite each time, does mmap make the problem any worse?
- barrkel 13y agoMost app file formats do not have the same reliability testing level as filesystems. Filesystems need years of use by brave people before they're considered OK for production. If you're updating files in place without a whole rewrite, then you are effectively writing a filesystem. I prefer to use a different approach where possible: write a diff log, and occasionally do a rewrite of the whole file. When you do have to treat the on-disk file as a filesystem, I'd explicitly treat it as such. Have a low-level structure that does not need modification for different versions, is portable and highly tested. Do application-level structure with normal read-modify-write, but layered on top of the virtual filesystem layer. Consider having two modes, one where the virtual filesystem just proxies the real filesystem, and instead of a "file", you have a directory structure on disk. Consider using an existing component for this (zip archive libraries are popular, but not particularly efficient for writing). Mmap does make the problem worse. It encourages you to be lazy towards versioning and portability.