4 ms·
Looks to me like yet another reason to use sqlite instead of «flat files». E.g postgres seems like a many orders of magnitude increase in complexity. Possibly
by pletnes 9y ago
Looks to me like yet another reason to use sqlite instead of «flat files». E.g postgres seems like a many orders of magnitude increase in complexity.
Possibly related: We’ve been using hdf5 for a lot of data storage at work (raw image data). I often discover corrupt files, even though we (think we) are flushing files etc. I’d love to see some work on reliability there, but it’s hard to know if the article is relevant to those issues.
Also, what happens when you’re on RAID? Even more assumptions out the window I’d imagine?
- zejn 9y agosqlite also uses fsync, which at least on Linux has corner cases where errors don't even get reported. When fsync does report error, you should panic, as even if the subsequent fsync returns ok, that does not mean the data for which there was an error was actually written, in fact the opposite is true - kernel clears the dirty bit on non-written data thus neither the application nor kernel know which data is not written.
- nordsieck 9y ago> sqlite also uses fsync At least sqlite has done the hard work of figuring out how to be as reliable as possible. At some level, writing to disk is a bit like using crypto. It's a lot easier to get right if you use well tested libraries that only offer high level primitives.