4 ms·
" the question 'are these exact bits irrevocably committed to the media?' is pretty much impossible to answer" Well, kind of. fsync()'s job is only to ensure c
by throwaway09223 3y ago
" the question 'are these exact bits irrevocably committed to the media?' is pretty much impossible to answer"
Well, kind of. fsync()'s job is only to ensure caches are written to durable storage. Its job is not to ensure the integrity of durable storage. Your idea of fsync()'s "Right Thing" is not quite correct, because these types of durable storage failures can happen outside of the context of a write -- so it's a bit silly to point a finger at fsync().
For example, you can write bytes to disk today and the drive might experience corruption on that sector next week.
"(and even then, I would not bet my life on it)"
All hardware fails, so of course you shouldn't bet your life on it. That's why we have both onsite and offsite backups. The durable media itself can always fail and lose data -- even if there aren't any writes occurring. Data loss can even occur on a powered off system.
Again, fsync()'s job is only to ensure the data is moved to durable storage. Solving the problem of data loss is done in other ways -- for example raid5 addresses the problem of durable storage failing by using parity and extra copies.
The real takeaway from the blog is perhaps that Raft as a protocol is inadequate, in the same way a raid1 mirror system is inadequate in terms of protecting from data corruption on durable storage (which of the two mirrored drives has the correct version of the data?). I'm frankly surprised that this situation is undetectable. It's a solved problem in other storage layers.
"I'll probably stick to SQLite write-ahead logs, at least I can wrap my mind around those"
Well, keep in mind: those have the same problem in the event of durable storage failure.
- Siecje 3y ago> Again, fsync()'s job is only to ensure the data is moved to durable storage. The problem is that every layer in the stack lies. They say "yes it is definitely written to durable storage" when it is just in some cache layer and about to be written.
- dboreham 3y ago> it is just in some cache layer and about to be written Definitely not. I get that sometimes we find things that do lie, but lying about this is a huge P0 bug and everyone with an interest in data storage knows to be on the lookout for such brokenness. E.g. such people do not buy SSD that lack some sort of power fail flush protection.
- throwaway09223 3y ago"The problem is that every layer in the stack lies. " Not enterprise gear. Reliable storage does exist. I have tested many vendors myself, and gone through spec sheets under NDA (as mentioned above). "They say "yes it is definitely written to durable storage" when it is just in some cache layer and about to be written." Enterprise hardware contains batteries specifically designed so that caches can still be written out to durable storage in the event of power loss. Have you ever dealt with managing battery learning cycles on a Dell PERC?
- Proven 3y ago[dead]