3 ms·

But the spec doesn't imply in any way that you'll need to write() again if fsync() fails. And dropping dirty flags seems way out of spec because now you can re

But the spec doesn't imply in any way that you'll need to write() again if fsync() fails. And dropping dirty flags seems way out of spec because now you can read data from cache that is not on disk and never will be even if future fsync() succeeds. So I don't buy the spec argument. You have me curious now... I don't know anything about LMDB but I wonder if its msync()-based design really is immune... Could there be a way for an IO error to leave you with a page bogusly marked clean, which differs from the data on disk?

hyc_symas 8y ago
The spec leaves the system condition undefined after an fsync failure. The safe thing to do is assume everything failed and nothing was written. That's what LMDB does. Expecting anything else would be relying on implementation-specific knowledge, which is always a bad idea. > I don't know anything about LMDB but I wonder if its msync()-based design really is immune. By default, LMDB doesn't use msync, so this isn't really a realistic scenario. If there is an I/O error that the OS does not report, then sure, it's possible for LMDB to have a corrupted view of the world. But that would be an OS bug in the first place. Since we're talking about buffered writes - clearly it's possible for a write() to return success before its data actually gets flushed to disk. And it's possible for a background flush to occur independently of the app calling fsync(). The docs specifically state, if an I/O error occurs during writeback, it will be reported on all file descriptors open on that file. So assuming the OS doesn't have a critical bug here, no, there's no way for an I/O error to leave LMDB with an invalid view of the world.
- nh2 8y ago
  > The spec leaves the system condition undefined [...]. The safe thing to do is assume everything failed This is key. Often programmers do 'assumption based programming'. "Surely the function will do X, it's the only reasonable thing to do, right?". As much it is human, this is bad practice and leads to unreliable systems. If the spec doesn't say it, don't assume anything about it, and keep asking. To show that this approach is feasible for anyone, here is an example: Recently I needed to write an fsync-safe application. The question of whether close()+re-open()+fsync() is safe came up. I found it had been asked on StackOverflow (https://stackoverflow.com/questions/37288453/calling-fsync2-after-close2/50158433 https://stackoverflow.com/questions/37288453/calling-fsync2-...) but received no answers for a year. I put a 100-reputation bounty on it and quickly got a competent reply quoting the spec and summarising: > It is not 100% clear whether the 'all currently queued I/O operations associated with the file indicated by [the] file descriptor' applies across processes. Conceptually, I think it should, but the wording isn't there in black and white. With the spec being unclear, I took the question to the kernel developers (https://marc.info/?m=152535098505449 https://marc.info/?m=152535098505449), and was immediately pointed at the Postgres fsyncgate results. So by spending a few hours on not believing what I wished was true, I managed to avoid writing an unsafe system. Always ask those in the know (specs and people). Never assume.

hyc_symas 8y ago

The spec leaves the system condition undefined after an fsync failure. The safe thing to do is assume everything failed and nothing was written. That's what LMDB does. Expecting anything else would be relying on implementation-specific knowledge, which is always a bad idea. > I don't know anything about LMDB but I wonder if its msync()-based design really is immune. By default, LMDB doesn't use msync, so this isn't really a realistic scenario. If there is an I/O error that the OS does not report, then sure, it's possible for LMDB to have a corrupted view of the world. But that would be an OS bug in the first place. Since we're talking about buffered writes - clearly it's possible for a write() to return success before its data actually gets flushed to disk. And it's possible for a background flush to occur independently of the app calling fsync(). The docs specifically state, if an I/O error occurs during writeback, it will be reported on all file descriptors open on that file. So assuming the OS doesn't have a critical bug here, no, there's no way for an I/O error to leave LMDB with an invalid view of the world.

nh2 8y ago
> The spec leaves the system condition undefined [...]. The safe thing to do is assume everything failed This is key. Often programmers do 'assumption based programming'. "Surely the function will do X, it's the only reasonable thing to do, right?". As much it is human, this is bad practice and leads to unreliable systems. If the spec doesn't say it, don't assume anything about it, and keep asking. To show that this approach is feasible for anyone, here is an example: Recently I needed to write an fsync-safe application. The question of whether close()+re-open()+fsync() is safe came up. I found it had been asked on StackOverflow (https://stackoverflow.com/questions/37288453/calling-fsync2-after-close2/50158433 https://stackoverflow.com/questions/37288453/calling-fsync2-...) but received no answers for a year. I put a 100-reputation bounty on it and quickly got a competent reply quoting the spec and summarising: > It is not 100% clear whether the 'all currently queued I/O operations associated with the file indicated by [the] file descriptor' applies across processes. Conceptually, I think it should, but the wording isn't there in black and white. With the spec being unclear, I took the question to the kernel developers (https://marc.info/?m=152535098505449 https://marc.info/?m=152535098505449), and was immediately pointed at the Postgres fsyncgate results. So by spending a few hours on not believing what I wished was true, I managed to avoid writing an unsafe system. Always ask those in the know (specs and people). Never assume.