3 ms·

In the reported case, fsync reported an error, then (more data may or may not have been written), then fsync is tried again and reports a success, which masks t

In the reported case, fsync reported an error, then (more data may or may not have been written), then fsync is tried again and reports a success, which masks the fact that data from the previous fsync didn't get fully written. As I already wrote - in LMDB, after fsync fails, the transaction is aborted, so none of the partially written data matters.

jhallenworld 8y ago
Hold on, they are saying that if sync fails (for example if someone types "sync" at the console), then the database calls fsync() it will not fail even though the data is gone. I don't see how any database the uses the buffer cache could guard against this case. The kernel should never do this. If sync fails, all future syncs should also fail. This could be relaxed in various ways: sync fails, but we record which files have missing data, so any fsync for just those files also fails. (Otherwise I agree with LMDB- there should be no retry business on fsync fails).
- hyc_symas 8y ago
  You're right, that was another error case in the article that I missed the first time. In LMDB's case you'd need to be pretty unlucky for that bug to bite; all the writes are done at once during txn_commit so you have to issue sync() during the couple milliseconds that might take, before the fsync() at the end of the writes. Also, it has to be a single-sector failure, otherwise a write after the sync will still fail and be caught. E.g. LMDB other write write write sync write fsync If the device is totally hosed, all of those writes' destination pages would error out. In that case, the failure can only be missed if you're unlucky enough for the sync to start immediately after the last write and before the fsync. The window is on the order of nanoseconds. If only a single page is bad, and the majority of the writes are ok, then you still have to be unlucky enough for the sync to run after the bad write; if it runs before the bad write then fsync will still see the error.
  - deleted 8y ago
    [deleted]

jhallenworld 8y ago

Hold on, they are saying that if sync fails (for example if someone types "sync" at the console), then the database calls fsync() it will not fail even though the data is gone. I don't see how any database the uses the buffer cache could guard against this case. The kernel should never do this. If sync fails, all future syncs should also fail. This could be relaxed in various ways: sync fails, but we record which files have missing data, so any fsync for just those files also fails. (Otherwise I agree with LMDB- there should be no retry business on fsync fails).

hyc_symas 8y ago
You're right, that was another error case in the article that I missed the first time. In LMDB's case you'd need to be pretty unlucky for that bug to bite; all the writes are done at once during txn_commit so you have to issue sync() during the couple milliseconds that might take, before the fsync() at the end of the writes. Also, it has to be a single-sector failure, otherwise a write after the sync will still fail and be caught. E.g. LMDB other write write write sync write fsync If the device is totally hosed, all of those writes' destination pages would error out. In that case, the failure can only be missed if you're unlucky enough for the sync to start immediately after the last write and before the fsync. The window is on the order of nanoseconds. If only a single page is bad, and the majority of the writes are ok, then you still have to be unlucky enough for the sync to run after the bad write; if it runs before the bad write then fsync will still see the error.
- deleted 8y ago
  [deleted]