7 ms·

> Doesn't this affect all databases? Didn't affect LMDB. If an fsync fails the entire transaction is aborted/discarded. Retrying was always inherently OS-depen

> Doesn't this affect all databases? Didn't affect LMDB. If an fsync fails the entire transaction is aborted/discarded. Retrying was always inherently OS-dependent and unreliable, better to just toss it all and start over. Any dev who actually RTFM'd and followed POSIX specs would have been fine. LMDB's crash reliability is flawless.

evdubs 8y ago
Mr. Chu, I hope you never lose your tenacity with respect to writing blurbs on LMDB's performance and reliability. I have enjoyed the articles comparing LMDB with other databases' performance and hope you continue to point the spotlight on the superior design decisions of LMDB.
- hyc_symas 8y ago
  Thanks, glad you're getting something out of those writeups. Hopefully it helps other developers learn a better path.
  - diroussel 8y ago
    What if another process calls fsync? It will get the error. Then when LMDB calls fsync no error will be reported. And thus the transaction will not be retried. Is this scenario dealt with?
    - anarazel 8y ago
      Newer versions of linux (but not plenty of other OSs) guarantee that each write failure will be signalled once to each file descriptor open before the error occurred. So that ought to be handled, unless the FD is closed inbetween.
    - hyc_symas 8y ago
      LMDB is single-writer. Only one process can acquire the writelock at a time, and only the writer can call fsync() so this scenario doesn't arise. If you open the datafile with write access from another process without using the LMDB API, you're intentionally shooting yourself in the foot and all bets are off.
      - jorangreef 8y ago
        On some systems, fsync is system-wide. Another process fsyncing an unrelated file descriptor can still consume the error meant for the LMDB file descriptor. Same thing goes for a user running the sync command from the terminal. A write lock won't protect you from this, unless you can prevent all other processes from calling fsync. It's got nothing to do with opening the LMDB datafile concurrently. If you share a physical disk device with any other process, you're at risk.
        hyc_symas 8y ago
        Yes, acknowledged. https://news.ycombinator.com/item?id=19128325 https://news.ycombinator.com/item?id=19128325 fsync() is not documented to be system-wide (while sync() is). That behavior would also be an OS bug. The question this person asked was specifically about fsync().
        jorangreef 8y ago
        Sorry, that was confusing, I meant "system-wide" as in file system, not OS, i.e. "if you share a physical disk device with any other process". When you flush a particular FD, some file systems just flush every dirty buffer for the entire disk. I wouldn't actually be surprised though if there are some kernels that flush all disks either, regardless of whether it's considered a bug or not. "The question this person asked was specifically about fsync()." Sure, but as you acknowledged elsewhere, if sync is called, it flushes everything, and this impacts upon the person who "asked specifically about fsync" since there's a chance on some kernels that sync will eat the error that fsync was expecting.
macdice 8y ago
But the spec doesn't imply in any way that you'll need to write() again if fsync() fails. And dropping dirty flags seems way out of spec because now you can read data from cache that is not on disk and never will be even if future fsync() succeeds. So I don't buy the spec argument. You have me curious now... I don't know anything about LMDB but I wonder if its msync()-based design really is immune... Could there be a way for an IO error to leave you with a page bogusly marked clean, which differs from the data on disk?
- hyc_symas 8y ago
  The spec leaves the system condition undefined after an fsync failure. The safe thing to do is assume everything failed and nothing was written. That's what LMDB does. Expecting anything else would be relying on implementation-specific knowledge, which is always a bad idea. > I don't know anything about LMDB but I wonder if its msync()-based design really is immune. By default, LMDB doesn't use msync, so this isn't really a realistic scenario. If there is an I/O error that the OS does not report, then sure, it's possible for LMDB to have a corrupted view of the world. But that would be an OS bug in the first place. Since we're talking about buffered writes - clearly it's possible for a write() to return success before its data actually gets flushed to disk. And it's possible for a background flush to occur independently of the app calling fsync(). The docs specifically state, if an I/O error occurs during writeback, it will be reported on all file descriptors open on that file. So assuming the OS doesn't have a critical bug here, no, there's no way for an I/O error to leave LMDB with an invalid view of the world.
  - nh2 8y ago
    > The spec leaves the system condition undefined [...]. The safe thing to do is assume everything failed This is key. Often programmers do 'assumption based programming'. "Surely the function will do X, it's the only reasonable thing to do, right?". As much it is human, this is bad practice and leads to unreliable systems. If the spec doesn't say it, don't assume anything about it, and keep asking. To show that this approach is feasible for anyone, here is an example: Recently I needed to write an fsync-safe application. The question of whether close()+re-open()+fsync() is safe came up. I found it had been asked on StackOverflow (https://stackoverflow.com/questions/37288453/calling-fsync2-after-close2/50158433 https://stackoverflow.com/questions/37288453/calling-fsync2-...) but received no answers for a year. I put a 100-reputation bounty on it and quickly got a competent reply quoting the spec and summarising: > It is not 100% clear whether the 'all currently queued I/O operations associated with the file indicated by [the] file descriptor' applies across processes. Conceptually, I think it should, but the wording isn't there in black and white. With the spec being unclear, I took the question to the kernel developers (https://marc.info/?m=152535098505449 https://marc.info/?m=152535098505449), and was immediately pointed at the Postgres fsyncgate results. So by spending a few hours on not believing what I wished was true, I managed to avoid writing an unsafe system. Always ask those in the know (specs and people). Never assume.
grandinj 8y ago
in the reported case, fsync does not persist all data to disk, but it reports success. How does LMDB deal with that situation?
- hyc_symas 8y ago
  In the reported case, fsync reported an error, then (more data may or may not have been written), then fsync is tried again and reports a success, which masks the fact that data from the previous fsync didn't get fully written. As I already wrote - in LMDB, after fsync fails, the transaction is aborted, so none of the partially written data matters.
  - jhallenworld 8y ago
    Hold on, they are saying that if sync fails (for example if someone types "sync" at the console), then the database calls fsync() it will not fail even though the data is gone. I don't see how any database the uses the buffer cache could guard against this case. The kernel should never do this. If sync fails, all future syncs should also fail. This could be relaxed in various ways: sync fails, but we record which files have missing data, so any fsync for just those files also fails. (Otherwise I agree with LMDB- there should be no retry business on fsync fails).
    - hyc_symas 8y ago
      You're right, that was another error case in the article that I missed the first time. In LMDB's case you'd need to be pretty unlucky for that bug to bite; all the writes are done at once during txn_commit so you have to issue sync() during the couple milliseconds that might take, before the fsync() at the end of the writes. Also, it has to be a single-sector failure, otherwise a write after the sync will still fail and be caught. E.g. LMDB other write write write sync write fsync If the device is totally hosed, all of those writes' destination pages would error out. In that case, the failure can only be missed if you're unlucky enough for the sync to start immediately after the last write and before the fsync. The window is on the order of nanoseconds. If only a single page is bad, and the majority of the writes are ok, then you still have to be unlucky enough for the sync to run after the bad write; if it runs before the bad write then fsync will still see the error.
garyclarke27 8y ago
LMDB claimed speed and reliability seems remarkable (from a quick glance). I would guess is easier to achieve such, for a KV store, than for much more complex Relational Database. Got me thinking though. Mayby Postgres could take advantage of LMDB? Mayby by using LMDB as it’s cache? instead of using OS page cache, maybe writing the WAL to LMDB?
- lmb 8y ago
  LMDB is not a great fit for something like the WAL, where new data is written at the end, and old data discarded at the start. It leads to fragmentation (especially if WAL entries are larger than a single page).
- ddorian43 8y ago
  Maybe with https://github.com/kellabyte/rewind https://github.com/kellabyte/rewind which should support WAL for lmdb
- hyc_symas 8y ago
  LMDB itself only uses the OS page cache. The way for LMDB to improve an RDBMS is for it to replace the existing row and index store, and eliminate any WAL. This is what SQLightning does with SQLite. Have looked at replacing InnoDB in MySQL, but that code is a lot harder to read, so it's been slow going. Postgres doesn't have a modular storage interface, so it would be even uglier to overhaul.
  - garyclarke27 8y ago
    Thanks, make sense, I think Postgres are planning go have pluggable storage interface in nest version 12, would that help? Also nobody has mention data checksum added v9.3, do you know if this helps avoid this kind of fsync related corruption?
    - anarazel 8y ago
      > Also nobody has mention data checksum added v9.3, do you know if this helps avoid this kind of fsync related corruption? Not really, I think. Page-level checksums don't protect against entire writes going missing, unfortunately.
lmb 8y ago
We've been using LMDB at Cloudflare to store small-ish configuration data, it has been rock solid. Thank you and the rest of the contributors for such a great library.
pgaddict 8y ago
I'm pretty sure it does affect any database/application relying on buffered I/O. Even if you use the fsync() interface correctly, you're still affected by the bugs in error reporting.
- hyc_symas 8y ago
  Technically there is no bug in error reporting. fsync() reported an error as it should. The application continued processing, instead of stopping. fsync() didn't report the same error a second time, which leads to the app having problems. The application should have stopped the first time fsync() reported an error. LMDB does this, instead of futile retry attempts that Postgres et al do. Fundamentally, a library should not automatically retry anything - it should surface all errors back to the caller and let the caller decide what to do next. That's just good design.
  - pgaddict 8y ago
    No. Kernels before 4.13 may or may not report the fsync error correctly, depending on various conditions. There are more details in the talk [1] I posted earlier, and in the LWN articles related to this issue. [1] https://www.youtube.com/watch?v=74c19hwY2oE https://www.youtube.com/watch?v=74c19hwY2oE
    - hyc_symas 8y ago
      Ah I see. https://lwn.net/Articles/718734/ https://lwn.net/Articles/718734/ Someone issuing a sync() could cause an error to be cleared before the app's fsync() happens. That's a drag.
mehrdadn 8y ago
> Any dev who actually RTFM'd and followed POSIX specs would have been fine. So I'm trying to do that [1] and it seems to me the documentation directly implies that a second successful call to fsync() necessarily implies that all data was transferred, even if the previous call to fsync() had failed. I say this because the sentence says "all data for the open file descriptor" is to be transferred, not merely "all data written since the previous fsync to this file descriptor". It follows that any data not transferred in the previous call due to an error ("outstanding I/O operations are not guaranteed to have been completed") must now be transferred if this call is successful. What am I missing? [1] https://pubs.opengroup.org/onlinepubs/009695399/functions/fsync.html https://pubs.opengroup.org/onlinepubs/009695399/functions/fs...
- hyc_symas 8y ago
  Technically, there is nothing besides "all data written since the previous fsync" since the previous fsync already wrote all the outstanding data at that point in time. (Even if it actually failed.) I.e., there is never any to-be-written data remaining after fsync returns. Everything that was queued to be flushed was flushed and dequeued. Whether any particular write failed or not doesn't change that fact.
  - mehrdadn 8y ago
    > the previous fsync already wrote all the outstanding data at that point in time. (Even if it actually failed.) I'm sorry, but this... just doesn't make sense. An event can't both happen and also fail to happen. Also, the queue business is an implementation detail. Our debate is over what the spec mandates, not how a particular implementation behaves.
    - hyc_symas 8y ago
      The fsync() function shall request that all data for the open file descriptor named by fildes is to be transferred to the storage device associated with the file described by fildes. The nature of the transfer is implementation-defined. The fsync() function shall not return until the system has completed that action or until an error is detected. All of the data was transferred to the device. The device may have failed to persist some or all of what was transferred, but it all got transferred. There's no mention of the OS retrying, or leaving the system in a state that a subsequent fsync can retry from where it left off. So you can't assume anything along those lines.
loeg 8y ago
> Any dev who actually RTFM'd and followed POSIX specs would have been fine. Yeesh. The POSIX manual on fsync is intentionally vague to hide cross-platform differences. There are basically no guarantees once an error happens. I guess that's one interpretation of RTFMing, but... clearly it doesn't match user expectations.

evdubs 8y ago

Mr. Chu, I hope you never lose your tenacity with respect to writing blurbs on LMDB's performance and reliability. I have enjoyed the articles comparing LMDB with other databases' performance and hope you continue to point the spotlight on the superior design decisions of LMDB.

hyc_symas 8y ago
Thanks, glad you're getting something out of those writeups. Hopefully it helps other developers learn a better path.
- diroussel 8y ago
  What if another process calls fsync? It will get the error. Then when LMDB calls fsync no error will be reported. And thus the transaction will not be retried. Is this scenario dealt with?
  - anarazel 8y ago
    Newer versions of linux (but not plenty of other OSs) guarantee that each write failure will be signalled once to each file descriptor open before the error occurred. So that ought to be handled, unless the FD is closed inbetween.
  - hyc_symas 8y ago
    LMDB is single-writer. Only one process can acquire the writelock at a time, and only the writer can call fsync() so this scenario doesn't arise. If you open the datafile with write access from another process without using the LMDB API, you're intentionally shooting yourself in the foot and all bets are off.
    - jorangreef 8y ago
      On some systems, fsync is system-wide. Another process fsyncing an unrelated file descriptor can still consume the error meant for the LMDB file descriptor. Same thing goes for a user running the sync command from the terminal. A write lock won't protect you from this, unless you can prevent all other processes from calling fsync. It's got nothing to do with opening the LMDB datafile concurrently. If you share a physical disk device with any other process, you're at risk.
      - hyc_symas 8y ago
        Yes, acknowledged. https://news.ycombinator.com/item?id=19128325 https://news.ycombinator.com/item?id=19128325 fsync() is not documented to be system-wide (while sync() is). That behavior would also be an OS bug. The question this person asked was specifically about fsync().
        jorangreef 8y ago
        Sorry, that was confusing, I meant "system-wide" as in file system, not OS, i.e. "if you share a physical disk device with any other process". When you flush a particular FD, some file systems just flush every dirty buffer for the entire disk. I wouldn't actually be surprised though if there are some kernels that flush all disks either, regardless of whether it's considered a bug or not. "The question this person asked was specifically about fsync()." Sure, but as you acknowledged elsewhere, if sync is called, it flushes everything, and this impacts upon the person who "asked specifically about fsync" since there's a chance on some kernels that sync will eat the error that fsync was expecting.

macdice 8y ago

But the spec doesn't imply in any way that you'll need to write() again if fsync() fails. And dropping dirty flags seems way out of spec because now you can read data from cache that is not on disk and never will be even if future fsync() succeeds. So I don't buy the spec argument. You have me curious now... I don't know anything about LMDB but I wonder if its msync()-based design really is immune... Could there be a way for an IO error to leave you with a page bogusly marked clean, which differs from the data on disk?

hyc_symas 8y ago
The spec leaves the system condition undefined after an fsync failure. The safe thing to do is assume everything failed and nothing was written. That's what LMDB does. Expecting anything else would be relying on implementation-specific knowledge, which is always a bad idea. > I don't know anything about LMDB but I wonder if its msync()-based design really is immune. By default, LMDB doesn't use msync, so this isn't really a realistic scenario. If there is an I/O error that the OS does not report, then sure, it's possible for LMDB to have a corrupted view of the world. But that would be an OS bug in the first place. Since we're talking about buffered writes - clearly it's possible for a write() to return success before its data actually gets flushed to disk. And it's possible for a background flush to occur independently of the app calling fsync(). The docs specifically state, if an I/O error occurs during writeback, it will be reported on all file descriptors open on that file. So assuming the OS doesn't have a critical bug here, no, there's no way for an I/O error to leave LMDB with an invalid view of the world.
- nh2 8y ago
  > The spec leaves the system condition undefined [...]. The safe thing to do is assume everything failed This is key. Often programmers do 'assumption based programming'. "Surely the function will do X, it's the only reasonable thing to do, right?". As much it is human, this is bad practice and leads to unreliable systems. If the spec doesn't say it, don't assume anything about it, and keep asking. To show that this approach is feasible for anyone, here is an example: Recently I needed to write an fsync-safe application. The question of whether close()+re-open()+fsync() is safe came up. I found it had been asked on StackOverflow (https://stackoverflow.com/questions/37288453/calling-fsync2-after-close2/50158433 https://stackoverflow.com/questions/37288453/calling-fsync2-...) but received no answers for a year. I put a 100-reputation bounty on it and quickly got a competent reply quoting the spec and summarising: > It is not 100% clear whether the 'all currently queued I/O operations associated with the file indicated by [the] file descriptor' applies across processes. Conceptually, I think it should, but the wording isn't there in black and white. With the spec being unclear, I took the question to the kernel developers (https://marc.info/?m=152535098505449 https://marc.info/?m=152535098505449), and was immediately pointed at the Postgres fsyncgate results. So by spending a few hours on not believing what I wished was true, I managed to avoid writing an unsafe system. Always ask those in the know (specs and people). Never assume.

grandinj 8y ago

in the reported case, fsync does not persist all data to disk, but it reports success. How does LMDB deal with that situation?

hyc_symas 8y ago
In the reported case, fsync reported an error, then (more data may or may not have been written), then fsync is tried again and reports a success, which masks the fact that data from the previous fsync didn't get fully written. As I already wrote - in LMDB, after fsync fails, the transaction is aborted, so none of the partially written data matters.
- jhallenworld 8y ago
  Hold on, they are saying that if sync fails (for example if someone types "sync" at the console), then the database calls fsync() it will not fail even though the data is gone. I don't see how any database the uses the buffer cache could guard against this case. The kernel should never do this. If sync fails, all future syncs should also fail. This could be relaxed in various ways: sync fails, but we record which files have missing data, so any fsync for just those files also fails. (Otherwise I agree with LMDB- there should be no retry business on fsync fails).
  - hyc_symas 8y ago
    You're right, that was another error case in the article that I missed the first time. In LMDB's case you'd need to be pretty unlucky for that bug to bite; all the writes are done at once during txn_commit so you have to issue sync() during the couple milliseconds that might take, before the fsync() at the end of the writes. Also, it has to be a single-sector failure, otherwise a write after the sync will still fail and be caught. E.g. LMDB other write write write sync write fsync If the device is totally hosed, all of those writes' destination pages would error out. In that case, the failure can only be missed if you're unlucky enough for the sync to start immediately after the last write and before the fsync. The window is on the order of nanoseconds. If only a single page is bad, and the majority of the writes are ok, then you still have to be unlucky enough for the sync to run after the bad write; if it runs before the bad write then fsync will still see the error.

garyclarke27 8y ago

LMDB claimed speed and reliability seems remarkable (from a quick glance). I would guess is easier to achieve such, for a KV store, than for much more complex Relational Database. Got me thinking though. Mayby Postgres could take advantage of LMDB? Mayby by using LMDB as it’s cache? instead of using OS page cache, maybe writing the WAL to LMDB?

lmb 8y ago
LMDB is not a great fit for something like the WAL, where new data is written at the end, and old data discarded at the start. It leads to fragmentation (especially if WAL entries are larger than a single page).
ddorian43 8y ago
Maybe with https://github.com/kellabyte/rewind https://github.com/kellabyte/rewind which should support WAL for lmdb
hyc_symas 8y ago
LMDB itself only uses the OS page cache. The way for LMDB to improve an RDBMS is for it to replace the existing row and index store, and eliminate any WAL. This is what SQLightning does with SQLite. Have looked at replacing InnoDB in MySQL, but that code is a lot harder to read, so it's been slow going. Postgres doesn't have a modular storage interface, so it would be even uglier to overhaul.
- garyclarke27 8y ago
  Thanks, make sense, I think Postgres are planning go have pluggable storage interface in nest version 12, would that help? Also nobody has mention data checksum added v9.3, do you know if this helps avoid this kind of fsync related corruption?
  - anarazel 8y ago
    > Also nobody has mention data checksum added v9.3, do you know if this helps avoid this kind of fsync related corruption? Not really, I think. Page-level checksums don't protect against entire writes going missing, unfortunately.

lmb 8y ago

We've been using LMDB at Cloudflare to store small-ish configuration data, it has been rock solid. Thank you and the rest of the contributors for such a great library.

pgaddict 8y ago

I'm pretty sure it does affect any database/application relying on buffered I/O. Even if you use the fsync() interface correctly, you're still affected by the bugs in error reporting.

hyc_symas 8y ago
Technically there is no bug in error reporting. fsync() reported an error as it should. The application continued processing, instead of stopping. fsync() didn't report the same error a second time, which leads to the app having problems. The application should have stopped the first time fsync() reported an error. LMDB does this, instead of futile retry attempts that Postgres et al do. Fundamentally, a library should not automatically retry anything - it should surface all errors back to the caller and let the caller decide what to do next. That's just good design.
- pgaddict 8y ago
  No. Kernels before 4.13 may or may not report the fsync error correctly, depending on various conditions. There are more details in the talk [1] I posted earlier, and in the LWN articles related to this issue. [1] https://www.youtube.com/watch?v=74c19hwY2oE https://www.youtube.com/watch?v=74c19hwY2oE
  - hyc_symas 8y ago
    Ah I see. https://lwn.net/Articles/718734/ https://lwn.net/Articles/718734/ Someone issuing a sync() could cause an error to be cleared before the app's fsync() happens. That's a drag.

mehrdadn 8y ago

> Any dev who actually RTFM'd and followed POSIX specs would have been fine. So I'm trying to do that [1] and it seems to me the documentation directly implies that a second successful call to fsync() necessarily implies that all data was transferred, even if the previous call to fsync() had failed. I say this because the sentence says "all data for the open file descriptor" is to be transferred, not merely "all data written since the previous fsync to this file descriptor". It follows that any data not transferred in the previous call due to an error ("outstanding I/O operations are not guaranteed to have been completed") must now be transferred if this call is successful. What am I missing? [1] https://pubs.opengroup.org/onlinepubs/009695399/functions/fsync.html https://pubs.opengroup.org/onlinepubs/009695399/functions/fs...

hyc_symas 8y ago
Technically, there is nothing besides "all data written since the previous fsync" since the previous fsync already wrote all the outstanding data at that point in time. (Even if it actually failed.) I.e., there is never any to-be-written data remaining after fsync returns. Everything that was queued to be flushed was flushed and dequeued. Whether any particular write failed or not doesn't change that fact.
- mehrdadn 8y ago
  > the previous fsync already wrote all the outstanding data at that point in time. (Even if it actually failed.) I'm sorry, but this... just doesn't make sense. An event can't both happen and also fail to happen. Also, the queue business is an implementation detail. Our debate is over what the spec mandates, not how a particular implementation behaves.
  - hyc_symas 8y ago
    The fsync() function shall request that all data for the open file descriptor named by fildes is to be transferred to the storage device associated with the file described by fildes. The nature of the transfer is implementation-defined. The fsync() function shall not return until the system has completed that action or until an error is detected. All of the data was transferred to the device. The device may have failed to persist some or all of what was transferred, but it all got transferred. There's no mention of the OS retrying, or leaving the system in a state that a subsequent fsync can retry from where it left off. So you can't assume anything along those lines.

loeg 8y ago

> Any dev who actually RTFM'd and followed POSIX specs would have been fine. Yeesh. The POSIX manual on fsync is intentionally vague to hide cross-platform differences. There are basically no guarantees once an error happens. I guess that's one interpretation of RTFMing, but... clearly it doesn't match user expectations.