5 ms·

Mr. Chu, I hope you never lose your tenacity with respect to writing blurbs on LMDB's performance and reliability. I have enjoyed the articles comparing LMDB wi

by evdubs 8y ago

Mr. Chu, I hope you never lose your tenacity with respect to writing blurbs on LMDB's performance and reliability. I have enjoyed the articles comparing LMDB with other databases' performance and hope you continue to point the spotlight on the superior design decisions of LMDB.

hyc_symas 8y ago
Thanks, glad you're getting something out of those writeups. Hopefully it helps other developers learn a better path.
- diroussel 8y ago
  What if another process calls fsync? It will get the error. Then when LMDB calls fsync no error will be reported. And thus the transaction will not be retried. Is this scenario dealt with?
  - anarazel 8y ago
    Newer versions of linux (but not plenty of other OSs) guarantee that each write failure will be signalled once to each file descriptor open before the error occurred. So that ought to be handled, unless the FD is closed inbetween.
  - hyc_symas 8y ago
    LMDB is single-writer. Only one process can acquire the writelock at a time, and only the writer can call fsync() so this scenario doesn't arise. If you open the datafile with write access from another process without using the LMDB API, you're intentionally shooting yourself in the foot and all bets are off.
    - jorangreef 8y ago
      On some systems, fsync is system-wide. Another process fsyncing an unrelated file descriptor can still consume the error meant for the LMDB file descriptor. Same thing goes for a user running the sync command from the terminal. A write lock won't protect you from this, unless you can prevent all other processes from calling fsync. It's got nothing to do with opening the LMDB datafile concurrently. If you share a physical disk device with any other process, you're at risk.
      - hyc_symas 8y ago
        Yes, acknowledged. https://news.ycombinator.com/item?id=19128325 https://news.ycombinator.com/item?id=19128325 fsync() is not documented to be system-wide (while sync() is). That behavior would also be an OS bug. The question this person asked was specifically about fsync().
        jorangreef 8y ago
        Sorry, that was confusing, I meant "system-wide" as in file system, not OS, i.e. "if you share a physical disk device with any other process". When you flush a particular FD, some file systems just flush every dirty buffer for the entire disk. I wouldn't actually be surprised though if there are some kernels that flush all disks either, regardless of whether it's considered a bug or not. "The question this person asked was specifically about fsync()." Sure, but as you acknowledged elsewhere, if sync is called, it flushes everything, and this impacts upon the person who "asked specifically about fsync" since there's a chance on some kernels that sync will eat the error that fsync was expecting.

by evdubs 8y ago

hyc_symas 8y ago

Thanks, glad you're getting something out of those writeups. Hopefully it helps other developers learn a better path.

diroussel 8y ago
What if another process calls fsync? It will get the error. Then when LMDB calls fsync no error will be reported. And thus the transaction will not be retried. Is this scenario dealt with?
- anarazel 8y ago
  Newer versions of linux (but not plenty of other OSs) guarantee that each write failure will be signalled once to each file descriptor open before the error occurred. So that ought to be handled, unless the FD is closed inbetween.
- hyc_symas 8y ago
  LMDB is single-writer. Only one process can acquire the writelock at a time, and only the writer can call fsync() so this scenario doesn't arise. If you open the datafile with write access from another process without using the LMDB API, you're intentionally shooting yourself in the foot and all bets are off.
  - jorangreef 8y ago
    On some systems, fsync is system-wide. Another process fsyncing an unrelated file descriptor can still consume the error meant for the LMDB file descriptor. Same thing goes for a user running the sync command from the terminal. A write lock won't protect you from this, unless you can prevent all other processes from calling fsync. It's got nothing to do with opening the LMDB datafile concurrently. If you share a physical disk device with any other process, you're at risk.
    - hyc_symas 8y ago
      Yes, acknowledged. https://news.ycombinator.com/item?id=19128325 https://news.ycombinator.com/item?id=19128325 fsync() is not documented to be system-wide (while sync() is). That behavior would also be an OS bug. The question this person asked was specifically about fsync().
      - jorangreef 8y ago
        Sorry, that was confusing, I meant "system-wide" as in file system, not OS, i.e. "if you share a physical disk device with any other process". When you flush a particular FD, some file systems just flush every dirty buffer for the entire disk. I wouldn't actually be surprised though if there are some kernels that flush all disks either, regardless of whether it's considered a bug or not. "The question this person asked was specifically about fsync()." Sure, but as you acknowledged elsewhere, if sync is called, it flushes everything, and this impacts upon the person who "asked specifically about fsync" since there's a chance on some kernels that sync will eat the error that fsync was expecting.