4 ms·

> Starting from kernel 4.13, we can now reliably detect such errors during fsync. No. Not even close. See https://wiki.postgresql.org/wiki/Fsync_Errors https:

by lelf 8y ago

> Starting from kernel 4.13, we can now reliably detect such errors during fsync. No. Not even close. See https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors

zaarn 8y ago
Yes, they do. If fsync returns an error, they crash. The problem occurs when you retry fsync.
- lelf 8y ago
  Read the wiki page. [Not Postgres’ fault in any way of course.] > Linux 4.13 and 4.15: fsync() only reports writeback errors that occurred after you called open()
- gnulinux 8y ago
  My understanding is that that's not true, since if some other process (postgres or not) fsyncs the same file, then kernel will act as if it's been retried (because fsync called twice) so it will not report error correctly.
  - cryptonector 8y ago
    That's the bug in PostgreSQL. The fix is to not do the the fsync() in a different process that used an FD opened separately for the same file.
    - macdice 8y ago
      I think it's debatable whether that's a bug in PostgreSQL, an underspecified interface, or a bug somewhere else. A more interesting question is how we get it fixed. The new Linux errseq_t design makes sure that every fd that was open before the error will see the error, and that at least one fd will see the error even if it happened when no one had it open (but only for as long as the inode doesn't fall out the cache). Before errseq_t came along, Linux was undeniably buggy here, since the AS_EIO flag could apparently be cleared in various ways and userspace could never be told about it. The things achieved so far since the PostgreSQL community first crashed into this problem (thanks to the efforts of Craig Ringer): * PostgreSQL now panics on failure, in cases where it previously retried (unless you set data_sync_retry = on, which should be safe on eg FreeBSD, though I don't think there is much point in it so the setting was included just as a matter of principle, when rolling out such a drastic change) * Linux now reports errors to at least one fd in versions new enough to have errseq_t (in addition to reporting it in every fd that was open at the time); that came out of discussions between Linux and PostgreSQL people about all this * There have also been changes to OpenBSD, though I'm not sure what exactly * PostgreSQL hackers are working on a plan to make sure that file descriptors are held open until the data is synced, so that there is no reliance on the error state surviving in the inode cache during times when it's not open (this is complicated by the use of processes instead of threads) * Longer term, this whole thing boosts interest in developing DIO support for PostgreSQL (previously thought to be a performance feature)
      - cryptonector 8y ago
        This is definitely debatable. However, because *BSD had the same semantics as Linux (IIUC), we can infer that the fsync()-syncs-only-writes-through-this-open-file semantics is actually what's reasonable to implement.
        macdice 8y ago
        FreeBSD has had different semantics here for ~20 years: https://github.com/freebsd/freebsd/commit/e4e8fec98ae986357cdc208b04557dba55a59266 https://github.com/freebsd/freebsd/commit/e4e8fec98ae986357c... I wish I could find the discussion around that commit. It'll only start throwing away dirty buffers if the device actually goes away: https://github.com/freebsd/freebsd/commit/46b5afe7b1ae0ee655ec863ebe373a3cf16eef1d https://github.com/freebsd/freebsd/commit/46b5afe7b1ae0ee655...

by lelf 8y ago

> Starting from kernel 4.13, we can now reliably detect such errors during fsync. No. Not even close. See https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors

zaarn 8y ago

Yes, they do. If fsync returns an error, they crash. The problem occurs when you retry fsync.

lelf 8y ago
Read the wiki page. [Not Postgres’ fault in any way of course.] > Linux 4.13 and 4.15: fsync() only reports writeback errors that occurred after you called open()
gnulinux 8y ago
My understanding is that that's not true, since if some other process (postgres or not) fsyncs the same file, then kernel will act as if it's been retried (because fsync called twice) so it will not report error correctly.
- cryptonector 8y ago
  That's the bug in PostgreSQL. The fix is to not do the the fsync() in a different process that used an FD opened separately for the same file.
  - macdice 8y ago
    I think it's debatable whether that's a bug in PostgreSQL, an underspecified interface, or a bug somewhere else. A more interesting question is how we get it fixed. The new Linux errseq_t design makes sure that every fd that was open before the error will see the error, and that at least one fd will see the error even if it happened when no one had it open (but only for as long as the inode doesn't fall out the cache). Before errseq_t came along, Linux was undeniably buggy here, since the AS_EIO flag could apparently be cleared in various ways and userspace could never be told about it. The things achieved so far since the PostgreSQL community first crashed into this problem (thanks to the efforts of Craig Ringer): * PostgreSQL now panics on failure, in cases where it previously retried (unless you set data_sync_retry = on, which should be safe on eg FreeBSD, though I don't think there is much point in it so the setting was included just as a matter of principle, when rolling out such a drastic change) * Linux now reports errors to at least one fd in versions new enough to have errseq_t (in addition to reporting it in every fd that was open at the time); that came out of discussions between Linux and PostgreSQL people about all this * There have also been changes to OpenBSD, though I'm not sure what exactly * PostgreSQL hackers are working on a plan to make sure that file descriptors are held open until the data is synced, so that there is no reliance on the error state surviving in the inode cache during times when it's not open (this is complicated by the use of processes instead of threads) * Longer term, this whole thing boosts interest in developing DIO support for PostgreSQL (previously thought to be a performance feature)
    - cryptonector 8y ago
      This is definitely debatable. However, because *BSD had the same semantics as Linux (IIUC), we can infer that the fsync()-syncs-only-writes-through-this-open-file semantics is actually what's reasonable to implement.
      - macdice 8y ago
        FreeBSD has had different semantics here for ~20 years: https://github.com/freebsd/freebsd/commit/e4e8fec98ae986357cdc208b04557dba55a59266 https://github.com/freebsd/freebsd/commit/e4e8fec98ae986357c... I wish I could find the discussion around that commit. It'll only start throwing away dirty buffers if the device actually goes away: https://github.com/freebsd/freebsd/commit/46b5afe7b1ae0ee655ec863ebe373a3cf16eef1d https://github.com/freebsd/freebsd/commit/46b5afe7b1ae0ee655...