7 ms·

Linux Fsync Issue for Buffered IO and Its Preliminary Fix for PostgreSQL

137 points by avivallssa 8y ago

masklinn 8y ago
If you want an overview of the issue, here's a presentation from Tomas Vondra at FOSDEM 2019: https://youtu.be/1VWIGBQLtxo https://youtu.be/1VWIGBQLtxo Or an early recap of the "fsyncgate" issue in textual form: https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/ Related (also listed by Tomas Vondra): Linux's IO errors reporting https://youtu.be/74c19hwY2oE https://youtu.be/74c19hwY2oE A previous hn discussion on the subject: https://news.ycombinator.com/item?id=19119991 https://news.ycombinator.com/item?id=19119991 Also note that this is a broad issue with fsync, it's possible that your own software is affected: https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors links to mysql and mongodb fixes for the same assumptions, one of the posts from the original fsyncgate thread mentions that dpkg made the same incorrect assumption.
- fake-name 8y ago
  > fsyncgate Oh cripes, can we not?
dooglius 8y ago
> To understand it better, consider an example of Linux trying to write dirty pages from page cache to a USB stick that was removed during an fsync. Neither the ext4 file system nor the btrfs nor an xfs tries to retry the failed writes. A silently failing fsync may result in data loss, block corruption, table or index out of sync, foreign key or other data integrity issues… and deleted records may reappear. As opposed to what? If the drive isn't there anymore, there's not a whole lot that can be done. > With the new minor version for all supported PostgreSQL versions, a PANIC is triggered upon such error. This performs a database crash and initiates recovery from the last CHECKPOINT. How is a recovery possible if the hard drive is borked? I don't understand the model that leads to this "fix" making any difference.
- avivallssa 8y ago
  This issue is observed during a CHECHKPOINT when a BGWriter writes dirty or modified buffers from shared buffers to Disk. So, upon recovery after crash, the changes since the last checkpoint are applied from WAL, this time. So, we are using the changes that are in a WAL (Write-ahead-log) because the changes are already gone from Disk as you rightly said. A good question in fact.
  - pgaddict 8y ago
    nitpick: BGWrites is not involved in checkpoint - it's a separate process, doing something like checkpoint (so an alternative).
- LIV2 8y ago
  A better example might be a SAN briefly becoming unavailable due to a transient issue with your ISCSI network?
  - dooglius 8y ago
    Yeah, this more or less clears it up, I was assuming an error implied disk failure.
    - pgaddict 8y ago
      Right, this is about "ephemeral" failures which are becoming more common thanks to accessing storage over network, virtualization, thin provisioning etc.
      - mprovost 8y ago
        NFS has been around forever and has always had a bad reputation due to problems like this. It mostly handles transient failures by waiting (indefinitely) for the server to return, but it's unclear what a better option is.
        8y ago
mjw1007 8y ago
The Linux project takes the view « We don't attempt to rigorously document our API; instead we promise that if your program worked yesterday it will continue to work in the future. » I think this story shows a weakness in that approach: for rarely-exercised error handling paths, it's too likely that your program didn't work yesterday and you had no easy way to know that. (This is a separate issue from the fact that until recently the kernel implementation of fync itself had significant bugs, measured against what its maintainers thought ought to be guaranteed.)
- pgaddict 8y ago
  Except that this issue (both the behavior and lack of exact docs) applies to other kernels, not just Linux. See https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors So no, this is not just about Linux.
  - scottlamb 8y ago
    I agree with mjw1007 that lack of rigorous API documentation for error paths is a huge weakness and with you that it's not a Linux-only problem. There are a lot of related filesystem robustness questions I'd love to get authoritative answers on. Neither the Single UNIX Specification nor OS-specific kernel docs / manpages gives enough information to write a robust, performant program, and certainly you can't find one place that gives everything you'd want to know when writing a portable program. For example: * Does fsync() make guarantees about just the inode, or also the dirent? (iirc on Linux only the inode; for a freshly-created file you also have to fsync() the directory.) * What does fsync() success guarantee is written to permanent storage? from this whole thing, apparently on Linux recently (even ignoring the bugs) only things since a previous fsync() failure or the current open() call, whichever was later. Yuck. That's a terrible behavior, and even worse for being undocumented. * Does it even guarantee that if you don't say "Simon says"? On macOS, I gather you need to do this extra F_FULLFSYNC thing. Are the other platforms like that? I dunno. And there are certainly mentions of older hard drives where nothing can be trusted. Is there any database of hard drive behavior? Stress test program to tell if I have a broken model? * If you do a write and power is lost before fsync, what guarantees do you have about the current state? I was trying to figure out recently if an N-byte aligned overwrite is guaranteed to reflect the "old" or "new" states (for various Ns: 1, 512, 4096, st_blksize). The best I could find is here: <https://stackoverflow.com/a/2068608/23584> https://stackoverflow.com/a/2068608/23584> which suggests yes for N=512 "these days". Do I trust that? on all platforms? for hard drives made how recently? etc. * If you create a file, write to it, and rename() it into place and power is lost before fsync, what guarantees do you have about the current state? is it guaranteed that the dirent points to either a previous inode (if any) or the new one? if it points to the new one, is the file guaranteed to have the write length or contents? the conservative thing to do would be to create, write, fsync() the file, fsync() the directory, rename, fsync() the directory again. But three syncs is getting ridiculous. Is it safe to remove one or more?
lelf 8y ago
> Starting from kernel 4.13, we can now reliably detect such errors during fsync. No. Not even close. See https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors
- zaarn 8y ago
  Yes, they do. If fsync returns an error, they crash. The problem occurs when you retry fsync.
  - lelf 8y ago
    Read the wiki page. [Not Postgres’ fault in any way of course.] > Linux 4.13 and 4.15: fsync() only reports writeback errors that occurred after you called open()
  - gnulinux 8y ago
    My understanding is that that's not true, since if some other process (postgres or not) fsyncs the same file, then kernel will act as if it's been retried (because fsync called twice) so it will not report error correctly.
    - cryptonector 8y ago
      That's the bug in PostgreSQL. The fix is to not do the the fsync() in a different process that used an FD opened separately for the same file.
      - macdice 8y ago
        I think it's debatable whether that's a bug in PostgreSQL, an underspecified interface, or a bug somewhere else. A more interesting question is how we get it fixed. The new Linux errseq_t design makes sure that every fd that was open before the error will see the error, and that at least one fd will see the error even if it happened when no one had it open (but only for as long as the inode doesn't fall out the cache). Before errseq_t came along, Linux was undeniably buggy here, since the AS_EIO flag could apparently be cleared in various ways and userspace could never be told about it. The things achieved so far since the PostgreSQL community first crashed into this problem (thanks to the efforts of Craig Ringer): * PostgreSQL now panics on failure, in cases where it previously retried (unless you set data_sync_retry = on, which should be safe on eg FreeBSD, though I don't think there is much point in it so the setting was included just as a matter of principle, when rolling out such a drastic change) * Linux now reports errors to at least one fd in versions new enough to have errseq_t (in addition to reporting it in every fd that was open at the time); that came out of discussions between Linux and PostgreSQL people about all this * There have also been changes to OpenBSD, though I'm not sure what exactly * PostgreSQL hackers are working on a plan to make sure that file descriptors are held open until the data is synced, so that there is no reliance on the error state surviving in the inode cache during times when it's not open (this is complicated by the use of processes instead of threads) * Longer term, this whole thing boosts interest in developing DIO support for PostgreSQL (previously thought to be a performance feature)
bepvte 8y ago
https://edfile.pro/380a8f2 https://edfile.pro/380a8f2 Really enjoyable reading experience...
deleted 8y ago
[deleted]
mehrdadn 8y ago
Does anyone know if FlushFileBuffers() on Windows also forgets to flush data that previously failed to flush? i.e., if Windows has the same issue or not?
- masklinn 8y ago
  Somebody asked on SO: https://stackoverflow.com/questions/54660780/is-it-safe-to-call-the-win32-flushfilebuffers-function-again-if-it-previously https://stackoverflow.com/questions/54660780/is-it-safe-to-c... And while there was no hard answer and the documentation is basically on the level of POSIX (aka it says jack about the system's state on failure), the consensus seems to be "assume it's fucked": > The answer will probably depend on the file system driver, as FlushFileBuffers is just a user-mode wrapper around NtFlushBuffersFile, and it looks to me like that function just assembles a flush IRP (IRP_MJ_FLUSH_BUFFERS) and sends it via IoCallDriver. Of course, one should eye calls to FlushFileBuffers with a great deal of suspicion.
- macdice 8y ago
  I am also curious. I tried asking Microsoft via Twitter and they asked me to copy the information from that Wiki page into Microsoft One Drive so they could read it, and then suggested I try asking on Stack Overflow instead! https://twitter.com/windowsdev/status/989857799994822663 https://twitter.com/windowsdev/status/989857799994822663 Amusing social media exchanges aside, I'm quite curious to know as well. We don't have the answers for any non-open OS (though of course we can speculate about descendants of BSD except FreeBSD (probably throw away buffers on error), FreeBSD (probably keep buffers cached and dirty), and SysV systems (probably throw away buffers on error)). EDIT: They also suggested asking on MSDN Forums, which I didn't do because I don't have an account and am not a Windows developer at all, just a humble database hacker trying to understand how our stuff works on every platform. The code we committed assumes the worst by default, so not knowing the answer here isn't damaging. I dunno if it's possible to reach actual kernel hackers via MSDN Forums, but maybe someone should follow up with that. I think someone with the right skillset could possibly design an experiment to figure it out for Windows (several people have shown how to set up experiments on Linux and FreeBSD to test this).

137 points by avivallssa 8y ago

If you want an overview of the issue, here's a presentation from Tomas Vondra at FOSDEM 2019: https://youtu.be/1VWIGBQLtxo https://youtu.be/1VWIGBQLtxo Or an early recap of the "fsyncgate" issue in textual form: https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/ Related (also listed by Tomas Vondra): Linux's IO errors reporting https://youtu.be/74c19hwY2oE https://youtu.be/74c19hwY2oE A previous hn discussion on the subject: https://news.ycombinator.com/item?id=19119991 https://news.ycombinator.com/item?id=19119991 Also note that this is a broad issue with fsync, it's possible that your own software is affected: https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors links to mysql and mongodb fixes for the same assumptions, one of the posts from the original fsyncgate thread mentions that dpkg made the same incorrect assumption.

fake-name 8y ago
> fsyncgate Oh cripes, can we not?

dooglius 8y ago

> To understand it better, consider an example of Linux trying to write dirty pages from page cache to a USB stick that was removed during an fsync. Neither the ext4 file system nor the btrfs nor an xfs tries to retry the failed writes. A silently failing fsync may result in data loss, block corruption, table or index out of sync, foreign key or other data integrity issues… and deleted records may reappear. As opposed to what? If the drive isn't there anymore, there's not a whole lot that can be done. > With the new minor version for all supported PostgreSQL versions, a PANIC is triggered upon such error. This performs a database crash and initiates recovery from the last CHECKPOINT. How is a recovery possible if the hard drive is borked? I don't understand the model that leads to this "fix" making any difference.

avivallssa 8y ago
This issue is observed during a CHECHKPOINT when a BGWriter writes dirty or modified buffers from shared buffers to Disk. So, upon recovery after crash, the changes since the last checkpoint are applied from WAL, this time. So, we are using the changes that are in a WAL (Write-ahead-log) because the changes are already gone from Disk as you rightly said. A good question in fact.
- pgaddict 8y ago
  nitpick: BGWrites is not involved in checkpoint - it's a separate process, doing something like checkpoint (so an alternative).
LIV2 8y ago
A better example might be a SAN briefly becoming unavailable due to a transient issue with your ISCSI network?
- dooglius 8y ago
  Yeah, this more or less clears it up, I was assuming an error implied disk failure.
  - pgaddict 8y ago
    Right, this is about "ephemeral" failures which are becoming more common thanks to accessing storage over network, virtualization, thin provisioning etc.
    - mprovost 8y ago
      NFS has been around forever and has always had a bad reputation due to problems like this. It mostly handles transient failures by waiting (indefinitely) for the server to return, but it's unclear what a better option is.
      - 8y ago

mjw1007 8y ago

The Linux project takes the view « We don't attempt to rigorously document our API; instead we promise that if your program worked yesterday it will continue to work in the future. » I think this story shows a weakness in that approach: for rarely-exercised error handling paths, it's too likely that your program didn't work yesterday and you had no easy way to know that. (This is a separate issue from the fact that until recently the kernel implementation of fync itself had significant bugs, measured against what its maintainers thought ought to be guaranteed.)

pgaddict 8y ago
Except that this issue (both the behavior and lack of exact docs) applies to other kernels, not just Linux. See https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors So no, this is not just about Linux.
- scottlamb 8y ago
  I agree with mjw1007 that lack of rigorous API documentation for error paths is a huge weakness and with you that it's not a Linux-only problem. There are a lot of related filesystem robustness questions I'd love to get authoritative answers on. Neither the Single UNIX Specification nor OS-specific kernel docs / manpages gives enough information to write a robust, performant program, and certainly you can't find one place that gives everything you'd want to know when writing a portable program. For example: * Does fsync() make guarantees about just the inode, or also the dirent? (iirc on Linux only the inode; for a freshly-created file you also have to fsync() the directory.) * What does fsync() success guarantee is written to permanent storage? from this whole thing, apparently on Linux recently (even ignoring the bugs) only things since a previous fsync() failure or the current open() call, whichever was later. Yuck. That's a terrible behavior, and even worse for being undocumented. * Does it even guarantee that if you don't say "Simon says"? On macOS, I gather you need to do this extra F_FULLFSYNC thing. Are the other platforms like that? I dunno. And there are certainly mentions of older hard drives where nothing can be trusted. Is there any database of hard drive behavior? Stress test program to tell if I have a broken model? * If you do a write and power is lost before fsync, what guarantees do you have about the current state? I was trying to figure out recently if an N-byte aligned overwrite is guaranteed to reflect the "old" or "new" states (for various Ns: 1, 512, 4096, st_blksize). The best I could find is here: <https://stackoverflow.com/a/2068608/23584> https://stackoverflow.com/a/2068608/23584> which suggests yes for N=512 "these days". Do I trust that? on all platforms? for hard drives made how recently? etc. * If you create a file, write to it, and rename() it into place and power is lost before fsync, what guarantees do you have about the current state? is it guaranteed that the dirent points to either a previous inode (if any) or the new one? if it points to the new one, is the file guaranteed to have the write length or contents? the conservative thing to do would be to create, write, fsync() the file, fsync() the directory, rename, fsync() the directory again. But three syncs is getting ridiculous. Is it safe to remove one or more?

lelf 8y ago

> Starting from kernel 4.13, we can now reliably detect such errors during fsync. No. Not even close. See https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors

zaarn 8y ago
Yes, they do. If fsync returns an error, they crash. The problem occurs when you retry fsync.
- lelf 8y ago
  Read the wiki page. [Not Postgres’ fault in any way of course.] > Linux 4.13 and 4.15: fsync() only reports writeback errors that occurred after you called open()
- gnulinux 8y ago
  My understanding is that that's not true, since if some other process (postgres or not) fsyncs the same file, then kernel will act as if it's been retried (because fsync called twice) so it will not report error correctly.
  - cryptonector 8y ago
    That's the bug in PostgreSQL. The fix is to not do the the fsync() in a different process that used an FD opened separately for the same file.
    - macdice 8y ago
      I think it's debatable whether that's a bug in PostgreSQL, an underspecified interface, or a bug somewhere else. A more interesting question is how we get it fixed. The new Linux errseq_t design makes sure that every fd that was open before the error will see the error, and that at least one fd will see the error even if it happened when no one had it open (but only for as long as the inode doesn't fall out the cache). Before errseq_t came along, Linux was undeniably buggy here, since the AS_EIO flag could apparently be cleared in various ways and userspace could never be told about it. The things achieved so far since the PostgreSQL community first crashed into this problem (thanks to the efforts of Craig Ringer): * PostgreSQL now panics on failure, in cases where it previously retried (unless you set data_sync_retry = on, which should be safe on eg FreeBSD, though I don't think there is much point in it so the setting was included just as a matter of principle, when rolling out such a drastic change) * Linux now reports errors to at least one fd in versions new enough to have errseq_t (in addition to reporting it in every fd that was open at the time); that came out of discussions between Linux and PostgreSQL people about all this * There have also been changes to OpenBSD, though I'm not sure what exactly * PostgreSQL hackers are working on a plan to make sure that file descriptors are held open until the data is synced, so that there is no reliance on the error state surviving in the inode cache during times when it's not open (this is complicated by the use of processes instead of threads) * Longer term, this whole thing boosts interest in developing DIO support for PostgreSQL (previously thought to be a performance feature)

bepvte 8y ago

https://edfile.pro/380a8f2 https://edfile.pro/380a8f2 Really enjoyable reading experience...

deleted 8y ago

[deleted]

mehrdadn 8y ago

Does anyone know if FlushFileBuffers() on Windows also forgets to flush data that previously failed to flush? i.e., if Windows has the same issue or not?

masklinn 8y ago
Somebody asked on SO: https://stackoverflow.com/questions/54660780/is-it-safe-to-call-the-win32-flushfilebuffers-function-again-if-it-previously https://stackoverflow.com/questions/54660780/is-it-safe-to-c... And while there was no hard answer and the documentation is basically on the level of POSIX (aka it says jack about the system's state on failure), the consensus seems to be "assume it's fucked": > The answer will probably depend on the file system driver, as FlushFileBuffers is just a user-mode wrapper around NtFlushBuffersFile, and it looks to me like that function just assembles a flush IRP (IRP_MJ_FLUSH_BUFFERS) and sends it via IoCallDriver. Of course, one should eye calls to FlushFileBuffers with a great deal of suspicion.
macdice 8y ago
I am also curious. I tried asking Microsoft via Twitter and they asked me to copy the information from that Wiki page into Microsoft One Drive so they could read it, and then suggested I try asking on Stack Overflow instead! https://twitter.com/windowsdev/status/989857799994822663 https://twitter.com/windowsdev/status/989857799994822663 Amusing social media exchanges aside, I'm quite curious to know as well. We don't have the answers for any non-open OS (though of course we can speculate about descendants of BSD except FreeBSD (probably throw away buffers on error), FreeBSD (probably keep buffers cached and dirty), and SysV systems (probably throw away buffers on error)). EDIT: They also suggested asking on MSDN Forums, which I didn't do because I don't have an account and am not a Windows developer at all, just a humble database hacker trying to understand how our stuff works on every platform. The code we committed assumes the worst by default, so not knowing the answer here isn't damaging. I dunno if it's possible to reach actual kernel hackers via MSDN Forums, but maybe someone should follow up with that. I think someone with the right skillset could possibly design an experiment to figure it out for Windows (several people have shown how to set up experiments on Linux and FreeBSD to test this).