6 ms·

Well, I mean, they can claim it's unclear, but to me they're obviously wrong—and you don't have to take my word over theirs; just go read the spec [1]. It's per

Well, I mean, they can claim it's unclear, but to me they're obviously wrong—and you don't have to take my word over theirs; just go read the spec [1]. It's perfectly clear what "all data up to the time of the fsync() call" means. You obviously can't just read that phrase and claim it's in any way synonymous with "all data up to the time of the fsync() call since the previous call to fsync(), even if it failed (!)". "All data until X event" just means all data before X—plain, simple, and unambiguous. It just doesn't mean "all data until X since Y"! [1] https://pubs.opengroup.org/onlinepubs/009695399/functions/fsync.html https://pubs.opengroup.org/onlinepubs/009695399/functions/fs...

belter 5y ago
This paper: "Can Applications Recover from fsync Failures?" https://www.usenix.org/system/files/atc20-rebello.pdf https://www.usenix.org/system/files/atc20-rebello.pdf argues the following: "#1: Existing file systems do not handle fsync failures uniformly. In an effort to hide cross-platform differences, POSIX is intentionally vague on how failures are handled. Thus, different file systems behave differently after an fsync failure (as seen in Table 1), leading to non-deterministic outcomes for applications that treat all file systems equally. We believe that the POSIX specification for fsync needs to be clarified and the expected failure behavior described in more detail."
- dataflow 5y ago
  People can say whatever they want (even in papers!), and people go through contortions to justify arbitrary things, but like I said, you don't have to take my word any more than you have to take theirs when the spec is right there. You can just read the spec yourself and witness that the meaning is pretty clear.
  - belter 5y ago
    Ok. Let me try to make a resume of this, until somebody can chime and correct me... 1) Based on this https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors and this https://lwn.net/Articles/752105/ https://lwn.net/Articles/752105/ particularly this justification: "...Next, the reason why fsync() has the behaviour that it does is one ofhe the most common cases of I/O storage errors in buffered use cases, certainly as seen by the community distros, is the user who pulls out USB stick while it is in use. In that case, if there are dirtied pages in the page cache, the question is what can you do? Sooner or later the writes will time out, and if you leave the pages dirty, then it effectively becomes a permanent memory leak. You can't unmount the file system --- that requires writing out all of the pages such that the dirty bit is turned off. And if you don't clear the dirty bit on an I/O error, then they can never be cleaned. You can't even re-insert the USB stick; the re-inserted USB stick will get a new block device. Worse, when the USB stick was pulled, it will have suffered a power drop, and see above about what could happen after a power drop for non-power fail certified flash devices --- it goes double for the cheap sh*t USB sticks found in the checkout aisle of Micro Center. So this is the explanation for why Linux handles I/O errors by clearing the dirty bit after reporting the error up to user space. And why there is not eagerness to solve the problem simply by "don't clear the dirty bit". For every one Postgres installation that might have a better recover after an I/O error, there's probably a thousand clueless Fedora and Ubuntu users who will have a much worse user experience after a USB stick pull happens. I can think of things that could be done --- for example, it could be switchable on a per-block device basis (or maybe a per-mount basis) whether or not the dirty bit gets cleared after the error is reported to userspace. And perhaps there could be a new unmount flag that causes all dirty pages to be wiped out, which could be used to recover after a permanent loss of the block device. But the question is who is going to invest the time to make these changes? If there is a company who is willing to pay to comission this work, it's almost certainly soluble..." and this part of the paragraph: "...But again, of the companies who have client code where we care about robustness and proper handling of failed disk drives, and which have a kernel team on staff, pretty much all of the ones I can think of (e.g., Oracle, Google, etc.) use O_DIRECT and they don't try to make buffered writes and error reporting via fsync(2) work well..." I conclude that: 1) Fixing fsync() would cause other issues and might be technically challenging, and some within Linux community dont have it as their highest priority 2) The onus falls mostly mostly on Postgres team, to re-implement their code so as to use the "better" technical solution, as implemented for Linux for similar scenarios by Oracle, Google and others...
    - throw0101a 5y ago
      > In that case, if there are dirtied pages in the page cache, the question is what can you do? Sooner or later the writes will time out, and if you leave the pages dirty, then it effectively becomes a permanent memory leak. FreeBSD keeps the dirty buffer (always marked as dirty) in memory as long as the device exists: * https://github.com/freebsd/freebsd-src/commit/e4e8fec98ae986357cdc208b04557dba55a59266 https://github.com/freebsd/freebsd-src/commit/e4e8fec98ae986... You only toss the data when the device associated with the data goes away: * https://github.com/freebsd/freebsd-src/commit/46b5afe7b1ae0ee655ec863ebe373a3cf16eef1d https://github.com/freebsd/freebsd-src/commit/46b5afe7b1ae0e...
      - Dylan16807 5y ago
        Honestly even if a device disappears it's probably worth keeping around those pending writes for several seconds. It might come back, and if it comes back fast enough it probably hasn't been altered by any other machine.
    - dataflow 5y ago
      Super interesting (I hadn't seen it before), thanks for sharing! I feel like there might be lots of potential solutions they're ignoring though. > Sooner or later the writes will time out, and if you leave the pages dirty, then it effectively becomes a permanent memory leak. Why not just get rid of the pages after the underlying medium is removed? Especially since, after that point, you wouldn't be able to guarantee much about the device's contents anyway, even if it were plugged back in. Also, even if I buy that this is a trade-off with a permanent memory leak (which I don't, for lots of reasons like the above), isn't that better than outright corruption...? At least you can reboot to get rid of a leak. There's no guarantee you can get back corrupted data!
      - belter 5y ago
        Looking at it from another side...The whole Direct I/O path looks like a worst can of worms...maybe it's easier to review again fsync :-) "...The exact semantics of Direct I/O (O_DIRECT) are not well specified. It is not a part of POSIX, or SUS, or any other formal standards specification. The exact meaning of O_DIRECT has historically been negotiated in non-public discussions between powerful enterprise database companies and proprietary Unix systems, and its behaviour has generally been passed down as oral lore rather than as a formal set of requirements and specifications..." "The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances." -- Linus https://ext4.wiki.kernel.org/index.php/Clarifying_Direct_IO%27s_Semantics https://ext4.wiki.kernel.org/index.php/Clarifying_Direct_IO%...
  - cbsmith 5y ago
    There's an art to reading specs for standards like POSIX. ;-) This isn't the only place where POSIX leaves things up to being an implementation detail.
Kranar 5y ago
You're quoting a sentence fragment of a non-normative section of the spec, in other words you're half quoting a part of the spec that is not intended to be interpreted formally, literally, or even required to be adhered to. The rationale section of the POSIX spec is strictly informative. The fact that you left out the part of the sentence that says "The fsync() function is intended to..." suggests that you may be intentionally hiding this fact to support your point. Even if we go by the non-normative text, that section explicitly states that the spec allows for, and I quote "It is explicitly intended that a null implementation is permitted." At any rate, the normative section of the spec states only the following: "The fsync() function shall request that all data for the open file descriptor named by fildes is to be transferred to the storage device associated with the file described by fildes. The nature of the transfer is implementation-defined." Emphasis mine. It would be unwise to dismiss the wealth of resources that have been provided to you in other comments to better understand this issue, all of them quite extensive and written by experts on this subject, just because of an informal section of the standard
- thaumasiotes 5y ago
  > The fact that you left out the part of the sentence that says "The fsync() function is intended to..." suggests that you may be intentionally hiding this fact to support your point. It's very difficult to read this accusation as having been made in good faith, since the person you're accusing of intentionally omitting part of a quote already quoted that very text in their previous comment: >>>> The tl;dr (and blunt truth) is that Linux doesn't implement POSIX's semantics of fsync() correctly. The standard says "the fsync() function is intended to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk".
  - Kranar 5y ago
    >...the person you're accusing of intentionally omitting part of a quote already quoted that very text in their previous comment: That was edited back in after I made my comment (this is admitted to by OP themselves). To the extent that I make an accusation it's that it's absurd to think you can dismiss the wealth of information on this subject and come to a remotely comprehensive understanding of how to implement reliable IO functions by simply reading a couple of sentences of the POSIX spec and dismiss the numerous other sources that go into very fine detail on this subject. This is not a subject you just read a few sentences on and then make bold claims about how Linux doesn't implement fsync properly, implying that the experts who have gone to great lengths to implement fsync on Linux just don't understand it; that all the papers on this subject, the numerous mailing list posts, the very submission itself, can all readily be dismissed as a simple failure to read what is as clear as day as if none of the people discussing this issue have ever bothered to read the spec.

belter 5y ago

This paper: "Can Applications Recover from fsync Failures?" https://www.usenix.org/system/files/atc20-rebello.pdf https://www.usenix.org/system/files/atc20-rebello.pdf argues the following: "#1: Existing file systems do not handle fsync failures uniformly. In an effort to hide cross-platform differences, POSIX is intentionally vague on how failures are handled. Thus, different file systems behave differently after an fsync failure (as seen in Table 1), leading to non-deterministic outcomes for applications that treat all file systems equally. We believe that the POSIX specification for fsync needs to be clarified and the expected failure behavior described in more detail."

dataflow 5y ago
People can say whatever they want (even in papers!), and people go through contortions to justify arbitrary things, but like I said, you don't have to take my word any more than you have to take theirs when the spec is right there. You can just read the spec yourself and witness that the meaning is pretty clear.
- belter 5y ago
  Ok. Let me try to make a resume of this, until somebody can chime and correct me... 1) Based on this https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors and this https://lwn.net/Articles/752105/ https://lwn.net/Articles/752105/ particularly this justification: "...Next, the reason why fsync() has the behaviour that it does is one ofhe the most common cases of I/O storage errors in buffered use cases, certainly as seen by the community distros, is the user who pulls out USB stick while it is in use. In that case, if there are dirtied pages in the page cache, the question is what can you do? Sooner or later the writes will time out, and if you leave the pages dirty, then it effectively becomes a permanent memory leak. You can't unmount the file system --- that requires writing out all of the pages such that the dirty bit is turned off. And if you don't clear the dirty bit on an I/O error, then they can never be cleaned. You can't even re-insert the USB stick; the re-inserted USB stick will get a new block device. Worse, when the USB stick was pulled, it will have suffered a power drop, and see above about what could happen after a power drop for non-power fail certified flash devices --- it goes double for the cheap sh*t USB sticks found in the checkout aisle of Micro Center. So this is the explanation for why Linux handles I/O errors by clearing the dirty bit after reporting the error up to user space. And why there is not eagerness to solve the problem simply by "don't clear the dirty bit". For every one Postgres installation that might have a better recover after an I/O error, there's probably a thousand clueless Fedora and Ubuntu users who will have a much worse user experience after a USB stick pull happens. I can think of things that could be done --- for example, it could be switchable on a per-block device basis (or maybe a per-mount basis) whether or not the dirty bit gets cleared after the error is reported to userspace. And perhaps there could be a new unmount flag that causes all dirty pages to be wiped out, which could be used to recover after a permanent loss of the block device. But the question is who is going to invest the time to make these changes? If there is a company who is willing to pay to comission this work, it's almost certainly soluble..." and this part of the paragraph: "...But again, of the companies who have client code where we care about robustness and proper handling of failed disk drives, and which have a kernel team on staff, pretty much all of the ones I can think of (e.g., Oracle, Google, etc.) use O_DIRECT and they don't try to make buffered writes and error reporting via fsync(2) work well..." I conclude that: 1) Fixing fsync() would cause other issues and might be technically challenging, and some within Linux community dont have it as their highest priority 2) The onus falls mostly mostly on Postgres team, to re-implement their code so as to use the "better" technical solution, as implemented for Linux for similar scenarios by Oracle, Google and others...
  - throw0101a 5y ago
    > In that case, if there are dirtied pages in the page cache, the question is what can you do? Sooner or later the writes will time out, and if you leave the pages dirty, then it effectively becomes a permanent memory leak. FreeBSD keeps the dirty buffer (always marked as dirty) in memory as long as the device exists: * https://github.com/freebsd/freebsd-src/commit/e4e8fec98ae986357cdc208b04557dba55a59266 https://github.com/freebsd/freebsd-src/commit/e4e8fec98ae986... You only toss the data when the device associated with the data goes away: * https://github.com/freebsd/freebsd-src/commit/46b5afe7b1ae0ee655ec863ebe373a3cf16eef1d https://github.com/freebsd/freebsd-src/commit/46b5afe7b1ae0e...
    - Dylan16807 5y ago
      Honestly even if a device disappears it's probably worth keeping around those pending writes for several seconds. It might come back, and if it comes back fast enough it probably hasn't been altered by any other machine.
  - dataflow 5y ago
    Super interesting (I hadn't seen it before), thanks for sharing! I feel like there might be lots of potential solutions they're ignoring though. > Sooner or later the writes will time out, and if you leave the pages dirty, then it effectively becomes a permanent memory leak. Why not just get rid of the pages after the underlying medium is removed? Especially since, after that point, you wouldn't be able to guarantee much about the device's contents anyway, even if it were plugged back in. Also, even if I buy that this is a trade-off with a permanent memory leak (which I don't, for lots of reasons like the above), isn't that better than outright corruption...? At least you can reboot to get rid of a leak. There's no guarantee you can get back corrupted data!
    - belter 5y ago
      Looking at it from another side...The whole Direct I/O path looks like a worst can of worms...maybe it's easier to review again fsync :-) "...The exact semantics of Direct I/O (O_DIRECT) are not well specified. It is not a part of POSIX, or SUS, or any other formal standards specification. The exact meaning of O_DIRECT has historically been negotiated in non-public discussions between powerful enterprise database companies and proprietary Unix systems, and its behaviour has generally been passed down as oral lore rather than as a formal set of requirements and specifications..." "The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances." -- Linus https://ext4.wiki.kernel.org/index.php/Clarifying_Direct_IO%27s_Semantics https://ext4.wiki.kernel.org/index.php/Clarifying_Direct_IO%...
- cbsmith 5y ago
  There's an art to reading specs for standards like POSIX. ;-) This isn't the only place where POSIX leaves things up to being an implementation detail.

Kranar 5y ago

You're quoting a sentence fragment of a non-normative section of the spec, in other words you're half quoting a part of the spec that is not intended to be interpreted formally, literally, or even required to be adhered to. The rationale section of the POSIX spec is strictly informative. The fact that you left out the part of the sentence that says "The fsync() function is intended to..." suggests that you may be intentionally hiding this fact to support your point. Even if we go by the non-normative text, that section explicitly states that the spec allows for, and I quote "It is explicitly intended that a null implementation is permitted." At any rate, the normative section of the spec states only the following: "The fsync() function shall request that all data for the open file descriptor named by fildes is to be transferred to the storage device associated with the file described by fildes. The nature of the transfer is implementation-defined." Emphasis mine. It would be unwise to dismiss the wealth of resources that have been provided to you in other comments to better understand this issue, all of them quite extensive and written by experts on this subject, just because of an informal section of the standard

thaumasiotes 5y ago
> The fact that you left out the part of the sentence that says "The fsync() function is intended to..." suggests that you may be intentionally hiding this fact to support your point. It's very difficult to read this accusation as having been made in good faith, since the person you're accusing of intentionally omitting part of a quote already quoted that very text in their previous comment: >>>> The tl;dr (and blunt truth) is that Linux doesn't implement POSIX's semantics of fsync() correctly. The standard says "the fsync() function is intended to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk".
- Kranar 5y ago
  >...the person you're accusing of intentionally omitting part of a quote already quoted that very text in their previous comment: That was edited back in after I made my comment (this is admitted to by OP themselves). To the extent that I make an accusation it's that it's absurd to think you can dismiss the wealth of information on this subject and come to a remotely comprehensive understanding of how to implement reliable IO functions by simply reading a couple of sentences of the POSIX spec and dismiss the numerous other sources that go into very fine detail on this subject. This is not a subject you just read a few sentences on and then make bold claims about how Linux doesn't implement fsync properly, implying that the experts who have gone to great lengths to implement fsync on Linux just don't understand it; that all the papers on this subject, the numerous mailing list posts, the very submission itself, can all readily be dismissed as a simple failure to read what is as clear as day as if none of the people discussing this issue have ever bothered to read the spec.