5 ms·
The tl;dr (and blunt truth) is that Linux doesn't implement POSIX's semantics of fsync() correctly. The standard says "the fsync() function is intended to assur
by dataflow 5y ago
The tl;dr (and blunt truth) is that Linux doesn't implement POSIX's semantics of fsync() correctly. The standard says "the fsync() function is intended to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk". Instead, Linux merely provides this guarantee for "all data since the last* call to fsync(), whether successful or not", which means data that failed to flush once will continue to fail to get flushed on subsequent calls to fsync(). But nobody I've seen has been willing to admit this is a broken implementation yet. People instead go through extreme contortions to justify the status quo, blaming Postgres for not using a broken fsync() "correctly".
Given how it took some 2 decades for Linux folks to change their minds on other issues, like realizing the notion of a pidfd is a useful one, I'm guessing they might eventually get around to this too, but it might take another couple of decades for views to change.
- a-dub 5y agoto be fair, nothing really perfectly confirmed to POSIX now that i think of it. who remembers the myriad choices in synchronization primitives that could be used for the main loop in preforking apache? (to work around all the different flavors of broken in unix derivatives) i remember one desperate choice falling all the way through to flock() on a sentinel file on disk...
- smoldesu 5y agoProbably because Linux isn't POSIX-compliant (nor does it really try to be)
- masklinn 5y agomacOS is a certified unix, and after the postgresql people took a look at the situation they found out that it also loses data in that situation. So do OpenBSD and NetBSD. Only FreeBSD would avoid data loss on fsync() failure. AFAIK that remains the case, the only major change I'm aware of is that OpenBSD's fsync is now "locked" into failure after the first failure, at least until you close the file (postgres also played silly bugger on that front).
- smoldesu 5y agoThe only certified UNIX you've listed is macOS, and it doesn't surprise me in the slightest that their frankenkernel doesn't fully comply with POSIX spec. Besides that, I don't know what to tell you. The whole business of "true UNIX" is pretty much thrown to the wayside these days anyways, the list of server-ready POSIX-compliant OSes is tiny, and their market share is even smaller.
- belter 5y agoI though the conclusion was that POSIX spec was not clear on this specific scenario, while the Linux kernel team advised the only real solution was for Postgres to move to direct I/O (DIO) instead, something that is still ongoing work? "PostgreSQL's fsync() surprise" https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/ "Asynchronous and "direct" IO support for PostgreSQL" https://www.postgresql.org/message-id/flat/20210901055659.52doj7oc5uglvezv%40alap3.anarazel.de#cf92ae5ad01b7bd2ec5729d6dcf3c529 https://www.postgresql.org/message-id/flat/20210901055659.52...
- dataflow 5y agoWell, I mean, they can claim it's unclear, but to me they're obviously wrong—and you don't have to take my word over theirs; just go read the spec [1]. It's perfectly clear what "all data up to the time of the fsync() call" means. You obviously can't just read that phrase and claim it's in any way synonymous with "all data up to the time of the fsync() call since the previous call to fsync(), even if it failed (!)". "All data until X event" just means all data before X—plain, simple, and unambiguous. It just doesn't mean "all data until X since Y"! [1] https://pubs.opengroup.org/onlinepubs/009695399/functions/fsync.html https://pubs.opengroup.org/onlinepubs/009695399/functions/fs...
- belter 5y agoThis paper: "Can Applications Recover from fsync Failures?" https://www.usenix.org/system/files/atc20-rebello.pdf https://www.usenix.org/system/files/atc20-rebello.pdf argues the following: "#1: Existing file systems do not handle fsync failures uniformly. In an effort to hide cross-platform differences, POSIX is intentionally vague on how failures are handled. Thus, different file systems behave differently after an fsync failure (as seen in Table 1), leading to non-deterministic outcomes for applications that treat all file systems equally. We believe that the POSIX specification for fsync needs to be clarified and the expected failure behavior described in more detail."
- dataflow 5y agoPeople can say whatever they want (even in papers!), and people go through contortions to justify arbitrary things, but like I said, you don't have to take my word any more than you have to take theirs when the spec is right there. You can just read the spec yourself and witness that the meaning is pretty clear.