10 ms·

Fsyncgate: Errors on fsync are unrecoverable (2018)

219 points by Darkstryder 7y ago

mjw1007 7y ago
The other half of the thread was on linux-ext4, starting here: https://lists.openwall.net/linux-ext4/2018/04/10/33 https://lists.openwall.net/linux-ext4/2018/04/10/33 The part I found most interesting was here: https://lists.openwall.net/linux-ext4/2018/04/12/8 https://lists.openwall.net/linux-ext4/2018/04/12/8 where the ext4 maintainer writes: « The solution we use at Google is that we watch for I/O errors using a completely different process that is responsible for monitoring machine health. It used to scrape dmesg, but we now arrange to have I/O errors get sent via a netlink channel to the machine health monitoring daemon. » He later says that the netlink channel stuff was never submitted to the upstream kernel. It all feels like a situation where the people maintaining this code knew deep down that it wasn't really up to scratch, but were sufficiently used to workarounds ("use direct io", "scrape dmesg") that they no longer thought of it as a problem.
- derefr 7y ago
  > It all feels like a situation where the people maintaining this code knew deep down that it wasn't really up to scratch, but were sufficiently used to workarounds ("use direct io", "scrape dmesg") that they no longer thought of it as a problem. It's an ops-level solution. The one thing you never really do, as an ops person, is reach into the black-box components that make up your infrastructure to fix them at an architectural level. That's not your job. The system is already in production; it's your job to make it run, without changing it—to shim around the black boxes to make them work despite their architectural flaws. Sure, you can file a ticket with the dev team about how dumb you think an architectural choice was—but, in most businesses, most of the black-box components you're working with are third-party, so filing a ticket isn't likely to get you much. (Google is an exception, but even Google has to hire their SREs from somewhere, and they'll come in enculturated to the "it's all black boxes and we're here to ziptie them together" mindset.) And, even in an environment like Google's where you have access to the dev-teams of every component you're running, dev cycles are still longer than ops SLA deadlines. Ops solutions are chosen because they're quick to get into production. (Component can't handle the load because it's coded poorly? Replicate it and throw a load balancer in front! Five minute job.) So even if you do file that ticket, you've still got to solve the problem in the here-and-now. And once you do solve the immediate problem, it's no longer a hair-on-fire problem, so that ticket isn't going to be very high-priority to fix.
  - jimmaswell 7y ago
    If fixing the thing itself is easier, and also benefits everyone else, why not?
    - derefr 7y ago
      Fixing the thing isn't usually easier. In an ops environment, you're not dealing with Git repos full of source code that you can just patch and run through CI; you're usually dealing with pre-built packages from an apt server, or pre-built Docker images, or pre-built virtual-appliance VMs. And usually these are all created in drastically-varying, heterogeneous ways, many again by third-parties (public Apt packages, public Docker images, etc.), but even if internal, then by different internal teams with different favored approaches toward building and deploying things. From the position in the pipeline where ops people operate, it's more work to figure out how to fork a build of software X, patch it, build it, and get out the same kind of artifact your deploy pipeline is expecting, hosted in the same place; as it is to just throw more spaghetti at the wall to solve the problem. Half of the reason behind Google's monorepo, and the focus on Golang, is to empower SREs to patch broken components when possible, by making there be only One True Way to do that, and only one language (without weird per-project DSLs) to know. But that still doesn't completely fix the problem, since you still need to understand the architecture of the software project, and sometimes that's the hardest part.
  - mjw1007 7y ago
    I don't think there was anything wrong with Google using this workaround. But Google were also employing the ext4 maintainer, and fixing those components presumably was part of his job.
  - kbenson 7y ago
    It depends quite a bit on the company and the software they are running. I've worked at places that use open source for the majority of services they run on the servers specifically because they can both look at the source as an advanced form of debugging, and provide local patches to software until it's accepted upstream (or indefinitely if it's not accepted or too company specific). A lot of patches for open source projects come from companies like this. Is there also a lot of working around buggy software? Yes. It's not always that way though, and if it's gotten to the point that you have devs for that software on staff, it really shouldn't be, because if those people who are financially motivated to care don't, who will?
- zokier 7y ago
  That was also suggested in the pg list: > An crazy idea would be to have a daemon that checks the [kernel] logs and stops Postgres when it seems something wrong. > [..] you could have stable log messages or implement some kind of "fsync error log notification" via whatever is the most sane way to get this out of kernel.
- ajross 7y ago
  I dunno, hardware errors really aren't a thing that is well handled by local APIs. I mean, it all feels to me like the PG folks (and you) are expecting this to have all been fine if they just called fsync() a second time to be sure the busted write got to disk, and then everything would have been OK. No, that's not the way it works. In virtually all circumstances, that second write is going to fail too, because the machine done broke! And even if it works, you have a situation where the machine broke and something somewhere worked around it, and you need to be screaming like crazy for an operator to fix it. That's not an API situation, that's a deployment thing, and it's exactly the regime where Google applied the protocols that you don't seem to like. That's not to say that the linux fsync behavior is correct here. I think clearing the error having reported it once is kinda broken too. But... fixing that doesn't actually "fix" the real problem. Really the best suggestion in that thread was up at the top, where it was pointed out that ext3/4 support remounting the filesystem read-only on errors, which sounds like a great policy to me.
  - mjw1007 7y ago
    It seems pretty clear from the first message in the thread that what the postgres code (and so the "PG folks") was expecting was that fsync would continue to report errors as long as it hadn't succeeded. That way the checkpoint wouldn't complete, and the relevant write-ahead log files wouldn't get deleted (and "checkpoints aren't happening and WAL is piling up" is the sort of thing people already monitor).
    - ajross 7y ago
      > the sort of thing people already monitor Exactly! So it's a devops solution either way. The choice is between one based on system-level monitoring and one based on tool-level error reporting. In neither case does a non-broken fsync() actually fix anyone's problem. My point was that I don't see tools like remount-ro and dmesg monitoring as "workarounds" for an underlying problem. They're correct solutions implemented at the right level of abstraction. They just happen not to be shipped with PostgreSQL, which... I mean, yeah, basically I don't care much either way.
saagarjha 7y ago
I didn't read the entire thread, so maybe this was answered: has anyone actually made a system that's "fully" correct with regards to file system errors? Most people throw them away, but even programs that try to account for them get them wrong on some system (or the system changes behavior from out under them…). Is there a library that does this?
- SomeOldThrow 7y ago
  Sqlite has put immense effort into this—it’s a better fopen! That said, my understanding is that handling this with cross-platform, posix-only code is mostly impossible due to behavior like what’s described in the article.
- loeg 7y ago
  FreeBSD gets the kernel side correct (dirty unwritable blocks continue to report IO errors, including to fsync()) so you can at least build reliable libraries and applications on top of it.
- Annatar 7y ago
  Solaris, from which came OpenSolaris, which was forked into illumos, from which SmartOS is built. fsync(2) there does exactly what POSIX says: it will only return success if the write safely made it to disk. This is the #1 reason why all my infrastructure runs on a combination of Solaris and SmartOS and the primary reason why I don't run GNU/Linux on anything that's mine. #2 reason is that Solaris / illumos and therefore SmartOS will not overcommit memory, whereas in GNU/Linux this must be explicitly disabled, since overcommit is enabled by default.
ChuckMcM 7y ago
This is sad, I know that one large user of Linux found this problem in 2009 or so and fixed it for the version of Linux they used in their fleet of servers. I am surprised it didn't make it upstream from then.
- beering 7y ago
  I don't know if it's the same person, but someone working on Atlassian's cloud offerings said he reported this issue (along with a working patch) to Postgres and they declined it. Sounds like what they do for cloud services is run a lot of Postgres instances where the DB data is on a big NFS, and this bug was a bigger problem due to how NFS works. But after patching the fsync handling in Postgres, they continued using Postgres-on-NFS with great success. Unfortunately I can't find the original comment, but I think it was on another HN story about Postgres+fsync. Edit: I found it: https://news.ycombinator.com/item?id=19126601 https://news.ycombinator.com/item?id=19126601 and I may or may not have hallucinated the part about Atlassian.
  - dboreham 7y ago
    In case someone reads this and gets the impression that running PG over NFS is in general safe or a good idea, I'm pretty sure it still isn't, unless like the OP you have a complete understanding of what you're doing.
    - merb 7y ago
      OT: it depends on how much writes/reads you get. for most people nfs is probably an easier solution for HA on postgres than anything else.
    - rosser 7y ago
      I have quite successfully run pg atop NFS — even, in limited, point-solution type roles, in production. My experience doing that didn't leave me with the impression that it was a particularly egregious thing to do, though I would definitely take many, many additional steps to ensure redundancy and availability if I were going to use it more generally. You're right though: you really do want to have some idea what you're doing, if you're going to go there. Source: my day job is PostgreSQL DBA, and has been for ~15 years now. EDIT: Phrasing.
tobias3 7y ago
Well, at least in this case one can abort (or go into some read-only mode) in case of fsync() returning failure. With most storage media that is the correct thing to do anyway. Having multiple processes and having the fsync() error returned to only one of them is problematic though. I recently found out that syncfs() doesn't return an error at all in most cases (through data loss :/). It's being worked on ... https://lkml.org/lkml/2018/6/1/640 https://lkml.org/lkml/2018/6/1/640 It's astonishing that such critical issues are still present in such a widely used piece of software.
- techslave 7y ago
  > Well, at least in this case one can abort (or go into some read-only mode) in case of fsync() returning failure. the problem here isn’t if fsync() returns failure. it’s that it always returns success, even if it failed.
  - masklinn 7y ago
    > it’s that it always returns success, even if it failed. It doesn't (except on the most broken of old Linux kernels, and even then it mostly lost async write errors). Rather it's that on most systems (basically all of them except FreeBSD and Illumos, possibly OpenBSD after some recent changes) fsync will only report errors once, but that call will clear the flags and subsequent fsync calls will succeed (unless new errors have occurred). Basically, you can only rely on fsync reporting errors having happened since the last fsync, which is obviously concerning for all sort of reasons (not least being concurrent updates). They also mention a bit about fsync not necessarily reporting errors from before the file was opened.
- deleted 7y ago
  [deleted]
pknopf 7y ago
Imagine a juggler, juggling 5 balls at a time. At a set period of time, he will drop one ball and accept another ball thrown at him. He handles this very well because there is order/cadence. Now imagine asking him to support being thrown balls randomly, at any interval. He may make it work, but I'd imagine he will stutter a bit. In my experience, anytime you interrupt the page cache's normal routine, it stutters everything. I've seen the "sync" command freeze my Ubuntu machine (music player, GUI, etc). I work on embedded devices, and my employer wanted to reduce the window at which data-loss would happen for 30mb+ files (video capture). It wasn't a supported use case, "But why not! It makes our product theoretically better!" I put my foot down. We aren't touching page-cache until there is a clear benefit to the user. It almost got me fired, but good riddance if so.
- mjw1007 7y ago
  The annoying thing is that the Postgres people don't even really need "sync this file right now" for the main data files. AIUI what they'd ideally like is "sync this file reasonably soon, and let me know when it's finished".
  - the8472 7y ago
    io_uring supports issuing async sync_file_range requests, which can be used to do most of the heavy lifting. You will still need a final fsync to also get the metadata written, but that should faster at that point.
    - cyphar 7y ago
      It should be noted that io_uring landed (in 5.1) a fair while after the whole Postgres back-and-forth started in 2018.
      - masklinn 7y ago
        It should also be noted that even sync_file_range's manpage warns you against using it, which tells you how reliable that thing is.
AnssiH 7y ago
Related article: PostgreSQL's fsync() surprise https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/ (April 18, 2018) And the followup coverage from LSFMM summit (linked also in the OP discussion): https://lwn.net/Articles/752613/ https://lwn.net/Articles/752613/
- beering 7y ago
  Upvoted this because the LWN article seems to give a much better picture of the what and why, including key kernel devs explaining what tgl called "kernel brain damage". The posted danluu.com link seems to be just the pgsql mailing list.
bsaul 7y ago
I read the start and the end of thread but couldn’t get an understranding of what the current situation is : did linux update its fsync behavior ? Does pg now panics on linux on the first fsync ?
- pgaddict 7y ago
  Linux kernel behavior is still the same. The error reporting issues (failure to report I/O errors in various cases) has been fixed on recent kernels, AFAIK. PostgreSQL now PANICs on I/O errors during fsync, forcing a recovery.
a-dub 7y ago
If fsync() fails isn't valhalla lost anyhow? If you can't write things down because your pencil is broken, probably time to stop what you're doing and get a new pencil. If the kernel can't flush dirty write buffers, maybe it's time to send up a flag and panic in the kernel itself?
- tlb 7y ago
  That can happen when you remove a removable storage device, like a USB drive. You don't want to panic the kernel for that.
  - Gibbon1 7y ago
    Reminds me. On CPM, hit save when the drive door is open, so sorry. All gone.
  - wruza 7y ago
    FreeBSD lived with that up to circa 2010, afair. Few fast-pulling desktop linux trolls complained at the time, but nothing serious. Anyway, if given such choice, as a user/admin I would prefer an instant fail with a panic message, rather than a corrupted 10gb online client database with no clear recovery path. I’m always scared by this kind of story when something fails to fail underneath and I even didn’t ever hear of it.
    - Dylan16807 7y ago
      Isn't your data already corrupted most of the time when writes start to fail? If the drive is gone, a panic won't help or hurt, it will just ruin everything else you have open. Redoing failed writes should be the method that ends in the least amount of corruption.
      - wruza 7y ago
        Not going into details, at the end of the day one should have a plan to recover their value from a potential failure. By the means of version control systems, transaction logs, backups, mirrors, etc. But if a failure makes its way to a recovery storage undetected, the entire plan is doomed. I don’t care if I lose few last hours of my local work, since this time is buffered in a project or is not precious at all (like game saves). That throws back me a little, but not anyone else. I also don’t really care about losing downloads or experimental files. But when it comes to a business database or a project repository, that value can be orders of magnitude bigger and can cost responsible one their job at the very least. “Redoing failed writes should be the method that ends in the least amount of corruption.” But in fact it led to a database corruption that was only fixed by some black magic. As far as I understood, they did manual rollback of failed writes. I wouldn’t blame anyone for subj bug even if met it by myself — writing low-level software is hard, the behavior was unintended, and both linux, xfs & pg are free as in beer and free of any charges. But if it happened and was not detected for longer than a recovery window allows it, costs were imminent. That’s why this trade-off is easy on a distance.
dang 7y ago
At the time: https://news.ycombinator.com/item?id=16974033 https://news.ycombinator.com/item?id=16974033 This February: https://news.ycombinator.com/item?id=19119991 https://news.ycombinator.com/item?id=19119991 https://news.ycombinator.com/item?id=19238121 https://news.ycombinator.com/item?id=19238121
hedora 7y ago
Every computer component can fail in arbitrary ways, including drives. If you’re not robust against that, then when things like fsync fail, then you’ll lose availability and/or data. Even though Linux’s fsync behavior is clearly broken, it is far from the craziest behavior I’ve seen from the I/O stack. Anyway, the main lesson here is that untested error handling is worse than no error handling. They should have figured out how to test that this path actually proceeds correctly (on real, intermittently failing hardware) or just panicked the process.
- bonzini 7y ago
  It is broken but there's no behavior that isn't, and the Postgres developers quickly understood why changing Linux's behavior isn't really possible. From https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/: "Linux is not unique in behaving this way; OpenBSD and NetBSD can also fail to report write errors to user space. [...] If some process was copying a lot of data to that drive, the result will be an accumulation of dirty pages in memory, perhaps to the point that the system as a whole runs out of memory for anything else [...] a fair amount of attention was paid to the idea that write failures should result in the affected pages being kept in memory, in their dirty state. But the PostgreSQL developers had quickly moved on from that idea and were not asking for it".
  - pgaddict 7y ago
    > It is broken but there's no behavior that isn't, and the Postgres developers quickly understood why changing Linux's behavior isn't really possible. I think there's still a fair number of PostgreSQL developers who think the way Linux kernel behaves makes the fsync() API rather difficult to use for anything but the simplest scenarios. The reason why the community decided to accept it was the realization that there's about 0.001% chance of convincing kernel devs to change it, and the fact that we'd still have to deal with existing kernels in foreseeable future.
- pgaddict 7y ago
  > Every computer component can fail in arbitrary ways, including drives. > > If you’re not robust against that, then when things like fsync fail, then you’ll lose availability and/or data. The fact is that often the I/O issues are temporary, and those situations are becoming more and more common (think running out of disk space with thin provisioning, networking issues with NFS, etc.). So it might be quite valuable to handle those issues gracefully, without essentially crashing the database (which is pretty much what PANIC does). > Even though Linux’s fsync behavior is clearly broken, it is far from the craziest behavior I’ve seen from the I/O stack. "clearly broken" might be bit too harsh, but it certainly makes it way harder to use. > Anyway, the main lesson here is that untested error handling is worse than no error handling. They should have figured out how to test that this path actually proceeds correctly (on real, intermittently failing hardware) or just panicked the process. That's true, of course. It's a sad fact of life that error paths are the least tested part of almost any code base. It's however also true that testing I/O errors is pretty difficult to do (especially before "error" dm target), and a significant part of the fsyncgate was that kernel was not reporting errors reliably. It's also true that the behavior is somewhat filesystem-specific (some keep the data in page cache but marked as "not dirty", some will discard the data, ...). That makes testing pretty hard.
asdfasgasdgasdg 7y ago
One thing I took away from this thread: when someone tells you something surprising, it's best to ask, rather than deny. It doesn't look good to say things like: "Moreover, POSIX is entirely clear that successful fsync means all preceding writes for the file have been completed, full stop, doesn't matter when they were issued." When you have plainly not actually verified that it is the case. Instead, you could say, "doesn't Posix say . . .?" This has the following benefits: you avoid egg on your face, the conversation takes on a less aggressive tone, and problems are resolved quicker.
- mehrdadn 7y ago
  100% agree with that takeaway in general, except that in this case I think POSIX does actually say this. I pointed out and discussed this a few months ago [1] (and it's already discussed on the page as well), but basically, those who disagree believe that fsync's task is to merely "send" the data to the storage device, not actually make sure it's written persistently... which I see as being blatantly inconsistent with "to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded to the disk". There's nothing on the page that says "data since the last call to fsync", and that sentence says that was definitely not the intention, and yet somehow that's how people read it to support the notion that current implementations are correct (or vice-versa: they use current implementations as evidence that this is the correct reading). [1] https://news.ycombinator.com/item?id=19128228 https://news.ycombinator.com/item?id=19128228
  - asdfasgasdgasdg 7y ago
    Perhaps? The people in the thread seem fairly convinced that's not what it says. When I read it, I see "... all data for the open file descriptor named by fildes is to be transferred to the storage device ..." So, in the event of a failure in the underlying hardware, the data may have been transferred, but subsequently an error has occurred. I don't see how the sentence obliges the implementation to transfer multiple times. I'm not an expert though. In any case, the problem is with the attitude and how opinions are expressed, not the exact state of which opinion is being expressed. :) Also, it is plainly not "entirely clear" since we two apparently reasonable people disagree on what the sentence means. So even if the person is right about what was ultimately intended when the spec was written, they are still incorrect in their view that it's clear what POSIX says.
based2 7y ago
PostgreSQL will now PANIC on fsync() failure https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=9ccdd7f66e3324d2b6d3dec282cfa9ff084083f1 https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit... https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/
verisimilitudes 7y ago
I sure wonder how IBM mainframes and other computer systems handle this intractable failure case. Joking aside, here's an excerpt from ''The UNIX-HATERS Handbook'': Only the Most Perfect Disk Pack Need Apply One common problem with Unix is perfection: while offering none of its own, the operating system demands perfection from the hardware upon which it runs. That's because Unix programs usually don't check for hardware errors--they just blindly stumble along when things begin to fail, until they trip and panic. (Few people see this behavior nowadays, though, because most SCSI hard disks do know how to detect and map out blocks as the blocks begin to fail.) ... In recent years, the Unix file system has appeared slightly more tolerant of disk woes simply because modern disk drives contain controllers that present the illusion of a perfect hard disk. (Indeed, when a modern SCSI hard disk controller detects a block going bad, it copies the data to another block elsewhere on the disk and then rewrites a mapping table. Unix never knows what happened.) But, as Seymour Cray used to say, ''You can't fake what you don't have.'' Sooner or later, the disk goes bad, and then the beauty of UFS shows through.
- ds2643 7y ago
  lol hi alex
patrec 7y ago
It's kinda depressing that after a quarter of a century of work in the region of a hundred thousand dev years and billions of dollars of investment the world's most used operating system fails at the most basic tasks like reliably writing files or allocating memory.
fortran77 7y ago
This is one reason we choose to use Windows Storage Spaces Direct and the tranactional NTFS. They really are better. https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-direct-overview https://docs.microsoft.com/en-us/windows-server/storage/stor...
Kenji 7y ago
That moment when I read this text and whenever it says 'Pg' I read "Paul Graham" so, in my head, the text says things like : Paul Graham should PANIC on fsync() EIO return. or Paul Graham wrote some blocks, which went to OS dirty buffers for writeback
Too 7y ago
So if fsync fails, what is one supposed to do? You can't retry it and you don't know how much of the file that has been synced? Only feasible option is to create a completely new file and retry writing there? And if that fails your disk is probably bust or ejected, which should require user interaction about the new file location. Doesn't seem too unreasonable? This would require you to have the complete file contents elsewhere so you can rewrite it? Or would it still be possible to read from the original file being in the unflushed buffer? And in the disk ejected+remounted case, the old contents should still be there intact thanks to ext4 journaling?
ysleepy 7y ago
Love that FreeBSD is doing things right - and has been for 20 years. https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors
jorangreef 7y ago
2007, Linus rant: https://lkml.org/lkml/2007/1/10/233 https://lkml.org/lkml/2007/1/10/233 The right way to do it is to just not use O_DIRECT. The whole notion of "direct IO" is totally braindamaged. Just say no. This is your brain: O This is your brain on O_DIRECT: . Any questions? I should have fought back harder. There really is no valid reason for EVER using O_DIRECT. You need a buffer whatever IO you do, and it might as well be the page cache. There are better ways to control the page cache than play games and think that a page cache isn't necessary. So don't use O_DIRECT. Use things like madvise() and posix_fadvise() instead. 2019, how things are: This is your brain: O This is your brain on O_DIRECT: . And... this is your brain when cached: ?! The right way to do it is to just use O_DIRECT. The whole notion of "kernel IO" is fsync and games. Just say no.

219 points by Darkstryder 7y ago

The other half of the thread was on linux-ext4, starting here: https://lists.openwall.net/linux-ext4/2018/04/10/33 https://lists.openwall.net/linux-ext4/2018/04/10/33 The part I found most interesting was here: https://lists.openwall.net/linux-ext4/2018/04/12/8 https://lists.openwall.net/linux-ext4/2018/04/12/8 where the ext4 maintainer writes: « The solution we use at Google is that we watch for I/O errors using a completely different process that is responsible for monitoring machine health. It used to scrape dmesg, but we now arrange to have I/O errors get sent via a netlink channel to the machine health monitoring daemon. » He later says that the netlink channel stuff was never submitted to the upstream kernel. It all feels like a situation where the people maintaining this code knew deep down that it wasn't really up to scratch, but were sufficiently used to workarounds ("use direct io", "scrape dmesg") that they no longer thought of it as a problem.

derefr 7y ago
> It all feels like a situation where the people maintaining this code knew deep down that it wasn't really up to scratch, but were sufficiently used to workarounds ("use direct io", "scrape dmesg") that they no longer thought of it as a problem. It's an ops-level solution. The one thing you never really do, as an ops person, is reach into the black-box components that make up your infrastructure to fix them at an architectural level. That's not your job. The system is already in production; it's your job to make it run, without changing it—to shim around the black boxes to make them work despite their architectural flaws. Sure, you can file a ticket with the dev team about how dumb you think an architectural choice was—but, in most businesses, most of the black-box components you're working with are third-party, so filing a ticket isn't likely to get you much. (Google is an exception, but even Google has to hire their SREs from somewhere, and they'll come in enculturated to the "it's all black boxes and we're here to ziptie them together" mindset.) And, even in an environment like Google's where you have access to the dev-teams of every component you're running, dev cycles are still longer than ops SLA deadlines. Ops solutions are chosen because they're quick to get into production. (Component can't handle the load because it's coded poorly? Replicate it and throw a load balancer in front! Five minute job.) So even if you do file that ticket, you've still got to solve the problem in the here-and-now. And once you do solve the immediate problem, it's no longer a hair-on-fire problem, so that ticket isn't going to be very high-priority to fix.
- jimmaswell 7y ago
  If fixing the thing itself is easier, and also benefits everyone else, why not?
  - derefr 7y ago
    Fixing the thing isn't usually easier. In an ops environment, you're not dealing with Git repos full of source code that you can just patch and run through CI; you're usually dealing with pre-built packages from an apt server, or pre-built Docker images, or pre-built virtual-appliance VMs. And usually these are all created in drastically-varying, heterogeneous ways, many again by third-parties (public Apt packages, public Docker images, etc.), but even if internal, then by different internal teams with different favored approaches toward building and deploying things. From the position in the pipeline where ops people operate, it's more work to figure out how to fork a build of software X, patch it, build it, and get out the same kind of artifact your deploy pipeline is expecting, hosted in the same place; as it is to just throw more spaghetti at the wall to solve the problem. Half of the reason behind Google's monorepo, and the focus on Golang, is to empower SREs to patch broken components when possible, by making there be only One True Way to do that, and only one language (without weird per-project DSLs) to know. But that still doesn't completely fix the problem, since you still need to understand the architecture of the software project, and sometimes that's the hardest part.
- mjw1007 7y ago
  I don't think there was anything wrong with Google using this workaround. But Google were also employing the ext4 maintainer, and fixing those components presumably was part of his job.
- kbenson 7y ago
  It depends quite a bit on the company and the software they are running. I've worked at places that use open source for the majority of services they run on the servers specifically because they can both look at the source as an advanced form of debugging, and provide local patches to software until it's accepted upstream (or indefinitely if it's not accepted or too company specific). A lot of patches for open source projects come from companies like this. Is there also a lot of working around buggy software? Yes. It's not always that way though, and if it's gotten to the point that you have devs for that software on staff, it really shouldn't be, because if those people who are financially motivated to care don't, who will?
zokier 7y ago
That was also suggested in the pg list: > An crazy idea would be to have a daemon that checks the [kernel] logs and stops Postgres when it seems something wrong. > [..] you could have stable log messages or implement some kind of "fsync error log notification" via whatever is the most sane way to get this out of kernel.
ajross 7y ago
I dunno, hardware errors really aren't a thing that is well handled by local APIs. I mean, it all feels to me like the PG folks (and you) are expecting this to have all been fine if they just called fsync() a second time to be sure the busted write got to disk, and then everything would have been OK. No, that's not the way it works. In virtually all circumstances, that second write is going to fail too, because the machine done broke! And even if it works, you have a situation where the machine broke and something somewhere worked around it, and you need to be screaming like crazy for an operator to fix it. That's not an API situation, that's a deployment thing, and it's exactly the regime where Google applied the protocols that you don't seem to like. That's not to say that the linux fsync behavior is correct here. I think clearing the error having reported it once is kinda broken too. But... fixing that doesn't actually "fix" the real problem. Really the best suggestion in that thread was up at the top, where it was pointed out that ext3/4 support remounting the filesystem read-only on errors, which sounds like a great policy to me.
- mjw1007 7y ago
  It seems pretty clear from the first message in the thread that what the postgres code (and so the "PG folks") was expecting was that fsync would continue to report errors as long as it hadn't succeeded. That way the checkpoint wouldn't complete, and the relevant write-ahead log files wouldn't get deleted (and "checkpoints aren't happening and WAL is piling up" is the sort of thing people already monitor).
  - ajross 7y ago
    > the sort of thing people already monitor Exactly! So it's a devops solution either way. The choice is between one based on system-level monitoring and one based on tool-level error reporting. In neither case does a non-broken fsync() actually fix anyone's problem. My point was that I don't see tools like remount-ro and dmesg monitoring as "workarounds" for an underlying problem. They're correct solutions implemented at the right level of abstraction. They just happen not to be shipped with PostgreSQL, which... I mean, yeah, basically I don't care much either way.

saagarjha 7y ago

I didn't read the entire thread, so maybe this was answered: has anyone actually made a system that's "fully" correct with regards to file system errors? Most people throw them away, but even programs that try to account for them get them wrong on some system (or the system changes behavior from out under them…). Is there a library that does this?

SomeOldThrow 7y ago
Sqlite has put immense effort into this—it’s a better fopen! That said, my understanding is that handling this with cross-platform, posix-only code is mostly impossible due to behavior like what’s described in the article.
loeg 7y ago
FreeBSD gets the kernel side correct (dirty unwritable blocks continue to report IO errors, including to fsync()) so you can at least build reliable libraries and applications on top of it.
Annatar 7y ago
Solaris, from which came OpenSolaris, which was forked into illumos, from which SmartOS is built. fsync(2) there does exactly what POSIX says: it will only return success if the write safely made it to disk. This is the #1 reason why all my infrastructure runs on a combination of Solaris and SmartOS and the primary reason why I don't run GNU/Linux on anything that's mine. #2 reason is that Solaris / illumos and therefore SmartOS will not overcommit memory, whereas in GNU/Linux this must be explicitly disabled, since overcommit is enabled by default.

ChuckMcM 7y ago

This is sad, I know that one large user of Linux found this problem in 2009 or so and fixed it for the version of Linux they used in their fleet of servers. I am surprised it didn't make it upstream from then.

beering 7y ago
I don't know if it's the same person, but someone working on Atlassian's cloud offerings said he reported this issue (along with a working patch) to Postgres and they declined it. Sounds like what they do for cloud services is run a lot of Postgres instances where the DB data is on a big NFS, and this bug was a bigger problem due to how NFS works. But after patching the fsync handling in Postgres, they continued using Postgres-on-NFS with great success. Unfortunately I can't find the original comment, but I think it was on another HN story about Postgres+fsync. Edit: I found it: https://news.ycombinator.com/item?id=19126601 https://news.ycombinator.com/item?id=19126601 and I may or may not have hallucinated the part about Atlassian.
- dboreham 7y ago
  In case someone reads this and gets the impression that running PG over NFS is in general safe or a good idea, I'm pretty sure it still isn't, unless like the OP you have a complete understanding of what you're doing.
  - merb 7y ago
    OT: it depends on how much writes/reads you get. for most people nfs is probably an easier solution for HA on postgres than anything else.
  - rosser 7y ago
    I have quite successfully run pg atop NFS — even, in limited, point-solution type roles, in production. My experience doing that didn't leave me with the impression that it was a particularly egregious thing to do, though I would definitely take many, many additional steps to ensure redundancy and availability if I were going to use it more generally. You're right though: you really do want to have some idea what you're doing, if you're going to go there. Source: my day job is PostgreSQL DBA, and has been for ~15 years now. EDIT: Phrasing.

tobias3 7y ago

Well, at least in this case one can abort (or go into some read-only mode) in case of fsync() returning failure. With most storage media that is the correct thing to do anyway. Having multiple processes and having the fsync() error returned to only one of them is problematic though. I recently found out that syncfs() doesn't return an error at all in most cases (through data loss :/). It's being worked on ... https://lkml.org/lkml/2018/6/1/640 https://lkml.org/lkml/2018/6/1/640 It's astonishing that such critical issues are still present in such a widely used piece of software.

techslave 7y ago
> Well, at least in this case one can abort (or go into some read-only mode) in case of fsync() returning failure. the problem here isn’t if fsync() returns failure. it’s that it always returns success, even if it failed.
- masklinn 7y ago
  > it’s that it always returns success, even if it failed. It doesn't (except on the most broken of old Linux kernels, and even then it mostly lost async write errors). Rather it's that on most systems (basically all of them except FreeBSD and Illumos, possibly OpenBSD after some recent changes) fsync will only report errors once, but that call will clear the flags and subsequent fsync calls will succeed (unless new errors have occurred). Basically, you can only rely on fsync reporting errors having happened since the last fsync, which is obviously concerning for all sort of reasons (not least being concurrent updates). They also mention a bit about fsync not necessarily reporting errors from before the file was opened.
deleted 7y ago
[deleted]

pknopf 7y ago

Imagine a juggler, juggling 5 balls at a time. At a set period of time, he will drop one ball and accept another ball thrown at him. He handles this very well because there is order/cadence. Now imagine asking him to support being thrown balls randomly, at any interval. He may make it work, but I'd imagine he will stutter a bit. In my experience, anytime you interrupt the page cache's normal routine, it stutters everything. I've seen the "sync" command freeze my Ubuntu machine (music player, GUI, etc). I work on embedded devices, and my employer wanted to reduce the window at which data-loss would happen for 30mb+ files (video capture). It wasn't a supported use case, "But why not! It makes our product theoretically better!" I put my foot down. We aren't touching page-cache until there is a clear benefit to the user. It almost got me fired, but good riddance if so.

mjw1007 7y ago
The annoying thing is that the Postgres people don't even really need "sync this file right now" for the main data files. AIUI what they'd ideally like is "sync this file reasonably soon, and let me know when it's finished".
- the8472 7y ago
  io_uring supports issuing async sync_file_range requests, which can be used to do most of the heavy lifting. You will still need a final fsync to also get the metadata written, but that should faster at that point.
  - cyphar 7y ago
    It should be noted that io_uring landed (in 5.1) a fair while after the whole Postgres back-and-forth started in 2018.
    - masklinn 7y ago
      It should also be noted that even sync_file_range's manpage warns you against using it, which tells you how reliable that thing is.

AnssiH 7y ago

Related article: PostgreSQL's fsync() surprise https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/ (April 18, 2018) And the followup coverage from LSFMM summit (linked also in the OP discussion): https://lwn.net/Articles/752613/ https://lwn.net/Articles/752613/

beering 7y ago
Upvoted this because the LWN article seems to give a much better picture of the what and why, including key kernel devs explaining what tgl called "kernel brain damage". The posted danluu.com link seems to be just the pgsql mailing list.

bsaul 7y ago

I read the start and the end of thread but couldn’t get an understranding of what the current situation is : did linux update its fsync behavior ? Does pg now panics on linux on the first fsync ?

pgaddict 7y ago
Linux kernel behavior is still the same. The error reporting issues (failure to report I/O errors in various cases) has been fixed on recent kernels, AFAIK. PostgreSQL now PANICs on I/O errors during fsync, forcing a recovery.

a-dub 7y ago

If fsync() fails isn't valhalla lost anyhow? If you can't write things down because your pencil is broken, probably time to stop what you're doing and get a new pencil. If the kernel can't flush dirty write buffers, maybe it's time to send up a flag and panic in the kernel itself?

tlb 7y ago
That can happen when you remove a removable storage device, like a USB drive. You don't want to panic the kernel for that.
- Gibbon1 7y ago
  Reminds me. On CPM, hit save when the drive door is open, so sorry. All gone.
- wruza 7y ago
  FreeBSD lived with that up to circa 2010, afair. Few fast-pulling desktop linux trolls complained at the time, but nothing serious. Anyway, if given such choice, as a user/admin I would prefer an instant fail with a panic message, rather than a corrupted 10gb online client database with no clear recovery path. I’m always scared by this kind of story when something fails to fail underneath and I even didn’t ever hear of it.
  - Dylan16807 7y ago
    Isn't your data already corrupted most of the time when writes start to fail? If the drive is gone, a panic won't help or hurt, it will just ruin everything else you have open. Redoing failed writes should be the method that ends in the least amount of corruption.
    - wruza 7y ago
      Not going into details, at the end of the day one should have a plan to recover their value from a potential failure. By the means of version control systems, transaction logs, backups, mirrors, etc. But if a failure makes its way to a recovery storage undetected, the entire plan is doomed. I don’t care if I lose few last hours of my local work, since this time is buffered in a project or is not precious at all (like game saves). That throws back me a little, but not anyone else. I also don’t really care about losing downloads or experimental files. But when it comes to a business database or a project repository, that value can be orders of magnitude bigger and can cost responsible one their job at the very least. “Redoing failed writes should be the method that ends in the least amount of corruption.” But in fact it led to a database corruption that was only fixed by some black magic. As far as I understood, they did manual rollback of failed writes. I wouldn’t blame anyone for subj bug even if met it by myself — writing low-level software is hard, the behavior was unintended, and both linux, xfs & pg are free as in beer and free of any charges. But if it happened and was not detected for longer than a recovery window allows it, costs were imminent. That’s why this trade-off is easy on a distance.

dang 7y ago

At the time: https://news.ycombinator.com/item?id=16974033 https://news.ycombinator.com/item?id=16974033 This February: https://news.ycombinator.com/item?id=19119991 https://news.ycombinator.com/item?id=19119991 https://news.ycombinator.com/item?id=19238121 https://news.ycombinator.com/item?id=19238121

hedora 7y ago

Every computer component can fail in arbitrary ways, including drives. If you’re not robust against that, then when things like fsync fail, then you’ll lose availability and/or data. Even though Linux’s fsync behavior is clearly broken, it is far from the craziest behavior I’ve seen from the I/O stack. Anyway, the main lesson here is that untested error handling is worse than no error handling. They should have figured out how to test that this path actually proceeds correctly (on real, intermittently failing hardware) or just panicked the process.

bonzini 7y ago
It is broken but there's no behavior that isn't, and the Postgres developers quickly understood why changing Linux's behavior isn't really possible. From https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/: "Linux is not unique in behaving this way; OpenBSD and NetBSD can also fail to report write errors to user space. [...] If some process was copying a lot of data to that drive, the result will be an accumulation of dirty pages in memory, perhaps to the point that the system as a whole runs out of memory for anything else [...] a fair amount of attention was paid to the idea that write failures should result in the affected pages being kept in memory, in their dirty state. But the PostgreSQL developers had quickly moved on from that idea and were not asking for it".
- pgaddict 7y ago
  > It is broken but there's no behavior that isn't, and the Postgres developers quickly understood why changing Linux's behavior isn't really possible. I think there's still a fair number of PostgreSQL developers who think the way Linux kernel behaves makes the fsync() API rather difficult to use for anything but the simplest scenarios. The reason why the community decided to accept it was the realization that there's about 0.001% chance of convincing kernel devs to change it, and the fact that we'd still have to deal with existing kernels in foreseeable future.
pgaddict 7y ago
> Every computer component can fail in arbitrary ways, including drives. > > If you’re not robust against that, then when things like fsync fail, then you’ll lose availability and/or data. The fact is that often the I/O issues are temporary, and those situations are becoming more and more common (think running out of disk space with thin provisioning, networking issues with NFS, etc.). So it might be quite valuable to handle those issues gracefully, without essentially crashing the database (which is pretty much what PANIC does). > Even though Linux’s fsync behavior is clearly broken, it is far from the craziest behavior I’ve seen from the I/O stack. "clearly broken" might be bit too harsh, but it certainly makes it way harder to use. > Anyway, the main lesson here is that untested error handling is worse than no error handling. They should have figured out how to test that this path actually proceeds correctly (on real, intermittently failing hardware) or just panicked the process. That's true, of course. It's a sad fact of life that error paths are the least tested part of almost any code base. It's however also true that testing I/O errors is pretty difficult to do (especially before "error" dm target), and a significant part of the fsyncgate was that kernel was not reporting errors reliably. It's also true that the behavior is somewhat filesystem-specific (some keep the data in page cache but marked as "not dirty", some will discard the data, ...). That makes testing pretty hard.

asdfasgasdgasdg 7y ago

One thing I took away from this thread: when someone tells you something surprising, it's best to ask, rather than deny. It doesn't look good to say things like: "Moreover, POSIX is entirely clear that successful fsync means all preceding writes for the file have been completed, full stop, doesn't matter when they were issued." When you have plainly not actually verified that it is the case. Instead, you could say, "doesn't Posix say . . .?" This has the following benefits: you avoid egg on your face, the conversation takes on a less aggressive tone, and problems are resolved quicker.

mehrdadn 7y ago
100% agree with that takeaway in general, except that in this case I think POSIX does actually say this. I pointed out and discussed this a few months ago [1] (and it's already discussed on the page as well), but basically, those who disagree believe that fsync's task is to merely "send" the data to the storage device, not actually make sure it's written persistently... which I see as being blatantly inconsistent with "to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded to the disk". There's nothing on the page that says "data since the last call to fsync", and that sentence says that was definitely not the intention, and yet somehow that's how people read it to support the notion that current implementations are correct (or vice-versa: they use current implementations as evidence that this is the correct reading). [1] https://news.ycombinator.com/item?id=19128228 https://news.ycombinator.com/item?id=19128228
- asdfasgasdgasdg 7y ago
  Perhaps? The people in the thread seem fairly convinced that's not what it says. When I read it, I see "... all data for the open file descriptor named by fildes is to be transferred to the storage device ..." So, in the event of a failure in the underlying hardware, the data may have been transferred, but subsequently an error has occurred. I don't see how the sentence obliges the implementation to transfer multiple times. I'm not an expert though. In any case, the problem is with the attitude and how opinions are expressed, not the exact state of which opinion is being expressed. :) Also, it is plainly not "entirely clear" since we two apparently reasonable people disagree on what the sentence means. So even if the person is right about what was ultimately intended when the spec was written, they are still incorrect in their view that it's clear what POSIX says.

based2 7y ago

PostgreSQL will now PANIC on fsync() failure https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=9ccdd7f66e3324d2b6d3dec282cfa9ff084083f1 https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit... https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/

verisimilitudes 7y ago

I sure wonder how IBM mainframes and other computer systems handle this intractable failure case. Joking aside, here's an excerpt from ''The UNIX-HATERS Handbook'': Only the Most Perfect Disk Pack Need Apply One common problem with Unix is perfection: while offering none of its own, the operating system demands perfection from the hardware upon which it runs. That's because Unix programs usually don't check for hardware errors--they just blindly stumble along when things begin to fail, until they trip and panic. (Few people see this behavior nowadays, though, because most SCSI hard disks do know how to detect and map out blocks as the blocks begin to fail.) ... In recent years, the Unix file system has appeared slightly more tolerant of disk woes simply because modern disk drives contain controllers that present the illusion of a perfect hard disk. (Indeed, when a modern SCSI hard disk controller detects a block going bad, it copies the data to another block elsewhere on the disk and then rewrites a mapping table. Unix never knows what happened.) But, as Seymour Cray used to say, ''You can't fake what you don't have.'' Sooner or later, the disk goes bad, and then the beauty of UFS shows through.

ds2643 7y ago
lol hi alex

patrec 7y ago

It's kinda depressing that after a quarter of a century of work in the region of a hundred thousand dev years and billions of dollars of investment the world's most used operating system fails at the most basic tasks like reliably writing files or allocating memory.

fortran77 7y ago

This is one reason we choose to use Windows Storage Spaces Direct and the tranactional NTFS. They really are better. https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-direct-overview https://docs.microsoft.com/en-us/windows-server/storage/stor...

Kenji 7y ago

That moment when I read this text and whenever it says 'Pg' I read "Paul Graham" so, in my head, the text says things like : Paul Graham should PANIC on fsync() EIO return. or Paul Graham wrote some blocks, which went to OS dirty buffers for writeback

Too 7y ago

So if fsync fails, what is one supposed to do? You can't retry it and you don't know how much of the file that has been synced? Only feasible option is to create a completely new file and retry writing there? And if that fails your disk is probably bust or ejected, which should require user interaction about the new file location. Doesn't seem too unreasonable? This would require you to have the complete file contents elsewhere so you can rewrite it? Or would it still be possible to read from the original file being in the unflushed buffer? And in the disk ejected+remounted case, the old contents should still be there intact thanks to ext4 journaling?

ysleepy 7y ago

Love that FreeBSD is doing things right - and has been for 20 years. https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors

jorangreef 7y ago

2007, Linus rant: https://lkml.org/lkml/2007/1/10/233 https://lkml.org/lkml/2007/1/10/233 The right way to do it is to just not use O_DIRECT. The whole notion of "direct IO" is totally braindamaged. Just say no. This is your brain: O This is your brain on O_DIRECT: . Any questions? I should have fought back harder. There really is no valid reason for EVER using O_DIRECT. You need a buffer whatever IO you do, and it might as well be the page cache. There are better ways to control the page cache than play games and think that a page cache isn't necessary. So don't use O_DIRECT. Use things like madvise() and posix_fadvise() instead. 2019, how things are: This is your brain: O This is your brain on O_DIRECT: . And... this is your brain when cached: ?! The right way to do it is to just use O_DIRECT. The whole notion of "kernel IO" is fsync and games. Just say no.