12 ms·

The other half of the thread was on linux-ext4, starting here: https://lists.openwall.net/linux-ext4/2018/04/10/33 https://lists.openwall.net/linux-ext4/2018/04

The other half of the thread was on linux-ext4, starting here: https://lists.openwall.net/linux-ext4/2018/04/10/33 https://lists.openwall.net/linux-ext4/2018/04/10/33 The part I found most interesting was here: https://lists.openwall.net/linux-ext4/2018/04/12/8 https://lists.openwall.net/linux-ext4/2018/04/12/8 where the ext4 maintainer writes: « The solution we use at Google is that we watch for I/O errors using a completely different process that is responsible for monitoring machine health. It used to scrape dmesg, but we now arrange to have I/O errors get sent via a netlink channel to the machine health monitoring daemon. » He later says that the netlink channel stuff was never submitted to the upstream kernel. It all feels like a situation where the people maintaining this code knew deep down that it wasn't really up to scratch, but were sufficiently used to workarounds ("use direct io", "scrape dmesg") that they no longer thought of it as a problem.

derefr 7y ago
> It all feels like a situation where the people maintaining this code knew deep down that it wasn't really up to scratch, but were sufficiently used to workarounds ("use direct io", "scrape dmesg") that they no longer thought of it as a problem. It's an ops-level solution. The one thing you never really do, as an ops person, is reach into the black-box components that make up your infrastructure to fix them at an architectural level. That's not your job. The system is already in production; it's your job to make it run, without changing it—to shim around the black boxes to make them work despite their architectural flaws. Sure, you can file a ticket with the dev team about how dumb you think an architectural choice was—but, in most businesses, most of the black-box components you're working with are third-party, so filing a ticket isn't likely to get you much. (Google is an exception, but even Google has to hire their SREs from somewhere, and they'll come in enculturated to the "it's all black boxes and we're here to ziptie them together" mindset.) And, even in an environment like Google's where you have access to the dev-teams of every component you're running, dev cycles are still longer than ops SLA deadlines. Ops solutions are chosen because they're quick to get into production. (Component can't handle the load because it's coded poorly? Replicate it and throw a load balancer in front! Five minute job.) So even if you do file that ticket, you've still got to solve the problem in the here-and-now. And once you do solve the immediate problem, it's no longer a hair-on-fire problem, so that ticket isn't going to be very high-priority to fix.
- jimmaswell 7y ago
  If fixing the thing itself is easier, and also benefits everyone else, why not?
  - derefr 7y ago
    Fixing the thing isn't usually easier. In an ops environment, you're not dealing with Git repos full of source code that you can just patch and run through CI; you're usually dealing with pre-built packages from an apt server, or pre-built Docker images, or pre-built virtual-appliance VMs. And usually these are all created in drastically-varying, heterogeneous ways, many again by third-parties (public Apt packages, public Docker images, etc.), but even if internal, then by different internal teams with different favored approaches toward building and deploying things. From the position in the pipeline where ops people operate, it's more work to figure out how to fork a build of software X, patch it, build it, and get out the same kind of artifact your deploy pipeline is expecting, hosted in the same place; as it is to just throw more spaghetti at the wall to solve the problem. Half of the reason behind Google's monorepo, and the focus on Golang, is to empower SREs to patch broken components when possible, by making there be only One True Way to do that, and only one language (without weird per-project DSLs) to know. But that still doesn't completely fix the problem, since you still need to understand the architecture of the software project, and sometimes that's the hardest part.
- mjw1007 7y ago
  I don't think there was anything wrong with Google using this workaround. But Google were also employing the ext4 maintainer, and fixing those components presumably was part of his job.
- kbenson 7y ago
  It depends quite a bit on the company and the software they are running. I've worked at places that use open source for the majority of services they run on the servers specifically because they can both look at the source as an advanced form of debugging, and provide local patches to software until it's accepted upstream (or indefinitely if it's not accepted or too company specific). A lot of patches for open source projects come from companies like this. Is there also a lot of working around buggy software? Yes. It's not always that way though, and if it's gotten to the point that you have devs for that software on staff, it really shouldn't be, because if those people who are financially motivated to care don't, who will?
zokier 7y ago
That was also suggested in the pg list: > An crazy idea would be to have a daemon that checks the [kernel] logs and stops Postgres when it seems something wrong. > [..] you could have stable log messages or implement some kind of "fsync error log notification" via whatever is the most sane way to get this out of kernel.
ajross 7y ago
I dunno, hardware errors really aren't a thing that is well handled by local APIs. I mean, it all feels to me like the PG folks (and you) are expecting this to have all been fine if they just called fsync() a second time to be sure the busted write got to disk, and then everything would have been OK. No, that's not the way it works. In virtually all circumstances, that second write is going to fail too, because the machine done broke! And even if it works, you have a situation where the machine broke and something somewhere worked around it, and you need to be screaming like crazy for an operator to fix it. That's not an API situation, that's a deployment thing, and it's exactly the regime where Google applied the protocols that you don't seem to like. That's not to say that the linux fsync behavior is correct here. I think clearing the error having reported it once is kinda broken too. But... fixing that doesn't actually "fix" the real problem. Really the best suggestion in that thread was up at the top, where it was pointed out that ext3/4 support remounting the filesystem read-only on errors, which sounds like a great policy to me.
- mjw1007 7y ago
  It seems pretty clear from the first message in the thread that what the postgres code (and so the "PG folks") was expecting was that fsync would continue to report errors as long as it hadn't succeeded. That way the checkpoint wouldn't complete, and the relevant write-ahead log files wouldn't get deleted (and "checkpoints aren't happening and WAL is piling up" is the sort of thing people already monitor).
  - ajross 7y ago
    > the sort of thing people already monitor Exactly! So it's a devops solution either way. The choice is between one based on system-level monitoring and one based on tool-level error reporting. In neither case does a non-broken fsync() actually fix anyone's problem. My point was that I don't see tools like remount-ro and dmesg monitoring as "workarounds" for an underlying problem. They're correct solutions implemented at the right level of abstraction. They just happen not to be shipped with PostgreSQL, which... I mean, yeah, basically I don't care much either way.

derefr 7y ago

> It all feels like a situation where the people maintaining this code knew deep down that it wasn't really up to scratch, but were sufficiently used to workarounds ("use direct io", "scrape dmesg") that they no longer thought of it as a problem. It's an ops-level solution. The one thing you never really do, as an ops person, is reach into the black-box components that make up your infrastructure to fix them at an architectural level. That's not your job. The system is already in production; it's your job to make it run, without changing it—to shim around the black boxes to make them work despite their architectural flaws. Sure, you can file a ticket with the dev team about how dumb you think an architectural choice was—but, in most businesses, most of the black-box components you're working with are third-party, so filing a ticket isn't likely to get you much. (Google is an exception, but even Google has to hire their SREs from somewhere, and they'll come in enculturated to the "it's all black boxes and we're here to ziptie them together" mindset.) And, even in an environment like Google's where you have access to the dev-teams of every component you're running, dev cycles are still longer than ops SLA deadlines. Ops solutions are chosen because they're quick to get into production. (Component can't handle the load because it's coded poorly? Replicate it and throw a load balancer in front! Five minute job.) So even if you do file that ticket, you've still got to solve the problem in the here-and-now. And once you do solve the immediate problem, it's no longer a hair-on-fire problem, so that ticket isn't going to be very high-priority to fix.

jimmaswell 7y ago
If fixing the thing itself is easier, and also benefits everyone else, why not?
- derefr 7y ago
  Fixing the thing isn't usually easier. In an ops environment, you're not dealing with Git repos full of source code that you can just patch and run through CI; you're usually dealing with pre-built packages from an apt server, or pre-built Docker images, or pre-built virtual-appliance VMs. And usually these are all created in drastically-varying, heterogeneous ways, many again by third-parties (public Apt packages, public Docker images, etc.), but even if internal, then by different internal teams with different favored approaches toward building and deploying things. From the position in the pipeline where ops people operate, it's more work to figure out how to fork a build of software X, patch it, build it, and get out the same kind of artifact your deploy pipeline is expecting, hosted in the same place; as it is to just throw more spaghetti at the wall to solve the problem. Half of the reason behind Google's monorepo, and the focus on Golang, is to empower SREs to patch broken components when possible, by making there be only One True Way to do that, and only one language (without weird per-project DSLs) to know. But that still doesn't completely fix the problem, since you still need to understand the architecture of the software project, and sometimes that's the hardest part.
mjw1007 7y ago
I don't think there was anything wrong with Google using this workaround. But Google were also employing the ext4 maintainer, and fixing those components presumably was part of his job.
kbenson 7y ago
It depends quite a bit on the company and the software they are running. I've worked at places that use open source for the majority of services they run on the servers specifically because they can both look at the source as an advanced form of debugging, and provide local patches to software until it's accepted upstream (or indefinitely if it's not accepted or too company specific). A lot of patches for open source projects come from companies like this. Is there also a lot of working around buggy software? Yes. It's not always that way though, and if it's gotten to the point that you have devs for that software on staff, it really shouldn't be, because if those people who are financially motivated to care don't, who will?

zokier 7y ago

That was also suggested in the pg list: > An crazy idea would be to have a daemon that checks the [kernel] logs and stops Postgres when it seems something wrong. > [..] you could have stable log messages or implement some kind of "fsync error log notification" via whatever is the most sane way to get this out of kernel.

ajross 7y ago

I dunno, hardware errors really aren't a thing that is well handled by local APIs. I mean, it all feels to me like the PG folks (and you) are expecting this to have all been fine if they just called fsync() a second time to be sure the busted write got to disk, and then everything would have been OK. No, that's not the way it works. In virtually all circumstances, that second write is going to fail too, because the machine done broke! And even if it works, you have a situation where the machine broke and something somewhere worked around it, and you need to be screaming like crazy for an operator to fix it. That's not an API situation, that's a deployment thing, and it's exactly the regime where Google applied the protocols that you don't seem to like. That's not to say that the linux fsync behavior is correct here. I think clearing the error having reported it once is kinda broken too. But... fixing that doesn't actually "fix" the real problem. Really the best suggestion in that thread was up at the top, where it was pointed out that ext3/4 support remounting the filesystem read-only on errors, which sounds like a great policy to me.

mjw1007 7y ago
It seems pretty clear from the first message in the thread that what the postgres code (and so the "PG folks") was expecting was that fsync would continue to report errors as long as it hadn't succeeded. That way the checkpoint wouldn't complete, and the relevant write-ahead log files wouldn't get deleted (and "checkpoints aren't happening and WAL is piling up" is the sort of thing people already monitor).
- ajross 7y ago
  > the sort of thing people already monitor Exactly! So it's a devops solution either way. The choice is between one based on system-level monitoring and one based on tool-level error reporting. In neither case does a non-broken fsync() actually fix anyone's problem. My point was that I don't see tools like remount-ro and dmesg monitoring as "workarounds" for an underlying problem. They're correct solutions implemented at the right level of abstraction. They just happen not to be shipped with PostgreSQL, which... I mean, yeah, basically I don't care much either way.