4 ms·
I dunno, hardware errors really aren't a thing that is well handled by local APIs. I mean, it all feels to me like the PG folks (and you) are expecting this to
by ajross 7y ago
I dunno, hardware errors really aren't a thing that is well handled by local APIs. I mean, it all feels to me like the PG folks (and you) are expecting this to have all been fine if they just called fsync() a second time to be sure the busted write got to disk, and then everything would have been OK.
No, that's not the way it works. In virtually all circumstances, that second write is going to fail too, because the machine done broke! And even if it works, you have a situation where the machine broke and something somewhere worked around it, and you need to be screaming like crazy for an operator to fix it. That's not an API situation, that's a deployment thing, and it's exactly the regime where Google applied the protocols that you don't seem to like.
That's not to say that the linux fsync behavior is correct here. I think clearing the error having reported it once is kinda broken too. But... fixing that doesn't actually "fix" the real problem.
Really the best suggestion in that thread was up at the top, where it was pointed out that ext3/4 support remounting the filesystem read-only on errors, which sounds like a great policy to me.
- mjw1007 7y agoIt seems pretty clear from the first message in the thread that what the postgres code (and so the "PG folks") was expecting was that fsync would continue to report errors as long as it hadn't succeeded. That way the checkpoint wouldn't complete, and the relevant write-ahead log files wouldn't get deleted (and "checkpoints aren't happening and WAL is piling up" is the sort of thing people already monitor).
- ajross 7y ago> the sort of thing people already monitor Exactly! So it's a devops solution either way. The choice is between one based on system-level monitoring and one based on tool-level error reporting. In neither case does a non-broken fsync() actually fix anyone's problem. My point was that I don't see tools like remount-ro and dmesg monitoring as "workarounds" for an underlying problem. They're correct solutions implemented at the right level of abstraction. They just happen not to be shipped with PostgreSQL, which... I mean, yeah, basically I don't care much either way.
- scottlamb 7y ago> In neither case does a non-broken fsync() actually fix anyone's problem. The system's broken either way, but failure modes matter! With fsync() reporting EIO only once (one time, to one caller), callers that don't expect this behavior (and _no_ callers were written to expect this POSIX-violating, undocumented behavior AFAIK) could corrupt data if writes intermittently succeeded. Eg marking a segment of WAL as fully applied when it wasn't, not only throwing away durability for that transaction but corrupting the whole database. And maybe an error is reported now and then, but that's not so hard to miss. With fsync() consistently reporting EIO, the system never reports a transaction as fully committed then loses its contents, and the existing database isn't corrupted. And someone will notice the problem. mjw1007 worded it perfectly: "'checkpoints aren't happening and WAL is piling up' is the sort of thing people already monitor". So yes, a non-broken fsync() doesn't magically fix your damaged platter, but it's worlds better.
- ajross 7y agoThat's just a restatement of the PG point in the linked thread. To which the reply was "it's better to do this at the system level", thus the rejoinder "but that's a workaround" and my "no, it's a perfectly valid way of solving a problem at the same level of abstraction", and now we're full circle because you're picking on my defense (which I did not make!) of the linux fsync behavior and ignoring all the context. You have to do system level reporting of errors, full stop. You can do that with a fixed fsync. You can do it with dmesg. You can do it even with the current fsync behavior. Pick one. Arguing about fsync is pontificating over the wrong problem. FWIW: I'm all but certain you're wrong about the "POSIX-violating" part. My read of the standard says the author was thinking only about the current request. There's certainly no language in there saying that fsync should return EIO because of a previous failure, nor even what the state of the system (c.f. remount-ro) should be following the first one.
- scottlamb 7y agoYou have to synchronously report the error to the application depending on the fsync, full stop. None of the other proposed mechanisms are a replacement for that; for one, because they are asynchronous. An application can't be written to "do an fsync, assume it's done...unless some system-wide error reporting mechanism says otherwise arbitrarily later". > FWIW: I'm all but certain you're wrong about the "POSIX-violating" part. My read of the standard says the author was thinking only about the current request. An fsync call is not a complete request (it doesn't specify the data to be written), so what on earth does "only about the current request" mean? I can't reconcile any of these arguments that Linux's behavior was correct with the RATIONALE section here: http://pubs.opengroup.org/onlinepubs/9699919799/functions/fsync.html http://pubs.opengroup.org/onlinepubs/9699919799/functions/fs... and particularly the phrase "all data up to the time of the fsync() call is recorded on the disk". There's no mention of "since a previous fsync() call" or any such thing there.