6 ms·

> In neither case does a non-broken fsync() actually fix anyone's problem. The system's broken either way, but failure modes matter! With fsync() reporting EI

> In neither case does a non-broken fsync() actually fix anyone's problem. The system's broken either way, but failure modes matter! With fsync() reporting EIO only once (one time, to one caller), callers that don't expect this behavior (and _no_ callers were written to expect this POSIX-violating, undocumented behavior AFAIK) could corrupt data if writes intermittently succeeded. Eg marking a segment of WAL as fully applied when it wasn't, not only throwing away durability for that transaction but corrupting the whole database. And maybe an error is reported now and then, but that's not so hard to miss. With fsync() consistently reporting EIO, the system never reports a transaction as fully committed then loses its contents, and the existing database isn't corrupted. And someone will notice the problem. mjw1007 worded it perfectly: "'checkpoints aren't happening and WAL is piling up' is the sort of thing people already monitor". So yes, a non-broken fsync() doesn't magically fix your damaged platter, but it's worlds better.

ajross 7y ago
That's just a restatement of the PG point in the linked thread. To which the reply was "it's better to do this at the system level", thus the rejoinder "but that's a workaround" and my "no, it's a perfectly valid way of solving a problem at the same level of abstraction", and now we're full circle because you're picking on my defense (which I did not make!) of the linux fsync behavior and ignoring all the context. You have to do system level reporting of errors, full stop. You can do that with a fixed fsync. You can do it with dmesg. You can do it even with the current fsync behavior. Pick one. Arguing about fsync is pontificating over the wrong problem. FWIW: I'm all but certain you're wrong about the "POSIX-violating" part. My read of the standard says the author was thinking only about the current request. There's certainly no language in there saying that fsync should return EIO because of a previous failure, nor even what the state of the system (c.f. remount-ro) should be following the first one.
- scottlamb 7y ago
  You have to synchronously report the error to the application depending on the fsync, full stop. None of the other proposed mechanisms are a replacement for that; for one, because they are asynchronous. An application can't be written to "do an fsync, assume it's done...unless some system-wide error reporting mechanism says otherwise arbitrarily later". > FWIW: I'm all but certain you're wrong about the "POSIX-violating" part. My read of the standard says the author was thinking only about the current request. An fsync call is not a complete request (it doesn't specify the data to be written), so what on earth does "only about the current request" mean? I can't reconcile any of these arguments that Linux's behavior was correct with the RATIONALE section here: http://pubs.opengroup.org/onlinepubs/9699919799/functions/fsync.html http://pubs.opengroup.org/onlinepubs/9699919799/functions/fs... and particularly the phrase "all data up to the time of the fsync() call is recorded on the disk". There's no mention of "since a previous fsync() call" or any such thing there.
  - ajross 7y ago
    > You have to synchronously report the error to the application depending on the fsync, full stop. Why? As reported above, Google itself isn't even doing that. We're not talking about a network timeout here. Hardware failures are hardware failures and not in the general case recoverable by software no matter how strongly held the opinions are about how that software should work. The fsync behavior on Linux was surprising and wrong. But fixing that won't fix any of the stuff you're yelling about. API hygiene can't fix machines, and the machine broke.
    - scottlamb 7y ago
      > Why? As reported above, Google itself isn't even doing that. I work at "Google itself". Typically there's a single process responsible for all the disk writes of significance on the machine, possibly bypassing the in-kernel filesystem entirely in favor of direct block device access. That process's caller waits until three machines (each with its own battery backup, and frequently on a separate PDU from the others) have acked the write before proceeding. Also, that process's caller is Spanner, and those three machines are necessary to say a single replica has accepted the write. Typically there are three replicas in different clusters (often in separate cities) and two of them have to accept the write (in Paxos terminology) before Spanner tells the application it's complete. Please don't take Google's production environment working successfully most of the time from the customer viewpoint as proof of anything about the Linux virtual filesystem layer or ext4. You can paper over a lot of brokenness when you have six independent battery-backed copies of the data in two cities before you depend on it. > We're not talking about a network timeout here. Hardware failures are hardware failures and not in the general case recoverable by software no matter how strongly held the opinions are about how that software should work. Software can't fix the broken hardware, and (without significant redundancy and cross-verification as Google does) can't correct for hardware simply lying. But correctly written software can absolutely change the likely failure modes when hardware reports error. And intermittent hardware failures happen a lot more than you might think anyway. Bugs that turn intermittent failure into permanent corruption are nasty.
- macdice 7y ago
  What about this: you write() some data, and then Linux throws it away because of a writeback error, and later you read() it, and Linux gives you an older version from before your write (because this error condition has somehow been fixed and now it reads that block back into the page cache). I'm pretty sure that violates POSIX, which says: "The read() function reads data previously written to a file."
  - ajross 7y ago
    I don't think that's the behavior, though. The blocks don't get flushed, they're still in cache and get read back. Probably. Again, the system broke. You've got no guarantees about anything. Fixing an API isn't going to change that.
    - Dylan16807 7y ago
      API guarantees are still important even when a peripheral breaks. It's the difference between knowing how and where performance is degraded and what to work around, versus utter chaos.
      - macdice 7y ago
        Totally agree. Speaking as a database hacker, just tell me it broke. I will deal with the rest.
        ajross 7y ago
        It did tell you it broke, though. It just didn't repeat the message when you asked again. Really, there is no "correct" behavior from getting an -EIO from a fsync beyond "shut it all down, notify everyone you can and hope for the best". This retry behavior was itself a bad design choice (and again, I'm not defending fsync here). And again, speaking as a systems hacker: it broke. You can't "deal with" broke. And the system absolutely did tell you it broke. I genuinely don't understand the insistence by folks on this list that OMG THE ERROR MUST BE HANDLED IN THE DATABASE SOFTWARE. It seems kinda ridiculous to me, honestly.

ajross 7y ago

That's just a restatement of the PG point in the linked thread. To which the reply was "it's better to do this at the system level", thus the rejoinder "but that's a workaround" and my "no, it's a perfectly valid way of solving a problem at the same level of abstraction", and now we're full circle because you're picking on my defense (which I did not make!) of the linux fsync behavior and ignoring all the context. You have to do system level reporting of errors, full stop. You can do that with a fixed fsync. You can do it with dmesg. You can do it even with the current fsync behavior. Pick one. Arguing about fsync is pontificating over the wrong problem. FWIW: I'm all but certain you're wrong about the "POSIX-violating" part. My read of the standard says the author was thinking only about the current request. There's certainly no language in there saying that fsync should return EIO because of a previous failure, nor even what the state of the system (c.f. remount-ro) should be following the first one.

scottlamb 7y ago
You have to synchronously report the error to the application depending on the fsync, full stop. None of the other proposed mechanisms are a replacement for that; for one, because they are asynchronous. An application can't be written to "do an fsync, assume it's done...unless some system-wide error reporting mechanism says otherwise arbitrarily later". > FWIW: I'm all but certain you're wrong about the "POSIX-violating" part. My read of the standard says the author was thinking only about the current request. An fsync call is not a complete request (it doesn't specify the data to be written), so what on earth does "only about the current request" mean? I can't reconcile any of these arguments that Linux's behavior was correct with the RATIONALE section here: http://pubs.opengroup.org/onlinepubs/9699919799/functions/fsync.html http://pubs.opengroup.org/onlinepubs/9699919799/functions/fs... and particularly the phrase "all data up to the time of the fsync() call is recorded on the disk". There's no mention of "since a previous fsync() call" or any such thing there.
- ajross 7y ago
  > You have to synchronously report the error to the application depending on the fsync, full stop. Why? As reported above, Google itself isn't even doing that. We're not talking about a network timeout here. Hardware failures are hardware failures and not in the general case recoverable by software no matter how strongly held the opinions are about how that software should work. The fsync behavior on Linux was surprising and wrong. But fixing that won't fix any of the stuff you're yelling about. API hygiene can't fix machines, and the machine broke.
  - scottlamb 7y ago
    > Why? As reported above, Google itself isn't even doing that. I work at "Google itself". Typically there's a single process responsible for all the disk writes of significance on the machine, possibly bypassing the in-kernel filesystem entirely in favor of direct block device access. That process's caller waits until three machines (each with its own battery backup, and frequently on a separate PDU from the others) have acked the write before proceeding. Also, that process's caller is Spanner, and those three machines are necessary to say a single replica has accepted the write. Typically there are three replicas in different clusters (often in separate cities) and two of them have to accept the write (in Paxos terminology) before Spanner tells the application it's complete. Please don't take Google's production environment working successfully most of the time from the customer viewpoint as proof of anything about the Linux virtual filesystem layer or ext4. You can paper over a lot of brokenness when you have six independent battery-backed copies of the data in two cities before you depend on it. > We're not talking about a network timeout here. Hardware failures are hardware failures and not in the general case recoverable by software no matter how strongly held the opinions are about how that software should work. Software can't fix the broken hardware, and (without significant redundancy and cross-verification as Google does) can't correct for hardware simply lying. But correctly written software can absolutely change the likely failure modes when hardware reports error. And intermittent hardware failures happen a lot more than you might think anyway. Bugs that turn intermittent failure into permanent corruption are nasty.
macdice 7y ago
What about this: you write() some data, and then Linux throws it away because of a writeback error, and later you read() it, and Linux gives you an older version from before your write (because this error condition has somehow been fixed and now it reads that block back into the page cache). I'm pretty sure that violates POSIX, which says: "The read() function reads data previously written to a file."
- ajross 7y ago
  I don't think that's the behavior, though. The blocks don't get flushed, they're still in cache and get read back. Probably. Again, the system broke. You've got no guarantees about anything. Fixing an API isn't going to change that.
  - Dylan16807 7y ago
    API guarantees are still important even when a peripheral breaks. It's the difference between knowing how and where performance is degraded and what to work around, versus utter chaos.
    - macdice 7y ago
      Totally agree. Speaking as a database hacker, just tell me it broke. I will deal with the rest.
      - ajross 7y ago
        It did tell you it broke, though. It just didn't repeat the message when you asked again. Really, there is no "correct" behavior from getting an -EIO from a fsync beyond "shut it all down, notify everyone you can and hope for the best". This retry behavior was itself a bad design choice (and again, I'm not defending fsync here). And again, speaking as a systems hacker: it broke. You can't "deal with" broke. And the system absolutely did tell you it broke. I genuinely don't understand the insistence by folks on this list that OMG THE ERROR MUST BE HANDLED IN THE DATABASE SOFTWARE. It seems kinda ridiculous to me, honestly.