5 ms·
Yeah, IMO it's a Linux fsync bug. fsync() should not succeed if writes failed. fsync() should not clean dirty pages if writes failed. Both of these behaviors
by loeg 8y ago
Yeah, IMO it's a Linux fsync bug. fsync() should not succeed if writes failed. fsync() should not clean dirty pages if writes failed. Both of these behaviors directly contravene the goals of user applications invoking fsync() as well as any reasonable API contract for safely persisting data.
Arguably, POSIX needs a more explicit fsync interface. I.e., sync these ranges; all dirty data, or just dirty data since last checkpoint; how should write errors be handled; etc. That doesn't excuse that Linux's fsync is totally broken and designed to eat data in the face of hardware errors.
That Dan Luu blog post you linked is fantastic and one I really enjoyed.
- macdice 8y agoI worked on PostgreSQL's investigation and response to this stuff and I agree with you FWIW. But apparently only FreeBSD (any filesystem) and ZFS (any OS) agree with us. Other systems I looked at throw away buffers and/or mark them clean, and this goes all the way back to ancient AT&T code. Though no one has commented on any closed kernel's behaviour. I doubt anyone worried much when disks just died completely in the good old days. This topic is suddenly more interesting now as virtualisation and network storage create more kinds of transient failures, I guess.
- IgorPartola 8y agoYet another reason to use ZFS I guess.
- loeg 8y agoI'm a FreeBSD dev, that probably colors my opinion :-).
- pgaddict 8y agoThere have been bugs involved on both sides. On the kernel side the error reporting was not working reliably in some cases (see [1] for details), so the application using fsync may not actually get the error at all. Hard to handle an error correctly when you don't even get notified about it. [1] https://www.youtube.com/watch?v=74c19hwY2oE https://www.youtube.com/watch?v=74c19hwY2oE On the PostgreSQL side, it was the incorrect assumption that fsync retries past writes. It's an understandable mistake, because without the retry it's damn difficult to write an application using fsync correctly. And of course we've found a bunch of other bugs in the ancient error-handling code (which just confirms the common wisdom that error-handling is the least tested part of any code base).
- loeg 8y agoRight; I've been following the headlines on this saga for years :-). > On the PostgreSQL side, it was the incorrect assumption that fsync retries past writes. That's the part I'm claiming is a Linux bug. Marking failed dirty writes as clean is self-induced data loss. > It's an understandable mistake, because without the retry it's damn difficult to write an application using fsync correctly. This is part of why it's a bug. Making it even more difficult for user applications to correctly reason about data integrity is not a great design choice (IMO). > And of course we've found a bunch of other bugs in the ancient error-handling code (which just confirms the common wisdom that error-handling is the least tested part of any code base). +100
- pgaddict 8y agoI'm not sure. It's easy to blame other layers for not behaving the way you want/expect, but I admit there are valid reasons why it behaves the way it does and not the way you imagine. The PostgreSQL fsync thread started with exactly "kernel is broken" but that opinion changed over time, I think. I still think it's damn difficult (border-line impossible) to use fsync correctly in anything but trivial applications, but well ... Amusingly, Linux has sync_file_range() which supposedly does one of the things you describe (syncing file range), but if you look at the man page it says "Warning: ... explanation why it's unsafe in many cases ...".
- blattimwind 8y agoUp until very recently (2-3 years ago) there was very little discussion of how to do consistent disk I/O properly, so it doesn't surprise me at all that many applications don't actually work. Combine this with I/O error handling often being broken (in kernels, file systems, applications) and applications that are supposed to implement transaction-semantics can easily turn into "he's dead jim" at the first issue.
- loeg 8y ago> It's easy to blame other layers for not behaving the way you want/expect, Sure; to be clear, I work on both sides of the this layer boundary (kernel side as well as on userspace applications trying to ensure data is persisted) on a FreeBSD-based appliance at $DAYJOB, but mostly on the kernel side. I'm saying — from the perspective of someone who works on kernel and filesystem internals and page/buffer cache — cleaning dirty data without successful write to media is intentional data loss. Not propagating IO errors to userspace makes it more difficult for userspace to reason about data loss. > but I admit there are valid reasons why it behaves the way it does and not the way you imagine. How do you imagine I imagine the Linux kernel's behavior here? > The PostgreSQL fsync thread started with exactly "kernel is broken" but that opinion changed over time, I think. I still think it's damn difficult (border-line impossible) to use fsync correctly in anything but trivial applications, but well ... The second sentence is a good argument for the Linux kernel's behavior being broken.
- 8y ago
- adaszko 8y agoTomas Vondra goes over this in the talk. There's a rationale for that behavior: pulling out a USB stick out of the USB socket may have been what's triggered the fsync() failure. In that case, there's no way the kernel will be able to retry reliably.
- pas 8y agoSilent error ignoring is never a great API. Especially for a data integrity operation. fsync in case of the USB disappearing should simply return an error and drop the dirty pages.