3 ms·

"listen to what close() says" and "don't believe what close() says" are two different things. The article is only (initially) talking about the first, and t

"listen to what close() says" and "don't believe what close() says" are two different things. The article is only (initially) talking about the first, and that is valid. The second is just second-guessing the OS and hardware environment, and is invalid. Here's the rule to figure out if you need fsync() or not: "If you think you might need fsync(), you don't." ;) There are almost no cases where you should worry about the underlying layers. Basically if you aren't writing the filesystem itself, then you shouldn't be calling fsync(). You have to check what open()/close() etc said, but if close() said it worked, then it worked. You're done. The fact that lightning might have struck the drive just exactly then is not your problem and not something you should try to do anything about. Unreliable networks, busses, batteries, etc none of that changes this. Those things all already have their own layers with their own responsibilities to be doing all the necessary testing and verifying and retrying before they return a success code to you. There is open(...,O_SYNC) and mount -o sync (udev rules) for the case of a camera or thumb drive connect by usb etc. It's not merely that you don't have to, it's that it's actively wrong to. fsync() is just joggling someone else's elbow while they're trying to do their job, and if it seems to solve some problem, that actually just exposes that you have some logic or order of operations problem and you aren't doing your own job. "trust the other layer" or "trust the api contract" is unrelated to and does not conflict with "Be forgiving in your inputs and strict in your outputs.". It just means: Do: Check the returned error from, say, malloc(). Don't: Get a success from malloc() and then go try to do things to prove that malloc() actually did work. That would be insane and impossible because it would have to apply equally to everything, including every single keyword or function you would use as part of the verification. How do you know when you so much as set a value to variable that it actually got set? If you printf the variable to prove it, how do you know printf didn't lie? The logic for trusting close() is no different from the logic for trusting malloc(). Our responsibility is just to stay within the bounds of defined behavior and not make any assumptions about anything that isn't promised.

lxgr 2y ago
> The fact that lightning might have struck the drive just exactly then is not your problem [...] If you think you might need fsync(), you don't. [...] Those things all already have their own layers with their own responsibilities to be doing all the necessary testing [...] You're mixing up the concepts of durability and consistency in pretty significant ways here, and are implying that everybody that is fine with a lack of the former will also be fine with a lack of the latter. This is absolutely not true, and can cause extremely painful and hard-to-track-down bugs. For better or worse, there is no way to directly tell the OS "whatever you do, make sure you don't reorder these writes I just did with those other writes I'm about to do". The next best (portable) thing we have to achieve that outcome is fsync. It's a bit heavy handed, in that it gives you durability even if you only want consistency. That absolutely doesn't mean it's redundant, though. > Basically if you aren't writing the filesystem itself, then you shouldn't be calling fsync(). Given that fsync is a syscall, but file systems are generally implemented in the kernel, this is a pretty nonsensical statement by itself. File systems usually have (and need, for performance) a much lower level view of the underlying block storage, and fine-grained control over it. Just as one example, Linux has the concept of write barriers. Not using these correctly (in the filesystem driver) can cause data leaks across files owned by different users and processes.
- Brian_K_White 2y ago
  "there is no way to directly tell the OS "whatever you do, make sure you don't reorder these writes I just did with those other writes I'm about to do" Perhaps because there is no reason for such a thing to exist. Tell me an example.
  - lxgr 2y ago
    Text editors: I want my file content to either be what it was before I pressed "save", or what it was afterwards; I don't want my file to be mangled halfway through. (In other words, in this case I need consistency, but not necessarily durability.) How would you do that without fsync? Databases: People generally don't like unrecoverable consistency errors just because their computer crashed during a write. Not generally possible with reordered writes. Sometimes people also need durability on top of consistency, e.g. for everything where you want to make at most one request to some server; you can do that by e.g. writing "I did the thing" to a log file, fsync'ing it, and then making your request.

lxgr 2y ago

> The fact that lightning might have struck the drive just exactly then is not your problem [...] If you think you might need fsync(), you don't. [...] Those things all already have their own layers with their own responsibilities to be doing all the necessary testing [...] You're mixing up the concepts of durability and consistency in pretty significant ways here, and are implying that everybody that is fine with a lack of the former will also be fine with a lack of the latter. This is absolutely not true, and can cause extremely painful and hard-to-track-down bugs. For better or worse, there is no way to directly tell the OS "whatever you do, make sure you don't reorder these writes I just did with those other writes I'm about to do". The next best (portable) thing we have to achieve that outcome is fsync. It's a bit heavy handed, in that it gives you durability even if you only want consistency. That absolutely doesn't mean it's redundant, though. > Basically if you aren't writing the filesystem itself, then you shouldn't be calling fsync(). Given that fsync is a syscall, but file systems are generally implemented in the kernel, this is a pretty nonsensical statement by itself. File systems usually have (and need, for performance) a much lower level view of the underlying block storage, and fine-grained control over it. Just as one example, Linux has the concept of write barriers. Not using these correctly (in the filesystem driver) can cause data leaks across files owned by different users and processes.

Brian_K_White 2y ago
"there is no way to directly tell the OS "whatever you do, make sure you don't reorder these writes I just did with those other writes I'm about to do" Perhaps because there is no reason for such a thing to exist. Tell me an example.
- lxgr 2y ago
  Text editors: I want my file content to either be what it was before I pressed "save", or what it was afterwards; I don't want my file to be mangled halfway through. (In other words, in this case I need consistency, but not necessarily durability.) How would you do that without fsync? Databases: People generally don't like unrecoverable consistency errors just because their computer crashed during a write. Not generally possible with reordered writes. Sometimes people also need durability on top of consistency, e.g. for everything where you want to make at most one request to some server; you can do that by e.g. writing "I did the thing" to a log file, fsync'ing it, and then making your request.