4 ms·

Which of course isn't really a "solution", as it results in ambiguous error semantics; this is especially the case with Go's defer, as it pushes the code to the

by saurik 2y ago

Which of course isn't really a "solution", as it results in ambiguous error semantics; this is especially the case with Go's defer, as it pushes the code to the end of the entire function, not merely some intermediate relevant scope... the result is that a file used near the beginning of a function might fail to close but that error will not be realized until after something else in the function fails, after the point of no return on the close. Really, this is kind of a fundamental limitation of automatic allocation semantics, which is effectively a monad that is being stacked with the error propagation monad in a confusing manner that means you "should" only defer operations which don't have errors. Even in languages with exceptions, such as C++ and Java, they had to wrangle with this problem and failed to solve it: in C++ 11 or 17 or whatever, deconstructors are now by default nothrow in an attempt to prevent this kind of mistake. ...but then, what does one do with close?! In some sense, the entire concept of close must not fail, and yet it exists in a world where we don't really believe in anything that can't fail, as we like moving around failure semantics. FWIW, Linus has suggested that the kernel should largely accept that application developers don't ever check the return value of close... but has also stated that developers should sync the file first if they care and also check close (at least, to be maximally correct). But like, what does one do then? How do you ever recover from this? I actually think you can't, if you are in a cleanup operation... not without breaking your error regime. I think--and this is also where the article eventually goes in the updates (though without noting you can add your own boolean)--you should therefore both explicitly close (and/or maybe flush/sync) the file after writing to it and have a "if I didn't close this, close it" in your cleanup handler; critically, the (explicit) former checks for errors, while the (implicit) latter doesn't.

d0mine 2y ago
The solution is that there are multiple solutions that are suitable in different situations. If you need "strict" requirements then it is likely impossible but you can get close enough if you use sqlite. Or more lightweight atomic/thread-safe/fault-tolerant file library. You could rollout your own: it easy to start, and continue until the error rate is tolerable for your application (though it may take more dev time). If you don't need db-like strict guarantees. Just write your app knowing that it may fail (data may be lost, corrupted). It may be ok in a lot of cases.
formerly_proven 2y ago
> FWIW, Linus has suggested that the kernel should largely accept that application developers don't ever check the return value of close... but has also stated that developers should sync the file first if they care and also check close (at least, to be maximally correct). Close just behaves very differently depending on the actual filesystem, too. Usually it’s very fast because it doesn’t do much of anything, but e.g. on NFS close will actually wait for writeback to the server to complete (due to close-to-open semantics).
everforward 2y ago
I think some of those issues with errors could be worked around. Wrapping errors provides a way to return a "cleanup failed" error that also includes the root cause of being a failed write. Likewise, there are packages for handling multi-errors. I think the reality is that most of us push anything where the return status of Close would be important into the database, specifically because it handles semantics like this and simultaneous writes for us. It's like half the selling point of SQLite; you could write JSON documents and handle all the edge cases yourself, or just jam it in SQLite and quite worrying about Close and simultaneous writes and all that junk.
- saurik 2y ago
  I mean, we can cause the same kind of problem there, as people might try to write a scope exit handler to commit a transaction. You'd then run into the same issue, and so "commit transaction" isn't a thing which should ever be in such a construct. Of course, deleting the objects for the transaction / closing the database connection / etc. would be fine to ignore errors from (and hopefully wouldn't/shouldn't fail anyway) and so those can and should be automated: you defer close but manually call and check a commit. Once you accept this reality, the file case is the same: putting the sync and/or first critical close inline is equivalent work. The issue is that you simply can't -- no matter what the mechanism is -- slip back and forth between your scope maintenance and your error handling monads, resulting in needing cleanup operations where failure is not an option; and, so, you either must not care about the error in the context of the call or must do something even more drastic like terminate the entire program for violating semantics. FWIW, I do appreciate that people are less likely to make that kind of mistake when working with a database, as people largely get that you should even try to commit a transaction if the code in it had failed somehow. Additionally, I appreciate that if you have a very tight scope -- which Go makes hard, but can still be pulled off -- the "close and throw an error if and only if we don't have an error right now" strategy is not at all horrible... it just isn't a "solution" to the underlying issue without an understanding of why. Put elsewise, I think it is useful to appreciate that there is more of a universal theoretical / math reason why this is awkward and why it kind of needs to be built in a specific way, and that this issue transcends the syntax or even the implementation details of how you are trying to manage errors: at the end of the end of the day, all of these techniques people discuss are in some sense equivalent, and, at best, most of these workarounds at offer are ways to incorrectly model the problem due to some systems giving you too much rope.

by saurik 2y ago

d0mine 2y ago

The solution is that there are multiple solutions that are suitable in different situations. If you need "strict" requirements then it is likely impossible but you can get close enough if you use sqlite. Or more lightweight atomic/thread-safe/fault-tolerant file library. You could rollout your own: it easy to start, and continue until the error rate is tolerable for your application (though it may take more dev time). If you don't need db-like strict guarantees. Just write your app knowing that it may fail (data may be lost, corrupted). It may be ok in a lot of cases.

formerly_proven 2y ago

> FWIW, Linus has suggested that the kernel should largely accept that application developers don't ever check the return value of close... but has also stated that developers should sync the file first if they care and also check close (at least, to be maximally correct). Close just behaves very differently depending on the actual filesystem, too. Usually it’s very fast because it doesn’t do much of anything, but e.g. on NFS close will actually wait for writeback to the server to complete (due to close-to-open semantics).

everforward 2y ago

I think some of those issues with errors could be worked around. Wrapping errors provides a way to return a "cleanup failed" error that also includes the root cause of being a failed write. Likewise, there are packages for handling multi-errors. I think the reality is that most of us push anything where the return status of Close would be important into the database, specifically because it handles semantics like this and simultaneous writes for us. It's like half the selling point of SQLite; you could write JSON documents and handle all the edge cases yourself, or just jam it in SQLite and quite worrying about Close and simultaneous writes and all that junk.

saurik 2y ago
I mean, we can cause the same kind of problem there, as people might try to write a scope exit handler to commit a transaction. You'd then run into the same issue, and so "commit transaction" isn't a thing which should ever be in such a construct. Of course, deleting the objects for the transaction / closing the database connection / etc. would be fine to ignore errors from (and hopefully wouldn't/shouldn't fail anyway) and so those can and should be automated: you defer close but manually call and check a commit. Once you accept this reality, the file case is the same: putting the sync and/or first critical close inline is equivalent work. The issue is that you simply can't -- no matter what the mechanism is -- slip back and forth between your scope maintenance and your error handling monads, resulting in needing cleanup operations where failure is not an option; and, so, you either must not care about the error in the context of the call or must do something even more drastic like terminate the entire program for violating semantics. FWIW, I do appreciate that people are less likely to make that kind of mistake when working with a database, as people largely get that you should even try to commit a transaction if the code in it had failed somehow. Additionally, I appreciate that if you have a very tight scope -- which Go makes hard, but can still be pulled off -- the "close and throw an error if and only if we don't have an error right now" strategy is not at all horrible... it just isn't a "solution" to the underlying issue without an understanding of why. Put elsewise, I think it is useful to appreciate that there is more of a universal theoretical / math reason why this is awkward and why it kind of needs to be built in a specific way, and that this issue transcends the syntax or even the implementation details of how you are trying to manage errors: at the end of the end of the day, all of these techniques people discuss are in some sense equivalent, and, at best, most of these workarounds at offer are ways to incorrectly model the problem due to some systems giving you too much rope.