3 ms·
This is a great article! But fsyncgate makes me cry inside. It sounds like PostgreSQL made the decision to crash instead of retry forever upon fsync errors (wh
by zackmorris 3y ago
This is a great article!
But fsyncgate makes me cry inside. It sounds like PostgreSQL made the decision to crash instead of retry forever upon fsync errors (which is the wrong way to do it). But it wasn't their fault, because OSs and filesystems don't always perform correctly when they run out of disk space. And they were able to avoid data loss by continuing from the last point in the log before the crash (and maybe have a supervisor process relaunch the database) so in the end, it may not matter.
The 2010s were a strange time. People went to great lengths to work around perceived limitations in the tooling. Stuff like, MongoDB would sometimes lose data, even when it claimed that a write completed. And people were ok with that, because they wanted the speed and freedom in order to iterate quickly and make money.
When we really should have been formally addressing problems from first principles. Because if our OS can't recover gracefully when the disk fills up, then we don't really have anything. And so many OSs (including our desktops) panic when the disk fills up, that we don't really have anything. At a low enough level, all of the tools we rely upon are trash.
But the point I wanted to make is: there's no such thing as exceptional behavior, there's only how many edge cases have been handled. Programs can be converted from imperative to functional by replacing exceptions with blocking (or spin-locking) retries so that their logic eventually finishes executing. In this case, by letting programs wait until the user has cleared off disk space, rather than forcing the user to orchestrate relaunching processes. In other words, synchronous blocking logic can be converted to a spreadsheet (flowchart) and reasoned about. But asynchronous nonblocking logic allows state explosion by forcing the user to deal with meta eventualities outside the core business logic.
So it's understandable how PostgreSQL made the decision to go with a nonblocking fault instead of blocking in 2018. But in 2023, a better approach might be to write something like a container with proper fsync behavior to protect PostgreSQL from immature filesystems.
- marcosdumay 3y agoI really envy your optimism, but no, filesystems at 2023 are fucked too, both on the host and the guest. Disks are still fucked in that they will tell you they saved your data, and still lose it due to some failure, or when they are full. The disk communication bus is a bit less fucked though, so that's improvement. But anyway, you still can't rely on good behavior from your fsync, or even that a successful fsync means your data is safe.
- deleted 3y ago[deleted]
- ElectricalUnion 3y ago> When we really should have been formally addressing problems from first principles. Because if our OS can't recover gracefully when the disk fills up, then we don't really have anything. And so many OSs (including our desktops) panic when the disk fills up, that we don't really have anything. At a low enough level, all of the tools we rely upon are trash. Unfortunately, the OS can't really "recover" most of the time because it is being constantly lied to by several(!!!) layers of uncooperative hardware/firmware. It's zebras all the way down: https://www.youtube.com/watch?v=fE2KDzZaxvE https://www.youtube.com/watch?v=fE2KDzZaxvE > In this case, by letting programs wait until the user has cleared off disk space If your system is really in a no-disk-space-left state you might not be able to login to fix it: https://unix.stackexchange.com/questions/165427/cant-login-due-disk-space-exhaustion https://unix.stackexchange.com/questions/165427/cant-login-d... And even assuming you can login, sometimes you can't delete files either; you need to free space in another way before being able to delete files. https://serverfault.com/questions/478733/rm-can-not-remove-xxx-no-space-left-on-device-on-btrfs https://serverfault.com/questions/478733/rm-can-not-remove-x... Filesystems and persistence are hard.
- charcircuit 3y ago>there's no such thing as exceptional behavior, there's only how many edge cases have been handled. Some edge cases can be hard to handle. What if the processor takes the wrong branch? What if one or more bit flips happens? At some point it is just better to crash than try to and actually handle such an edge case.