5 ms·
If fsync() fails isn't valhalla lost anyhow? If you can't write things down because your pencil is broken, probably time to stop what you're doing and get a ne
by a-dub 7y ago
If fsync() fails isn't valhalla lost anyhow? If you can't write things down because your pencil is broken, probably time to stop what you're doing and get a new pencil.
If the kernel can't flush dirty write buffers, maybe it's time to send up a flag and panic in the kernel itself?
- tlb 7y agoThat can happen when you remove a removable storage device, like a USB drive. You don't want to panic the kernel for that.
- Gibbon1 7y agoReminds me. On CPM, hit save when the drive door is open, so sorry. All gone.
- wruza 7y agoFreeBSD lived with that up to circa 2010, afair. Few fast-pulling desktop linux trolls complained at the time, but nothing serious. Anyway, if given such choice, as a user/admin I would prefer an instant fail with a panic message, rather than a corrupted 10gb online client database with no clear recovery path. I’m always scared by this kind of story when something fails to fail underneath and I even didn’t ever hear of it.
- Dylan16807 7y agoIsn't your data already corrupted most of the time when writes start to fail? If the drive is gone, a panic won't help or hurt, it will just ruin everything else you have open. Redoing failed writes should be the method that ends in the least amount of corruption.
- wruza 7y agoNot going into details, at the end of the day one should have a plan to recover their value from a potential failure. By the means of version control systems, transaction logs, backups, mirrors, etc. But if a failure makes its way to a recovery storage undetected, the entire plan is doomed. I don’t care if I lose few last hours of my local work, since this time is buffered in a project or is not precious at all (like game saves). That throws back me a little, but not anyone else. I also don’t really care about losing downloads or experimental files. But when it comes to a business database or a project repository, that value can be orders of magnitude bigger and can cost responsible one their job at the very least. “Redoing failed writes should be the method that ends in the least amount of corruption.” But in fact it led to a database corruption that was only fixed by some black magic. As far as I understood, they did manual rollback of failed writes. I wouldn’t blame anyone for subj bug even if met it by myself — writing low-level software is hard, the behavior was unintended, and both linux, xfs & pg are free as in beer and free of any charges. But if it happened and was not detected for longer than a recovery window allows it, costs were imminent. That’s why this trade-off is easy on a distance.
- Dylan16807 7y ago> But in fact it led to a database corruption You're saying you had a system that did redo failed writes to disk, unlike other systems discussed so far, and it managed to corrupt your database anyway?
- jcranmer 7y ago> If the kernel can't flush dirty write buffers, maybe it's time to send up a flag and panic in the kernel itself? Being unable to write to a disk is a recoverable scenario, especially in some conditions. The most common cause of disk write failures is "someone yanked the flash drive out of the computer," and the recovery is "pop up a dialog telling the user to put it back in."
- carbocation 7y agoAnother situation: a GCSFuse volume dealing with temporary networking errors.
- masklinn 7y agoOr NFS being NFS.
- londons_explore 7y agoI think there is a case for never throwing away dirty pages. Ie. either the write eventually succeeds, or all other writes will block or fail until it does.
- tinus_hn 7y agoThat would be a great improvement over the ‘tell the user he shouldn’t have done that’ we have now!
- a-dub 7y agoif you've got removable disks on your servers, valhalla was lost long ago!
- a-dub 7y agommm, maybe this is a little too aggressive there could be perfectly functional storage on the host. maybe something like "make the mount read only and ensure all open fds on the device return nothing but error and then dump the dirty blocks to swap in some kind of recovery log that could potentially be applied to the disk"
- rodgerd 7y agoSo kernel panic on disk full errors? Because that's one of the scenarios causing data loss here - fsync throwing away data on a full disk and lying about having committed it.
- limitthis 7y agono, the fs doesn’t overcommit storage like the memory system does.
- rodgerd 7y agoI don't think you understand the problem.
- masklinn 7y agoIn that thread, Craig Ringer and Thomas Munro notes most systems & FS won't report ENOSPC on fsync(), they'll do that on write(): > ENOSPC doesn't seem to be a concern during normal operation of major file systems (ext3, ext4, btrfs, xfs) because they reserve space before returning from write(). But if a buffered write does manage to fail due to ENOSPC we'll definitely see the same problems. This makes ENOSPC on NFS a potentially data corrupting condition since NFS doesn't preallocate space before returning from write().
- Upvoter33 7y agono, some errors are transient. a retry is a reasonable thing to be able to do in such a case. this happens with storage on SANs for example, due to a network blip.