4 ms·

> So yes, absolutely temporary and cache files should be checksummed. If that data is corrupted, it will cause crashes of the software using them and downstream

by derefr 4y ago

> So yes, absolutely temporary and cache files should be checksummed. If that data is corrupted, it will cause crashes of the software using them and downstream corruption after all. ...no? I don't care if a video in my browser's cache ends up with a few corrupt blocks when I play it again a year later. Video codecs are designed to be tolerant of that. You'll get a glitchy section in a few frames, and then hit the next keyframe and everything will clean up. In fact, most encodings — of images, audio, even text — are designed to be self-synchronizing in the face of corruption. I think you're thinking specifically of working-state files, which usually need to be perfect and guaranteed-trusted, because they're in normalized low-redundancy forms and are also used to derive other data from. But when I say "caching", I'm talking about cached final-form assets intended for direct human consumption. These get corrupted all the time, from network errors during download, disk storage errors on NASes, etc; and people mostly just don't care. For video, they just watch past it. For a web page, they hard-refresh it and everything's fine the second time around. If you think it's impossible to differentiate these two cases: well, that's because we don't explicitly ask developers to differentiate them. There could be separate ~/Library/ViewCache and ~/Library/StateCache directories. And before you ask, a good example of a large "ViewCache" asset that's not browser-related: a video-editor render-preview video file (the low-quality / thumbnail-sized kind, used for scrubbing.)

lazide 4y ago
If they are corrupted on disk the behavior is not so deterministic as a ‘broken image’ and a reload. Corrupted on disk content causes software crashes, hangs, and other broken behavior users definitely don’t like. Especially when it’s the filesystem metadata which gets corrupted. Because merely trying to read it can cause severe issues at the filesystem level. I take it you haven’t dealt with failing storage much before?
- derefr 4y ago
  I maintain database and object-storage clusters for a living. Dealing with failing storage is half my job. > Especially when it’s the filesystem metadata which gets corrupted. We're not talking about filesystem metadata, though. Filesystem metadata is all "of a piece" — if you have a checksumming filesystem, then you can't not checksum some of the filesystem metadata, because all the metadata lives in (the moral equivalent of) a single database file the filesystem maintains, and that database gets checksummed. It's all one data structure, where the checksumming is a thing you do to that data structure, not to individual nodes within it. (For a tree filesystem like btrfs, this would be the non-cryptographic equivalent of a merkle-tree hash.) The only way you could even potentially turn off filesystem features for some metadata (dirent, freelist, etc) nodes but not others, would be to split your filesystem into multiple filesystems. No, to be clear, we're specifically talking about what happens inside the filesystem's extents. Those can experience corruption without that causing any undue issues, besides "the data you get from fread(3) is wrong." Unlike filesystem metadata, which is all required for the filesystem's integrity, a checksumming filesystem can choose whether to "look" inside file extents, or to treat them as opaque. And it can (in theory) make that choice per file, if it likes. From the FS's perspective, an extent is just a range of reserved disk blocks. Now, an assumption: only storage arrays use spinning rust for anything any more. The only disk problems consumer devices face any more are SSD degradation problems, not HDD degradation problems. (Even if you don't agree with this assumption by itself, it's much more clear-cut if you consider only devices operated by people willing to choose to use a filesystem that's not the default one for their OS.) This assumption neatly cleaves the problem-space in two: - How should a filesystem on a RAID array, set up for a business or prosumer use-case, deal with HDD faults? - How should a single-device filesystem used in a consumer use-case deal with SDD faults? The HDD-faults case comes down to: filesystem-level storage pool management with filesystem-driven redundant reads, with kernel blocking-read timeouts to avoid hangs, with async bad-sector remapping for timed out reads. Y'know: ZFS. While the SDD-faults case comes down to: read the bad data. Deal with the bad data. You won't get any hangs, until the day the whole thing just stops working. The worst you'll get is bit-rot. And even then, it's rare, because NAND controllers use internal space for error-correction, entirely invisibly to the kernel. (See also: http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/) In fact, in my own personal experience, the most likely cause of incorrect or corrupt data ending up on an SSD/NVMe disk, is that the CPU or memory of the system is bad, and so one or the other is corrupting the memory that will be written to disk before or during the write. (I've personally had this happen at least twice. What to look for to diagnose this: PCIe "link training" errors.)
  - lazide 4y ago
    And yet, I've had several block level bit flips on SSDs and NVMe over the years in high heat/high load environments that weren't that. And one caused a kernel panic because the fs metadata now pointed towards impossible values. So tell me again how none of this matters?

by derefr 4y ago

lazide 4y ago

If they are corrupted on disk the behavior is not so deterministic as a ‘broken image’ and a reload. Corrupted on disk content causes software crashes, hangs, and other broken behavior users definitely don’t like. Especially when it’s the filesystem metadata which gets corrupted. Because merely trying to read it can cause severe issues at the filesystem level. I take it you haven’t dealt with failing storage much before?

derefr 4y ago
I maintain database and object-storage clusters for a living. Dealing with failing storage is half my job. > Especially when it’s the filesystem metadata which gets corrupted. We're not talking about filesystem metadata, though. Filesystem metadata is all "of a piece" — if you have a checksumming filesystem, then you can't not checksum some of the filesystem metadata, because all the metadata lives in (the moral equivalent of) a single database file the filesystem maintains, and that database gets checksummed. It's all one data structure, where the checksumming is a thing you do to that data structure, not to individual nodes within it. (For a tree filesystem like btrfs, this would be the non-cryptographic equivalent of a merkle-tree hash.) The only way you could even potentially turn off filesystem features for some metadata (dirent, freelist, etc) nodes but not others, would be to split your filesystem into multiple filesystems. No, to be clear, we're specifically talking about what happens inside the filesystem's extents. Those can experience corruption without that causing any undue issues, besides "the data you get from fread(3) is wrong." Unlike filesystem metadata, which is all required for the filesystem's integrity, a checksumming filesystem can choose whether to "look" inside file extents, or to treat them as opaque. And it can (in theory) make that choice per file, if it likes. From the FS's perspective, an extent is just a range of reserved disk blocks. Now, an assumption: only storage arrays use spinning rust for anything any more. The only disk problems consumer devices face any more are SSD degradation problems, not HDD degradation problems. (Even if you don't agree with this assumption by itself, it's much more clear-cut if you consider only devices operated by people willing to choose to use a filesystem that's not the default one for their OS.) This assumption neatly cleaves the problem-space in two: - How should a filesystem on a RAID array, set up for a business or prosumer use-case, deal with HDD faults? - How should a single-device filesystem used in a consumer use-case deal with SDD faults? The HDD-faults case comes down to: filesystem-level storage pool management with filesystem-driven redundant reads, with kernel blocking-read timeouts to avoid hangs, with async bad-sector remapping for timed out reads. Y'know: ZFS. While the SDD-faults case comes down to: read the bad data. Deal with the bad data. You won't get any hangs, until the day the whole thing just stops working. The worst you'll get is bit-rot. And even then, it's rare, because NAND controllers use internal space for error-correction, entirely invisibly to the kernel. (See also: http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/) In fact, in my own personal experience, the most likely cause of incorrect or corrupt data ending up on an SSD/NVMe disk, is that the CPU or memory of the system is bad, and so one or the other is corrupting the memory that will be written to disk before or during the write. (I've personally had this happen at least twice. What to look for to diagnose this: PCIe "link training" errors.)
- lazide 4y ago
  And yet, I've had several block level bit flips on SSDs and NVMe over the years in high heat/high load environments that weren't that. And one caused a kernel panic because the fs metadata now pointed towards impossible values. So tell me again how none of this matters?