4 ms·

> For many users, that cat video would be heartbreaking to lose, and they don’t have term papers to worry about. Depends on where that cat video is / how it en

by derefr 4y ago

> For many users, that cat video would be heartbreaking to lose, and they don’t have term papers to worry about. Depends on where that cat video is / how it ended up on the disk. The user explicitly saved it to their user-profile Downloads directory? Yeah, sure, the user might care a lot about preserving that data. There's intent there. The user's web browser implicitly saved it into the browser's cache directory? No, the user absolutely doesn't care. That directory is a pure transparent optimization over just loading the resource from the URL again; and the browser makes no guarantees of anything in it surviving for even a few minutes. The user doesn't even know they have the data; only the browser does. As such, the browser should be able to tell the filesystem that this data is discardable cache data, and the filesystem should be able to apply different storage policies based on that. This is already true of managed cache/spool/tmp directories vis-a-vis higher-level components of the OS. macOS, for example, knows that stuff that's under ~/Library/Caches can be purged when disk space is tight, so it counts it as "reclaimable space"; and in some cases (caches that use CoreData) the OS can even garbage-collect them itself. So, why not also avoid making these files a part of backups? Why not avoid checksumming them? Etc.

lazide 4y ago
Backups - possibly, but no one I know counts COW/Snapshots, etc. as backups. Backup software generally already avoids copying those. They can be ways to restore to a point in time deterministically - but then they are absolutely needed to do so! Otherwise, the software is going to be acting differently with a bunch of data gone from underneath it, no? Check summing is more about being able to detect errors (and deterministically know if data corruption is occurring). So yes, absolutely temporary and cache files should be checksummed. If that data is corrupted, it will cause crashes of the software using them and downstream corruption after all. Why would I not want that to get caught before my software crashes or my output document (for instance) is being silently corrupted because one of the temporary files used when editing it got corrupted to/from disk?
- derefr 4y ago
  > So yes, absolutely temporary and cache files should be checksummed. If that data is corrupted, it will cause crashes of the software using them and downstream corruption after all. ...no? I don't care if a video in my browser's cache ends up with a few corrupt blocks when I play it again a year later. Video codecs are designed to be tolerant of that. You'll get a glitchy section in a few frames, and then hit the next keyframe and everything will clean up. In fact, most encodings — of images, audio, even text — are designed to be self-synchronizing in the face of corruption. I think you're thinking specifically of working-state files, which usually need to be perfect and guaranteed-trusted, because they're in normalized low-redundancy forms and are also used to derive other data from. But when I say "caching", I'm talking about cached final-form assets intended for direct human consumption. These get corrupted all the time, from network errors during download, disk storage errors on NASes, etc; and people mostly just don't care. For video, they just watch past it. For a web page, they hard-refresh it and everything's fine the second time around. If you think it's impossible to differentiate these two cases: well, that's because we don't explicitly ask developers to differentiate them. There could be separate ~/Library/ViewCache and ~/Library/StateCache directories. And before you ask, a good example of a large "ViewCache" asset that's not browser-related: a video-editor render-preview video file (the low-quality / thumbnail-sized kind, used for scrubbing.)
  - lazide 4y ago
    If they are corrupted on disk the behavior is not so deterministic as a ‘broken image’ and a reload. Corrupted on disk content causes software crashes, hangs, and other broken behavior users definitely don’t like. Especially when it’s the filesystem metadata which gets corrupted. Because merely trying to read it can cause severe issues at the filesystem level. I take it you haven’t dealt with failing storage much before?
    - derefr 4y ago
      I maintain database and object-storage clusters for a living. Dealing with failing storage is half my job. > Especially when it’s the filesystem metadata which gets corrupted. We're not talking about filesystem metadata, though. Filesystem metadata is all "of a piece" — if you have a checksumming filesystem, then you can't not checksum some of the filesystem metadata, because all the metadata lives in (the moral equivalent of) a single database file the filesystem maintains, and that database gets checksummed. It's all one data structure, where the checksumming is a thing you do to that data structure, not to individual nodes within it. (For a tree filesystem like btrfs, this would be the non-cryptographic equivalent of a merkle-tree hash.) The only way you could even potentially turn off filesystem features for some metadata (dirent, freelist, etc) nodes but not others, would be to split your filesystem into multiple filesystems. No, to be clear, we're specifically talking about what happens inside the filesystem's extents. Those can experience corruption without that causing any undue issues, besides "the data you get from fread(3) is wrong." Unlike filesystem metadata, which is all required for the filesystem's integrity, a checksumming filesystem can choose whether to "look" inside file extents, or to treat them as opaque. And it can (in theory) make that choice per file, if it likes. From the FS's perspective, an extent is just a range of reserved disk blocks. Now, an assumption: only storage arrays use spinning rust for anything any more. The only disk problems consumer devices face any more are SSD degradation problems, not HDD degradation problems. (Even if you don't agree with this assumption by itself, it's much more clear-cut if you consider only devices operated by people willing to choose to use a filesystem that's not the default one for their OS.) This assumption neatly cleaves the problem-space in two: - How should a filesystem on a RAID array, set up for a business or prosumer use-case, deal with HDD faults? - How should a single-device filesystem used in a consumer use-case deal with SDD faults? The HDD-faults case comes down to: filesystem-level storage pool management with filesystem-driven redundant reads, with kernel blocking-read timeouts to avoid hangs, with async bad-sector remapping for timed out reads. Y'know: ZFS. While the SDD-faults case comes down to: read the bad data. Deal with the bad data. You won't get any hangs, until the day the whole thing just stops working. The worst you'll get is bit-rot. And even then, it's rare, because NAND controllers use internal space for error-correction, entirely invisibly to the kernel. (See also: http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/) In fact, in my own personal experience, the most likely cause of incorrect or corrupt data ending up on an SSD/NVMe disk, is that the CPU or memory of the system is bad, and so one or the other is corrupting the memory that will be written to disk before or during the write. (I've personally had this happen at least twice. What to look for to diagnose this: PCIe "link training" errors.)

by derefr 4y ago

lazide 4y ago

Backups - possibly, but no one I know counts COW/Snapshots, etc. as backups. Backup software generally already avoids copying those. They can be ways to restore to a point in time deterministically - but then they are absolutely needed to do so! Otherwise, the software is going to be acting differently with a bunch of data gone from underneath it, no? Check summing is more about being able to detect errors (and deterministically know if data corruption is occurring). So yes, absolutely temporary and cache files should be checksummed. If that data is corrupted, it will cause crashes of the software using them and downstream corruption after all. Why would I not want that to get caught before my software crashes or my output document (for instance) is being silently corrupted because one of the temporary files used when editing it got corrupted to/from disk?

derefr 4y ago
> So yes, absolutely temporary and cache files should be checksummed. If that data is corrupted, it will cause crashes of the software using them and downstream corruption after all. ...no? I don't care if a video in my browser's cache ends up with a few corrupt blocks when I play it again a year later. Video codecs are designed to be tolerant of that. You'll get a glitchy section in a few frames, and then hit the next keyframe and everything will clean up. In fact, most encodings — of images, audio, even text — are designed to be self-synchronizing in the face of corruption. I think you're thinking specifically of working-state files, which usually need to be perfect and guaranteed-trusted, because they're in normalized low-redundancy forms and are also used to derive other data from. But when I say "caching", I'm talking about cached final-form assets intended for direct human consumption. These get corrupted all the time, from network errors during download, disk storage errors on NASes, etc; and people mostly just don't care. For video, they just watch past it. For a web page, they hard-refresh it and everything's fine the second time around. If you think it's impossible to differentiate these two cases: well, that's because we don't explicitly ask developers to differentiate them. There could be separate ~/Library/ViewCache and ~/Library/StateCache directories. And before you ask, a good example of a large "ViewCache" asset that's not browser-related: a video-editor render-preview video file (the low-quality / thumbnail-sized kind, used for scrubbing.)
- lazide 4y ago
  If they are corrupted on disk the behavior is not so deterministic as a ‘broken image’ and a reload. Corrupted on disk content causes software crashes, hangs, and other broken behavior users definitely don’t like. Especially when it’s the filesystem metadata which gets corrupted. Because merely trying to read it can cause severe issues at the filesystem level. I take it you haven’t dealt with failing storage much before?
  - derefr 4y ago
    I maintain database and object-storage clusters for a living. Dealing with failing storage is half my job. > Especially when it’s the filesystem metadata which gets corrupted. We're not talking about filesystem metadata, though. Filesystem metadata is all "of a piece" — if you have a checksumming filesystem, then you can't not checksum some of the filesystem metadata, because all the metadata lives in (the moral equivalent of) a single database file the filesystem maintains, and that database gets checksummed. It's all one data structure, where the checksumming is a thing you do to that data structure, not to individual nodes within it. (For a tree filesystem like btrfs, this would be the non-cryptographic equivalent of a merkle-tree hash.) The only way you could even potentially turn off filesystem features for some metadata (dirent, freelist, etc) nodes but not others, would be to split your filesystem into multiple filesystems. No, to be clear, we're specifically talking about what happens inside the filesystem's extents. Those can experience corruption without that causing any undue issues, besides "the data you get from fread(3) is wrong." Unlike filesystem metadata, which is all required for the filesystem's integrity, a checksumming filesystem can choose whether to "look" inside file extents, or to treat them as opaque. And it can (in theory) make that choice per file, if it likes. From the FS's perspective, an extent is just a range of reserved disk blocks. Now, an assumption: only storage arrays use spinning rust for anything any more. The only disk problems consumer devices face any more are SSD degradation problems, not HDD degradation problems. (Even if you don't agree with this assumption by itself, it's much more clear-cut if you consider only devices operated by people willing to choose to use a filesystem that's not the default one for their OS.) This assumption neatly cleaves the problem-space in two: - How should a filesystem on a RAID array, set up for a business or prosumer use-case, deal with HDD faults? - How should a single-device filesystem used in a consumer use-case deal with SDD faults? The HDD-faults case comes down to: filesystem-level storage pool management with filesystem-driven redundant reads, with kernel blocking-read timeouts to avoid hangs, with async bad-sector remapping for timed out reads. Y'know: ZFS. While the SDD-faults case comes down to: read the bad data. Deal with the bad data. You won't get any hangs, until the day the whole thing just stops working. The worst you'll get is bit-rot. And even then, it's rare, because NAND controllers use internal space for error-correction, entirely invisibly to the kernel. (See also: http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/) In fact, in my own personal experience, the most likely cause of incorrect or corrupt data ending up on an SSD/NVMe disk, is that the CPU or memory of the system is bad, and so one or the other is corrupting the memory that will be written to disk before or during the write. (I've personally had this happen at least twice. What to look for to diagnose this: PCIe "link training" errors.)