5 ms·
Why is it ‘wasted’? Those things are mostly free on modern hardware. The challenge with your thesis here is that the only one who can know what is ‘that import
by lazide 4y ago
Why is it ‘wasted’? Those things are mostly free on modern hardware.
The challenge with your thesis here is that the only one who can know what is ‘that important’ is YOU, and your decision making and communication bandwidth is already the limiting factor.
For many users, that cat video would be heartbreaking to lose, and they don’t have term papers to worry about.
So having to decide or think what is or is not ‘important enough’ to you, and communicate that to the system, just makes everything slower than putting everything on a system good enough to protect the most sensitive and high value data you have.
- didgetmaster 4y agoNothing is free or even 'mostly free' when managing data. Data security (encryption), redundancy (backups), and integrity (checksums, etc.) all impose a cost on the system. Getting each piece of data properly classified will always be a challenge (AI or other tools may help with that), but it would still be nice to be able to do it. If I have a 50GB video file that I could easily re-download off the Internet, it would be nice to be able to turn off any security, redundancy, or integrity features for it. I wonder how many petabytes of storage space is being wasted by having multiple backups of all the operating system files that could be easily downloaded from multiple websites. Do I really need to encrypt that GB file that 10 million people also have a copy of? Am I worried if a single pixel in that high resolution photo has changed due to bit rot?
- Arnavion 4y ago>Do I really need to encrypt that GB file that 10 million people also have a copy of? Indeed you don't. Poettering has a similar idea in [1] (scroll down to "Summary of Resources and their Protections" for the tl;dr table), where he imagines OS files are only protected by dm-verity (for Silverblue-style immutable distros) / dm-integrity (for regular mutable distros). [1]: https://0pointer.net/blog/authenticated-boot-and-disk-encryption-on-linux.html https://0pointer.net/blog/authenticated-boot-and-disk-encryp...
- lazide 4y agoWorried if it gets lost or mangled? Not necessarily. Worried if it’s happening and I have no idea, and it’s spreading - due to hardware or software problems? And I’ll only discover it when something I do actually really care about and can’t easily replace? Absolutely! The big issue regarding duplication is really more a identification/‘supply chain’ issue. The last thing I want to be doing is trying to figure out how to get that other file from somewhere (that works), from whoever ‘has another copy’, when it gets mangled and I need a replacement ASAP. If you’re thinking of our local filesystems as potentially just a cache, we have no easy or secure way right now to fingerprint or recreate the other entries in the cache from sources (minus browser caches or the like, but even then, re-retrieving it may return different or not content). So backups are really keeping the computing equivalent of local ‘ability to manufacture’ handy as a mitigation against risk. It’s not waste, anymore than keeping the ability to manufacture it’s own tanks and weapons in-country is a waste for a nation. It’s an insurance cost against real world problems, and causes a moral hazard with other actors if not done. And CRC32/CRC32C (even in Java!) is > 16GB/s per core in modern processors, and more than adequate for block sized (typ. < 32MB) checksums. Blake3 is > 2GB/s per core, and is more than adequate for… well every use case we’re currently aware of. Modern OS’s really don’t have any excuses for not doing it.
- derefr 4y ago> For many users, that cat video would be heartbreaking to lose, and they don’t have term papers to worry about. Depends on where that cat video is / how it ended up on the disk. The user explicitly saved it to their user-profile Downloads directory? Yeah, sure, the user might care a lot about preserving that data. There's intent there. The user's web browser implicitly saved it into the browser's cache directory? No, the user absolutely doesn't care. That directory is a pure transparent optimization over just loading the resource from the URL again; and the browser makes no guarantees of anything in it surviving for even a few minutes. The user doesn't even know they have the data; only the browser does. As such, the browser should be able to tell the filesystem that this data is discardable cache data, and the filesystem should be able to apply different storage policies based on that. This is already true of managed cache/spool/tmp directories vis-a-vis higher-level components of the OS. macOS, for example, knows that stuff that's under ~/Library/Caches can be purged when disk space is tight, so it counts it as "reclaimable space"; and in some cases (caches that use CoreData) the OS can even garbage-collect them itself. So, why not also avoid making these files a part of backups? Why not avoid checksumming them? Etc.
- lazide 4y agoBackups - possibly, but no one I know counts COW/Snapshots, etc. as backups. Backup software generally already avoids copying those. They can be ways to restore to a point in time deterministically - but then they are absolutely needed to do so! Otherwise, the software is going to be acting differently with a bunch of data gone from underneath it, no? Check summing is more about being able to detect errors (and deterministically know if data corruption is occurring). So yes, absolutely temporary and cache files should be checksummed. If that data is corrupted, it will cause crashes of the software using them and downstream corruption after all. Why would I not want that to get caught before my software crashes or my output document (for instance) is being silently corrupted because one of the temporary files used when editing it got corrupted to/from disk?
- derefr 4y ago> So yes, absolutely temporary and cache files should be checksummed. If that data is corrupted, it will cause crashes of the software using them and downstream corruption after all. ...no? I don't care if a video in my browser's cache ends up with a few corrupt blocks when I play it again a year later. Video codecs are designed to be tolerant of that. You'll get a glitchy section in a few frames, and then hit the next keyframe and everything will clean up. In fact, most encodings — of images, audio, even text — are designed to be self-synchronizing in the face of corruption. I think you're thinking specifically of working-state files, which usually need to be perfect and guaranteed-trusted, because they're in normalized low-redundancy forms and are also used to derive other data from. But when I say "caching", I'm talking about cached final-form assets intended for direct human consumption. These get corrupted all the time, from network errors during download, disk storage errors on NASes, etc; and people mostly just don't care. For video, they just watch past it. For a web page, they hard-refresh it and everything's fine the second time around. If you think it's impossible to differentiate these two cases: well, that's because we don't explicitly ask developers to differentiate them. There could be separate ~/Library/ViewCache and ~/Library/StateCache directories. And before you ask, a good example of a large "ViewCache" asset that's not browser-related: a video-editor render-preview video file (the low-quality / thumbnail-sized kind, used for scrubbing.)