5 ms·

Nothing is free or even 'mostly free' when managing data. Data security (encryption), redundancy (backups), and integrity (checksums, etc.) all impose a cost on

Nothing is free or even 'mostly free' when managing data. Data security (encryption), redundancy (backups), and integrity (checksums, etc.) all impose a cost on the system. Getting each piece of data properly classified will always be a challenge (AI or other tools may help with that), but it would still be nice to be able to do it. If I have a 50GB video file that I could easily re-download off the Internet, it would be nice to be able to turn off any security, redundancy, or integrity features for it. I wonder how many petabytes of storage space is being wasted by having multiple backups of all the operating system files that could be easily downloaded from multiple websites. Do I really need to encrypt that GB file that 10 million people also have a copy of? Am I worried if a single pixel in that high resolution photo has changed due to bit rot?

Arnavion 4y ago
>Do I really need to encrypt that GB file that 10 million people also have a copy of? Indeed you don't. Poettering has a similar idea in [1] (scroll down to "Summary of Resources and their Protections" for the tl;dr table), where he imagines OS files are only protected by dm-verity (for Silverblue-style immutable distros) / dm-integrity (for regular mutable distros). [1]: https://0pointer.net/blog/authenticated-boot-and-disk-encryption-on-linux.html https://0pointer.net/blog/authenticated-boot-and-disk-encryp...
lazide 4y ago
Worried if it gets lost or mangled? Not necessarily. Worried if it’s happening and I have no idea, and it’s spreading - due to hardware or software problems? And I’ll only discover it when something I do actually really care about and can’t easily replace? Absolutely! The big issue regarding duplication is really more a identification/‘supply chain’ issue. The last thing I want to be doing is trying to figure out how to get that other file from somewhere (that works), from whoever ‘has another copy’, when it gets mangled and I need a replacement ASAP. If you’re thinking of our local filesystems as potentially just a cache, we have no easy or secure way right now to fingerprint or recreate the other entries in the cache from sources (minus browser caches or the like, but even then, re-retrieving it may return different or not content). So backups are really keeping the computing equivalent of local ‘ability to manufacture’ handy as a mitigation against risk. It’s not waste, anymore than keeping the ability to manufacture it’s own tanks and weapons in-country is a waste for a nation. It’s an insurance cost against real world problems, and causes a moral hazard with other actors if not done. And CRC32/CRC32C (even in Java!) is > 16GB/s per core in modern processors, and more than adequate for block sized (typ. < 32MB) checksums. Blake3 is > 2GB/s per core, and is more than adequate for… well every use case we’re currently aware of. Modern OS’s really don’t have any excuses for not doing it.

Arnavion 4y ago

>Do I really need to encrypt that GB file that 10 million people also have a copy of? Indeed you don't. Poettering has a similar idea in [1] (scroll down to "Summary of Resources and their Protections" for the tl;dr table), where he imagines OS files are only protected by dm-verity (for Silverblue-style immutable distros) / dm-integrity (for regular mutable distros). [1]: https://0pointer.net/blog/authenticated-boot-and-disk-encryption-on-linux.html https://0pointer.net/blog/authenticated-boot-and-disk-encryp...

lazide 4y ago

Worried if it gets lost or mangled? Not necessarily. Worried if it’s happening and I have no idea, and it’s spreading - due to hardware or software problems? And I’ll only discover it when something I do actually really care about and can’t easily replace? Absolutely! The big issue regarding duplication is really more a identification/‘supply chain’ issue. The last thing I want to be doing is trying to figure out how to get that other file from somewhere (that works), from whoever ‘has another copy’, when it gets mangled and I need a replacement ASAP. If you’re thinking of our local filesystems as potentially just a cache, we have no easy or secure way right now to fingerprint or recreate the other entries in the cache from sources (minus browser caches or the like, but even then, re-retrieving it may return different or not content). So backups are really keeping the computing equivalent of local ‘ability to manufacture’ handy as a mitigation against risk. It’s not waste, anymore than keeping the ability to manufacture it’s own tanks and weapons in-country is a waste for a nation. It’s an insurance cost against real world problems, and causes a moral hazard with other actors if not done. And CRC32/CRC32C (even in Java!) is > 16GB/s per core in modern processors, and more than adequate for block sized (typ. < 32MB) checksums. Blake3 is > 2GB/s per core, and is more than adequate for… well every use case we’re currently aware of. Modern OS’s really don’t have any excuses for not doing it.