4 ms·
I'll die on this hill... If may files that are named like this: DSC009847.JPG were actually named like this: DSC009847-b3-73ea2364d158.JPG where "-b
by TacticalCoder 1y ago
I'll die on this hill... If may files that are named like this:
DSC009847.JPG
were actually named like this:
DSC009847-b3-73ea2364d158.JPG
where "-b3-" means "what's coming before the extension are the first x bits (choose as many hexdigits as you want) of the Blake3 cryptographic hash of the file...
We'd be living in a better world.
I do that for many of my files. Notably family pictures and family movies, but also .iso files, tar/gzip'ed files, etc.
This makes detecting bitflips trivial.
I've create little shellscripts for verification, backups, etc. that work with files having such a naming scheme.
It's bliss.
My world is a better place now. I moved to such a scheme after I had a series of 20 pictures from vacation with old friends that were corrupted (thankfully I had backups, but the concept of "determining which one is the correct file" programmatically is not that easy).
And, yes, it detected one bitflip since I'm using it.
I don't always verify all the checksums: but I've got a script that does random sampling... It picks x% of the files with such a naming scheme and verifies the checksum of these x% of files picked randomly.
It's not incompatible with ZFS: I still run ZFS on my Proxmox server. It's not incompatible with restic/borg/etc. either.
This solves so many issues, including the "How do you know your data is correct?" (answer is: "Because I've already looked that family movie after the cryptographic hash was added to its name").
Not a panacea but doesn't hurt and it's really not much work.
- homebrewer 1y agoIt's an old idea and is also how some anime fansub groups prepare their releases: the filename of each episode contains the CRC32 of the file inside [square brackets]. Doesn't really make much sense for BitTorrent uploads (which provides its own much stronger hashes), it's a holdover from the era of IRC bots.
- networked 1y agoI prefer DSC009847.JPG.b3sum sidecar files [1] or per-directory checksum files like B3SUMS because they can be verified with standard tools. This scheme also allows you to checksum files whose names you can't or don't want to change. (Though in that situation you have an alternative of using a symlink for either the original name or the name with the checksum.) I have used the scheme less since I adopted ZFS. I do use very similar example.com/foo/bar/b3-abcd0123.html for https://example.com/foo/bar https://example.com/foo/bar in the archival tool for outgoing links on my website. It avoids the need to have a date prefix like in the Wayback Machine while preventing duplication. Speaking of .iso files. A recent PR [2] to my favorite Linux USB-disk-image burning tool Caligula has added support for detecting and verifying sidecar files like foo.iso.sha256 (albeit not Blake). [1] https://en.wikipedia.org/wiki/Sidecar_file https://en.wikipedia.org/wiki/Sidecar_file [2] https://github.com/ifd3f/caligula/pull/186 https://github.com/ifd3f/caligula/pull/186
- somat 1y agoThere are also tools like mtree to help you audit and maintain filesystem integrity. https://man.openbsd.org/mtree https://man.openbsd.org/mtree