5 ms·
File Integrity 6: Which image format is most resilient?
- social_quotient 7y agoIs “vandal” by chance an open source utility? I’d like to test some other files and formats.
- mark-r 7y agoIt doesn't sound like it would be hard to write your own. Not sure how useful it is though, since it doesn't make any attempt to replicate the kind of errors you would see in the wild.
- yjftsjthsd-h 7y agoAs an alternative option: zzuf[0] is meant to be a fuzzer, but its take on that is "randomly change random bytes in a file" (it can be run as a wrapper around a victim process like your image viewer, but it's happy to run on bare files). [0] http://caca.zoy.org/wiki/zzuf http://caca.zoy.org/wiki/zzuf
- mwest 7y agoDoesn't look like the code is freely available, although the author states they'll provide a copy of the app for research purposes if asked: https://eclecticlight.co/2020/04/19/last-week-on-my-mac-vandalism/ https://eclecticlight.co/2020/04/19/last-week-on-my-mac-vand...
- vitovito 7y agoJPEG can be made almost as resilient as an uncompressed format by adding in restart markers. Restart markers duplicate the JPEG headers within the file, at however many intervals you like, so the bitstream can be, well, restarted, limiting the effect of any corruption. `jpegtran` can insert them losslessly.
- sllabres 7y agoInteresting, didn't know. For arbitrary data I found "rsbep".
- nneonneo 7y agoIn general, most compressed file formats don't even bother to be resilient against errors, since error correction mostly runs against the point of compression (to make files as small as possible). Some file formats do have error detection - for example, PNG provides a CRC-32 of every chunk, and compressed formats like Zip and gzip have similar checksums. If you happen to know (or guess) that a compressed file with a checksum only has 1-2 bytes in error, it is definitely possible to recover the original file through bruteforce. You can probably even do this to a file without a checksum just by testing which possibilities parse correctly. However, corruption on filesystems rarely takes the form of a small number of bytes; oftentimes the file's metadata gets corrupted such that entire blocks of the file (anywhere from 512 bytes to 64KB) are swapped, dropped or lost. Instead of worrying about corruption-resistant image formats (since storing everything as uncompressed TIFF/BMP is incredibly wasteful), it's probably worth using a more corruption-resistant filesystem instead (i.e. not FAT!).
- cmurf 7y agoI guess this means the PNG reader in the article example ignores the CRC-32? I wonder if a smarter reader could warn on error, assume a bit flip and iterate to see if any one flip causes a CRC-32 check to no longer mismatch (what's embedded)? Of course, the embedded CRC-32 checksum itself could have been corrupted. While I agree this is better a function of filesystems, images invariably are written to FAT from the get go. And the default filesystem offerings on most OS's lack error detection: NTFS, APFS, and ext4 being top three. There are a few Linux distributions, most notably openSUSE, that defaults to Btrfs which has checksums for everything. (I'm not certain whether FreeBSD defaults to UFS or ZFS; of course ZFS also checksums everything.)
- yjftsjthsd-h 7y ago> I'm not certain whether FreeBSD defaults to UFS or ZFS; of course ZFS also checksums everything. There isn't really a default; the installer makes you pick UFS/ZFS.
- nneonneo 7y agoMany PNG readers that I've tested do verify the CRC-32, but there are obviously many that don't. Verification chews up cycles that are useless 99+% of the time, and for PNG, verifying a chunk's CRC32 would delay decoding if the file is being streamed. I made a CTF challenge about uncorrupting PNGs a few years back; here's a writeup from a team that solved it: https://github.com/ctfs/write-ups-2015/tree/master/plaidctf-2015/forensics/png-uncorrupt https://github.com/ctfs/write-ups-2015/tree/master/plaidctf-.... In this particular case, a PNG was uploaded as "text" to an FTP server, which corrupts newline sequences. A bit of cleverness allows you to recover the original. (The PNG header even explicitly contains "newline" sequences to catch this type of corruption!)
- mark-r 7y agoI like the idea of a utility that can add ECC to any arbitrary file. You'd need to ensure you had an uncorrupted copy of the utility that you could use in the future though, or the images become unreadable even if they're uncorrupted themselves - unless the ECC is contained in a sidecar file.
- zamadatix 7y agoPAR2 or most file compression formats can fit the bill. Of course if it's going to stay put a filesystem that does this might be more convenient.
- mark-r 7y agoYes, it might be more convenient when built into a filesystem. But now you must keep alive an OS that can handle that filesystem. That seems like a much greater burden for the long haul.
- SloopJon 7y agoA previous post in the series tests a Parchive utility called MacPAR deLuxe: https://eclecticlight.co/2020/04/20/file-integrity-5-how-well-does-error-correcting-code-work/ https://eclecticlight.co/2020/04/20/file-integrity-5-how-wel...
- mark-r 7y agoLooks like a very useful utility, and it keeps the ECC in a .par2 sidecar file exactly as I wanted. Unfortunately it's Mac only.
- GoblinSlayer 7y agohttps://en.wikipedia.org/wiki/Parchive#Multi-Platform https://en.wikipedia.org/wiki/Parchive#Multi-Platform
- mark-r 7y ago
- arthurfm 7y agoThis might be a stupid question, but with a container format such as HEIF could you simply store multiple identical copies of an image inside the container? If one image gets corrupted you have the other to fall back on. Two HEIF or AVIF images should be about the same size as a single JPEG too.
- exmadscientist 7y agoIt's both much less wasteful and much more useful to add an error correction (FEC) block instead. With random errors spread across two copies, how do you know which chunks of each are correct? Sure, you could use a Merkle tree or similar, but FEC is a better solution. For size penalties of order 10% in practical use, FEC blocks can detect and correct errors not just in the data but in the FEC blocks themselves too. Their biggest downside is that their computational complexity can be high. This is only a problem during creation and decoding on error; if the data is checksummed, you only need to take the slow path and touch the FEC block if an error is encountered. This is a great tradeoff for archival uses, and good for many other uses.
- mark-r 7y agoThere used to be a format that was highly resilient to errors. It was called film.
- jdminhbg 7y agoAs long as you didn't send it through an airport X-ray machine, among other issues.
- mark-r 7y agoX-ray machines are only a problem for undeveloped film. Developed film is much more resilient.
- avmich 7y agoOf course singular - or linear, say, from bad projection device - defects in that are rather final, and this format has a host of inconveniences like fire hazard, inability to get transferred over networks, comparatively huge costs of copying, large physical size demands etc. We can perhaps use a convenient digital format for images and add to it, ahem, anti-corruption measures. Single file is still going to be vulnerable - even though to a less extent - if you, say, add some Reed-Solomon protection; but now you can use distributed networks the size of a planet, or even more. Organization of such a storage could be an interesting problem - at least partially solved already.
- mark-r 7y agoI'm not saying film is perfect. But it's an often overlooked choice that has worked well for the last 50 years.
- cerberusss 7y agoIf you're afraid of file corruption, install par2 with homebrew. My procedure for pictures is that monthly, I copy them off of my phone onto my storage. Then I run par2 in that directory to add a parity file. That all goes onto the backup.
- brnt 7y agoI do the same. To help with 'maintenance' (adding par to newly added files, checking and handling integrity for existing ones) I wrote a small utility you may like: https://github.com/brenthuisman/par2deep https://github.com/brenthuisman/par2deep
- cerberusss 7y agoVery nice, thanks!
- jwr 7y agoI wrote my little consistency checker (https://github.com/jwr/ccheck https://github.com/jwr/ccheck) specifically for checking my image archives against corruption. I am appalled that in this day and age we still (mostly) use tools and filesystems that have no way of knowing if data has been corrupted. Before I wrote the tool, I searched online and couldn't find anything that would be small, simple, easily maintained, and would last for at least 10 years. I now use it with all my archives, which means I know if a given replica is correct.
- renewiltord 7y agoWould you outperform uncompressed TIFF with sufficient copies of a compressed file? Or compressed file plus parity file?
- BiteCode_dev 7y agoGenuine question: why would you want that? Isn't that the role of the file system or the data transport?