4 ms·

In general, most compressed file formats don't even bother to be resilient against errors, since error correction mostly runs against the point of compression (

In general, most compressed file formats don't even bother to be resilient against errors, since error correction mostly runs against the point of compression (to make files as small as possible). Some file formats do have error detection - for example, PNG provides a CRC-32 of every chunk, and compressed formats like Zip and gzip have similar checksums. If you happen to know (or guess) that a compressed file with a checksum only has 1-2 bytes in error, it is definitely possible to recover the original file through bruteforce. You can probably even do this to a file without a checksum just by testing which possibilities parse correctly. However, corruption on filesystems rarely takes the form of a small number of bytes; oftentimes the file's metadata gets corrupted such that entire blocks of the file (anywhere from 512 bytes to 64KB) are swapped, dropped or lost. Instead of worrying about corruption-resistant image formats (since storing everything as uncompressed TIFF/BMP is incredibly wasteful), it's probably worth using a more corruption-resistant filesystem instead (i.e. not FAT!).

cmurf 7y ago
I guess this means the PNG reader in the article example ignores the CRC-32? I wonder if a smarter reader could warn on error, assume a bit flip and iterate to see if any one flip causes a CRC-32 check to no longer mismatch (what's embedded)? Of course, the embedded CRC-32 checksum itself could have been corrupted. While I agree this is better a function of filesystems, images invariably are written to FAT from the get go. And the default filesystem offerings on most OS's lack error detection: NTFS, APFS, and ext4 being top three. There are a few Linux distributions, most notably openSUSE, that defaults to Btrfs which has checksums for everything. (I'm not certain whether FreeBSD defaults to UFS or ZFS; of course ZFS also checksums everything.)
- yjftsjthsd-h 7y ago
  > I'm not certain whether FreeBSD defaults to UFS or ZFS; of course ZFS also checksums everything. There isn't really a default; the installer makes you pick UFS/ZFS.
- nneonneo 7y ago
  Many PNG readers that I've tested do verify the CRC-32, but there are obviously many that don't. Verification chews up cycles that are useless 99+% of the time, and for PNG, verifying a chunk's CRC32 would delay decoding if the file is being streamed. I made a CTF challenge about uncorrupting PNGs a few years back; here's a writeup from a team that solved it: https://github.com/ctfs/write-ups-2015/tree/master/plaidctf-2015/forensics/png-uncorrupt https://github.com/ctfs/write-ups-2015/tree/master/plaidctf-.... In this particular case, a PNG was uploaded as "text" to an FTP server, which corrupts newline sequences. A bit of cleverness allows you to recover the original. (The PNG header even explicitly contains "newline" sequences to catch this type of corruption!)
elcritch 7y ago
What’s surprising to me is that no file system (even ZFS) or database utilize error correction controls. Well aside from 1 bit/byte ECC style codes or raid. There are a few forms of forward error correction [1], which are already widely used in hardware [2]. There are some projects trying for long term fixity of storage [3], but nothing default or built in, afaict. I assumed RocksDB or some other modern KV engine would include FEC, at least the meta-data. Using DB’s on embedded devices reveals how easily (and regularly) a whole DB of data can be corrupted. A few bytes of error and an entire IoT DB can be lost. FEC’s can be cpu expensive but not that expensive nowadays. 1: https://en.wikipedia.org/wiki/Forward_error_correction https://en.wikipedia.org/wiki/Forward_error_correction 2: https://www.electronicdesign.com/technologies/communications/article/21789969/use-forward-error-correction-to-improve-data-communications https://www.electronicdesign.com/technologies/communications... 3: https://github.com/lrq3000/pyFileFixity https://github.com/lrq3000/pyFileFixity
- zozbot234 7y ago
  You can just use dm-raid for that if you like. Just because it's named "raid" doesn't mean you have to use it across disks.
  - elcritch 7y ago
    True, and I’ve considered doing that, but it means you’re doing 2/1 or at best 3/2 storage storage rates. Raid only uses XOR, but Reed-Solomon or other FEC codes can produce way better error correction rates. Say full recovery for standard storage error rates using only 1/10 the extra space along with more computation at recovery time. Add CPU hardware support like AES and then even battery devices could use it.
    - elcritch 7y ago
      Article on fec’s and storage, seems Ceph utilizes FEC: https://www.networkcomputing.com/data-centers/putting-erasure-codes-work https://www.networkcomputing.com/data-centers/putting-erasur...
    - 7y ago

cmurf 7y ago

I guess this means the PNG reader in the article example ignores the CRC-32? I wonder if a smarter reader could warn on error, assume a bit flip and iterate to see if any one flip causes a CRC-32 check to no longer mismatch (what's embedded)? Of course, the embedded CRC-32 checksum itself could have been corrupted. While I agree this is better a function of filesystems, images invariably are written to FAT from the get go. And the default filesystem offerings on most OS's lack error detection: NTFS, APFS, and ext4 being top three. There are a few Linux distributions, most notably openSUSE, that defaults to Btrfs which has checksums for everything. (I'm not certain whether FreeBSD defaults to UFS or ZFS; of course ZFS also checksums everything.)

yjftsjthsd-h 7y ago
> I'm not certain whether FreeBSD defaults to UFS or ZFS; of course ZFS also checksums everything. There isn't really a default; the installer makes you pick UFS/ZFS.
nneonneo 7y ago
Many PNG readers that I've tested do verify the CRC-32, but there are obviously many that don't. Verification chews up cycles that are useless 99+% of the time, and for PNG, verifying a chunk's CRC32 would delay decoding if the file is being streamed. I made a CTF challenge about uncorrupting PNGs a few years back; here's a writeup from a team that solved it: https://github.com/ctfs/write-ups-2015/tree/master/plaidctf-2015/forensics/png-uncorrupt https://github.com/ctfs/write-ups-2015/tree/master/plaidctf-.... In this particular case, a PNG was uploaded as "text" to an FTP server, which corrupts newline sequences. A bit of cleverness allows you to recover the original. (The PNG header even explicitly contains "newline" sequences to catch this type of corruption!)

elcritch 7y ago

What’s surprising to me is that no file system (even ZFS) or database utilize error correction controls. Well aside from 1 bit/byte ECC style codes or raid. There are a few forms of forward error correction [1], which are already widely used in hardware [2]. There are some projects trying for long term fixity of storage [3], but nothing default or built in, afaict. I assumed RocksDB or some other modern KV engine would include FEC, at least the meta-data. Using DB’s on embedded devices reveals how easily (and regularly) a whole DB of data can be corrupted. A few bytes of error and an entire IoT DB can be lost. FEC’s can be cpu expensive but not that expensive nowadays. 1: https://en.wikipedia.org/wiki/Forward_error_correction https://en.wikipedia.org/wiki/Forward_error_correction 2: https://www.electronicdesign.com/technologies/communications/article/21789969/use-forward-error-correction-to-improve-data-communications https://www.electronicdesign.com/technologies/communications... 3: https://github.com/lrq3000/pyFileFixity https://github.com/lrq3000/pyFileFixity

zozbot234 7y ago
You can just use dm-raid for that if you like. Just because it's named "raid" doesn't mean you have to use it across disks.
- elcritch 7y ago
  True, and I’ve considered doing that, but it means you’re doing 2/1 or at best 3/2 storage storage rates. Raid only uses XOR, but Reed-Solomon or other FEC codes can produce way better error correction rates. Say full recovery for standard storage error rates using only 1/10 the extra space along with more computation at recovery time. Add CPU hardware support like AES and then even battery devices could use it.
  - elcritch 7y ago
    Article on fec’s and storage, seems Ceph utilizes FEC: https://www.networkcomputing.com/data-centers/putting-erasure-codes-work https://www.networkcomputing.com/data-centers/putting-erasur...
  - 7y ago