3 ms·

I'm curious: if storing lots of photos as .dng, .png or .jpg on ZFS without ECC, one presumably gets bit flips eventually. How does this affect the files? Do yo

I'm curious: if storing lots of photos as .dng, .png or .jpg on ZFS without ECC, one presumably gets bit flips eventually. How does this affect the files? Do you just get artifacts in the photo? Or does the file become unreadable? If so, can you recover the file (with artifacts)? I guess the answer boils down to how much non-recoverable but essential-for-reconstruction metadata there is in these file formats.

sosuke 9y ago
I had bit flips on a few JPGs and it renders them useless. Luckily I had a backup of a backup that had them uncorrupted. I'm still trying to find a complete solution to this problem. Presumably the TIFF or BMP file formats are more stable against bit flips. I'd been reading so much about it over the past year or so I got to wondering just how many times cosmic rays affect our brains and what kind of protections we're running up in our skulls.
- ianai 9y ago
  Our brains evolved through a chaotic, organic process. We're all the time storing new data and even losing data (selective memory). I'm thinking there's no mitigation process. If anything the random environmental noise might play some role in consciousness.
- semi-extrinsic 9y ago
  Have you tried (ab)using AFL-fuzz for this? It can create JPGs out of literally thin air, so I wouldn't be surprised. https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thin-air.html https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-th...
acchow 9y ago
Depends which part of the file gets corrupted, and the issues between PNG and JPG are dramatically different. There are key bytes in the file (like size of segment, start of segment, etc) that if corrupted would dramatically mess up your image. If it's just in the compressed image data you'll just see some artifacts, and a JPEG already has plenty of compression artifacts anyhow...

sosuke 9y ago

I had bit flips on a few JPGs and it renders them useless. Luckily I had a backup of a backup that had them uncorrupted. I'm still trying to find a complete solution to this problem. Presumably the TIFF or BMP file formats are more stable against bit flips. I'd been reading so much about it over the past year or so I got to wondering just how many times cosmic rays affect our brains and what kind of protections we're running up in our skulls.

ianai 9y ago
Our brains evolved through a chaotic, organic process. We're all the time storing new data and even losing data (selective memory). I'm thinking there's no mitigation process. If anything the random environmental noise might play some role in consciousness.
semi-extrinsic 9y ago
Have you tried (ab)using AFL-fuzz for this? It can create JPGs out of literally thin air, so I wouldn't be surprised. https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thin-air.html https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-th...

acchow 9y ago

Depends which part of the file gets corrupted, and the issues between PNG and JPG are dramatically different. There are key bytes in the file (like size of segment, start of segment, etc) that if corrupted would dramatically mess up your image. If it's just in the compressed image data you'll just see some artifacts, and a JPEG already has plenty of compression artifacts anyhow...