3 ms·
> most of the concern is over very theoretical issues I've seen photo and other binary files become corrupted that were sitting on RAID drives. The RAID swears
by sho 9y ago
> most of the concern is over very theoretical issues
I've seen photo and other binary files become corrupted that were sitting on RAID drives. The RAID swears they're fine, the filesystem swears they're fine, both are checksummed so I believe them. The only possibility that I can see is that they were corrupted while being modified or transferred on non-ECC desktops connected to the RAID.
I'm not afraid of my computer crashing. I'm afraid of data I take great pains to preserve being silently, indeed undetectably, corrupted while in flight or in use. So that's why ECC is worth it to me.
- wila 9y agoExactly! In the past I've had a flimsy RAM module in a macbook Pro and it was a real pain. Everything appeared to work just fine, but on stressing the RAM with a lot of virtual machines the host would crash. That was not the main issue, but took some time to diagnose as I was also running beta virtualisation software and was tempted to blame the change instead of the hardware. Copying virtual machines from one disk to another did end up corrupting the data. That was painful to find out.
- sho 9y agoAlmost exactly the same thing happened to me - marginal RAM in an old style macbook pro. Drove me absolutely crazy. People always talk about cosmic rays as what ECC is guarding against - not at all! It's shitty RAM, especially in laptops. When it's under stress.. for example buffering files in a large copy.. and you find out months later when it's too late to fix it. Not "theoretical" at all.
- semi-extrinsic 9y agoI'm curious: if storing lots of photos as .dng, .png or .jpg on ZFS without ECC, one presumably gets bit flips eventually. How does this affect the files? Do you just get artifacts in the photo? Or does the file become unreadable? If so, can you recover the file (with artifacts)? I guess the answer boils down to how much non-recoverable but essential-for-reconstruction metadata there is in these file formats.
- sosuke 9y agoI had bit flips on a few JPGs and it renders them useless. Luckily I had a backup of a backup that had them uncorrupted. I'm still trying to find a complete solution to this problem. Presumably the TIFF or BMP file formats are more stable against bit flips. I'd been reading so much about it over the past year or so I got to wondering just how many times cosmic rays affect our brains and what kind of protections we're running up in our skulls.
- ianai 9y agoOur brains evolved through a chaotic, organic process. We're all the time storing new data and even losing data (selective memory). I'm thinking there's no mitigation process. If anything the random environmental noise might play some role in consciousness.
- semi-extrinsic 9y agoHave you tried (ab)using AFL-fuzz for this? It can create JPGs out of literally thin air, so I wouldn't be surprised. https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thin-air.html https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-th...
- acchow 9y agoDepends which part of the file gets corrupted, and the issues between PNG and JPG are dramatically different. There are key bytes in the file (like size of segment, start of segment, etc) that if corrupted would dramatically mess up your image. If it's just in the compressed image data you'll just see some artifacts, and a JPEG already has plenty of compression artifacts anyhow...