3 ms·
> I'm regularly seeing bit flips. How?
by PrimeMcFly 3y ago
> I'm regularly seeing bit flips.
How?
- dist-epoch 3y agoFor example 7zip decompression CRC failures that resolve on a second try. It would be longer to explain how, but I tracked it multiple times to a single bit flip in the decompressed output.
- TillE 3y agoI've also seen that, often enough that a hardware issue is extraordinarily unlikely. It's almost certainly just a boring old software bug.
- dist-epoch 3y agoPopular compressors like 7zip are some of the best tested software on the planet, because corruption is immediately detected. And we know hardware bugs are real, that's the whole point of rowhammer. You might want to look at Facebook research: > “Silent data corruptions are not limited to rare one in a million occurrences within a large-scale infrastructure. These errors are systemic and are not as well understood as the other failure modes like Machine Check Exceptions.” A large part of the responsibility should be shared by device makers, Facebook says. https://www.nextplatform.com/2021/03/01/facebook-architects-around-silent-data-corruption/ https://www.nextplatform.com/2021/03/01/facebook-architects-...
- PrimeMcFly 3y agoThanks, that's pretty interesting. I'd actually be really interested if you would care to explain how, I think others would find it interesting also. I'm not even sure how I would approach trying to do that.
- dist-epoch 3y agoTypically decompressing software deletes the output file if there is a CRC error. But I use 7zip as a library, so if I get a CRC error I still have the output. Which in my case is JSON files. By careful diffing and going over them I could identify single bit flips like '{"base": 10}' being decompressed to '{"bbse": 10}'. I made this example up, letter 'a' becoming 'b' might be a multi-bit flip, but you get the idea.