4 ms·
> The reality is, for things like video and pictures it probably won't be a problem. However, if the data is sensitive, you should be using a raid 6 with 2 disk
by acdha 10y ago
> The reality is, for things like video and pictures it probably won't be a problem. However, if the data is sensitive, you should be using a raid 6 with 2 disk redundancy, coupled with constant data scrubbing to ensure that the disks remain bit rot free. The probability of bit rot occurring at the same place is probably less than 1 in 10^16, possibly even less if data scrubbing is constant.
The first assertion is simply wrong: the internet is littered with people who have corrupt media files which cause user-visible errors. Beyond that, however, RAID can be part of a strategy but is far from sufficient. The paper you linked provides a good discussion of the reasons why but the root mistake is assuming that failures are perfectly distributed random events at a single point.
End-to-end strong integrity checks are so important because e.g. in that scenario with a two-drive RAID array you'd still be subject to errors caused transferring the data from the host to the RAID controller, on the controller itself, in communication to the drives (one reason why it's important to mix vendors & models), and on the path back when reading data. You'd also have to worry about corruption in memory, errors caused by e.g. power fluctuation or other hardware events which may be grossly underestimated because they're silently detected and retried until you get lucky and one of the 100,000 errors happened to generate a valid checksum, etc.
Generating a cryptographic hash as early as possible and checking it at every stage is key both because it catches almost all of those scenarios but also because it allows you to confidently report a positive validation result rather than just the absence of detected errors.
- rubatuga 10y agoI agree with most of your points, and I agree that end-to-end integrity is important where information is critical. But my first assertion is not wrong. You are assuming that the flip of a single bit from a photo or video is catastrophic, both in the rendering of the file, as well as the resulting harm done to the user/individual. The reality is that most people don't value >99.999999% accuracy, and are fine with the current state of data storage. Out of 1,000,000 photos, will the user notice/care if 1 jpeg has a 16x16 block that is a different colour?
- luckydude 10y agoFor photos, sure, it's not that big of a deal. But not all files are independent like that. A single bit flip in an executable can be a huge deal. I'm sure you can think of other examples.
- pdog 10y agoFlipping a single bit of an encrypted file renders the whole thing useless.
- acdha 10y agoThere are other places bits flip which break photo or video decoding outright. More importantly, however, is that this isn't a reliably random process but often a highly-correlated one. Many users won't have a problem but the people who do often have many files affected. I've seen multiple cases where that wasn't noticed until after their backup software, cloud service, etc. had copied the corrupted bits, not to mention cases where data was corrupted on a RAID array which wasn't regularly scrubbed and so the corruption wasn't noticed until the other disk failed, long after the invalid data had been written to tape as well.