5 ms·

Addressing data integrity is a slippery slope of comparing hard drive bit read errors and ECC ram bit flips. The reality is, for things like video and pictures

Addressing data integrity is a slippery slope of comparing hard drive bit read errors and ECC ram bit flips. The reality is, for things like video and pictures it probably won't be a problem. However, if the data is sensitive, you should be using a raid 6 with 2 disk redundancy, coupled with constant data scrubbing to ensure that the disks remain bit rot free. The probability of bit rot occurring at the same place is probably less than 1 in 10^16, possibly even less if data scrubbing is constant. Here's a good article addressing bit rot: https://lockss.org/locksswiki/files/ACM2010.pdf https://lockss.org/locksswiki/files/ACM2010.pdf

roymurdock 10y ago
Really interesting to think about how these guys model out reliability estimates and come up with such crazy numbers. Thanks for the share.
acdha 10y ago
> The reality is, for things like video and pictures it probably won't be a problem. However, if the data is sensitive, you should be using a raid 6 with 2 disk redundancy, coupled with constant data scrubbing to ensure that the disks remain bit rot free. The probability of bit rot occurring at the same place is probably less than 1 in 10^16, possibly even less if data scrubbing is constant. The first assertion is simply wrong: the internet is littered with people who have corrupt media files which cause user-visible errors. Beyond that, however, RAID can be part of a strategy but is far from sufficient. The paper you linked provides a good discussion of the reasons why but the root mistake is assuming that failures are perfectly distributed random events at a single point. End-to-end strong integrity checks are so important because e.g. in that scenario with a two-drive RAID array you'd still be subject to errors caused transferring the data from the host to the RAID controller, on the controller itself, in communication to the drives (one reason why it's important to mix vendors & models), and on the path back when reading data. You'd also have to worry about corruption in memory, errors caused by e.g. power fluctuation or other hardware events which may be grossly underestimated because they're silently detected and retried until you get lucky and one of the 100,000 errors happened to generate a valid checksum, etc. Generating a cryptographic hash as early as possible and checking it at every stage is key both because it catches almost all of those scenarios but also because it allows you to confidently report a positive validation result rather than just the absence of detected errors.
- rubatuga 10y ago
  I agree with most of your points, and I agree that end-to-end integrity is important where information is critical. But my first assertion is not wrong. You are assuming that the flip of a single bit from a photo or video is catastrophic, both in the rendering of the file, as well as the resulting harm done to the user/individual. The reality is that most people don't value >99.999999% accuracy, and are fine with the current state of data storage. Out of 1,000,000 photos, will the user notice/care if 1 jpeg has a 16x16 block that is a different colour?
  - luckydude 10y ago
    For photos, sure, it's not that big of a deal. But not all files are independent like that. A single bit flip in an executable can be a huge deal. I'm sure you can think of other examples.
  - pdog 10y ago
    Flipping a single bit of an encrypted file renders the whole thing useless.
    - rubatuga 10y ago
      No it does not. Errors shouldn't propagate more than a block
      - pdog 10y ago
        Yes it does.
  - acdha 10y ago
    There are other places bits flip which break photo or video decoding outright. More importantly, however, is that this isn't a reliably random process but often a highly-correlated one. Many users won't have a problem but the people who do often have many files affected. I've seen multiple cases where that wasn't noticed until after their backup software, cloud service, etc. had copied the corrupted bits, not to mention cases where data was corrupted on a RAID array which wasn't regularly scrubbed and so the corruption wasn't noticed until the other disk failed, long after the invalid data had been written to tape as well.
Tanegashima 10y ago
People keep forgetting that this FS was designed by Apple for use in their appliances. Apple devices, save for their least sold model (MacPro), in their least sold product line (Mac) are consumer devices with no ECC RAM. Most people don't know what ECC RAM is, most don't care, and won't pay more for it. For Apple, it doesn't make sense to switch to ECC for system memory, however, if they implement storage ECC at block level in their SSD controller (Apple designes their own SSD controllers), they can obtain the same functionality at much lower cost (only flash controller RAM needs to be ECC).
- scott_karana 10y ago
  > Apple designes their own SSD controllers Huh! I assumed they used off the shelf stuff. Do you have any articles about this? :)
  - mikhailt 10y ago
    The first time I've heard of this was when Anandtech did their review of iPhone 7 and it has the Apple's custom SSD controller: http://www.anandtech.com/show/10685/the-iphone-7-and-iphone-7-plus-review/4 http://www.anandtech.com/show/10685/the-iphone-7-and-iphone-... They mentioned the '15 MacBook was the first to use Apple's custom SSD controller, which iFixit took a picture of here: https://www.ifixit.com/Teardown/Retina+Macbook+2015+Teardown/39841#s90315 https://www.ifixit.com/Teardown/Retina+Macbook+2015+Teardown... They most likely started off from Apple's purchase of Anobit, the custom flash controller designer, back in 2011: https://en.wikipedia.org/wiki/Anobit https://en.wikipedia.org/wiki/Anobit
- comex 10y ago
  Except that not all storage devices APFS will be used with have those SSD controllers. If nothing else, Apple still sells Macs with hard drives (not sure if all the SSDs they sell have their controllers). There's also external disks, which Apple will presumably support formatting as APFS.
  - Tanegashima 10y ago
cmurf 10y ago
Parity raid does not protect against bit rot.
jonkratz 10y ago
Also interesting is Facebook's handling of cold storage (2015). From their blog post: > The easiest way to protect against hardware failure is to keep multiple copies of the data in different hardware failure domains. While that does work, it is fairly resource-intensive, and we knew we could do better. We wondered, “Can we store fewer than two copies of the same data and still protect against loss?” https://code.facebook.com/posts/1433093613662262/-under-the-hood-facebook-s-cold-storage-system-/ https://code.facebook.com/posts/1433093613662262/-under-the-...
thesz 10y ago
You should have redundancy even for data in memory, if data is that sensitive.

roymurdock 10y ago

Really interesting to think about how these guys model out reliability estimates and come up with such crazy numbers. Thanks for the share.

acdha 10y ago

> The reality is, for things like video and pictures it probably won't be a problem. However, if the data is sensitive, you should be using a raid 6 with 2 disk redundancy, coupled with constant data scrubbing to ensure that the disks remain bit rot free. The probability of bit rot occurring at the same place is probably less than 1 in 10^16, possibly even less if data scrubbing is constant. The first assertion is simply wrong: the internet is littered with people who have corrupt media files which cause user-visible errors. Beyond that, however, RAID can be part of a strategy but is far from sufficient. The paper you linked provides a good discussion of the reasons why but the root mistake is assuming that failures are perfectly distributed random events at a single point. End-to-end strong integrity checks are so important because e.g. in that scenario with a two-drive RAID array you'd still be subject to errors caused transferring the data from the host to the RAID controller, on the controller itself, in communication to the drives (one reason why it's important to mix vendors & models), and on the path back when reading data. You'd also have to worry about corruption in memory, errors caused by e.g. power fluctuation or other hardware events which may be grossly underestimated because they're silently detected and retried until you get lucky and one of the 100,000 errors happened to generate a valid checksum, etc. Generating a cryptographic hash as early as possible and checking it at every stage is key both because it catches almost all of those scenarios but also because it allows you to confidently report a positive validation result rather than just the absence of detected errors.

rubatuga 10y ago
I agree with most of your points, and I agree that end-to-end integrity is important where information is critical. But my first assertion is not wrong. You are assuming that the flip of a single bit from a photo or video is catastrophic, both in the rendering of the file, as well as the resulting harm done to the user/individual. The reality is that most people don't value >99.999999% accuracy, and are fine with the current state of data storage. Out of 1,000,000 photos, will the user notice/care if 1 jpeg has a 16x16 block that is a different colour?
- luckydude 10y ago
  For photos, sure, it's not that big of a deal. But not all files are independent like that. A single bit flip in an executable can be a huge deal. I'm sure you can think of other examples.
- pdog 10y ago
  Flipping a single bit of an encrypted file renders the whole thing useless.
  - rubatuga 10y ago
    No it does not. Errors shouldn't propagate more than a block
    - pdog 10y ago
      Yes it does.
- acdha 10y ago
  There are other places bits flip which break photo or video decoding outright. More importantly, however, is that this isn't a reliably random process but often a highly-correlated one. Many users won't have a problem but the people who do often have many files affected. I've seen multiple cases where that wasn't noticed until after their backup software, cloud service, etc. had copied the corrupted bits, not to mention cases where data was corrupted on a RAID array which wasn't regularly scrubbed and so the corruption wasn't noticed until the other disk failed, long after the invalid data had been written to tape as well.

Tanegashima 10y ago

People keep forgetting that this FS was designed by Apple for use in their appliances. Apple devices, save for their least sold model (MacPro), in their least sold product line (Mac) are consumer devices with no ECC RAM. Most people don't know what ECC RAM is, most don't care, and won't pay more for it. For Apple, it doesn't make sense to switch to ECC for system memory, however, if they implement storage ECC at block level in their SSD controller (Apple designes their own SSD controllers), they can obtain the same functionality at much lower cost (only flash controller RAM needs to be ECC).

scott_karana 10y ago
> Apple designes their own SSD controllers Huh! I assumed they used off the shelf stuff. Do you have any articles about this? :)
- mikhailt 10y ago
  The first time I've heard of this was when Anandtech did their review of iPhone 7 and it has the Apple's custom SSD controller: http://www.anandtech.com/show/10685/the-iphone-7-and-iphone-7-plus-review/4 http://www.anandtech.com/show/10685/the-iphone-7-and-iphone-... They mentioned the '15 MacBook was the first to use Apple's custom SSD controller, which iFixit took a picture of here: https://www.ifixit.com/Teardown/Retina+Macbook+2015+Teardown/39841#s90315 https://www.ifixit.com/Teardown/Retina+Macbook+2015+Teardown... They most likely started off from Apple's purchase of Anobit, the custom flash controller designer, back in 2011: https://en.wikipedia.org/wiki/Anobit https://en.wikipedia.org/wiki/Anobit
comex 10y ago
Except that not all storage devices APFS will be used with have those SSD controllers. If nothing else, Apple still sells Macs with hard drives (not sure if all the SSDs they sell have their controllers). There's also external disks, which Apple will presumably support formatting as APFS.
- Tanegashima 10y ago

cmurf 10y ago

Parity raid does not protect against bit rot.

jonkratz 10y ago

Also interesting is Facebook's handling of cold storage (2015). From their blog post: > The easiest way to protect against hardware failure is to keep multiple copies of the data in different hardware failure domains. While that does work, it is fairly resource-intensive, and we knew we could do better. We wondered, “Can we store fewer than two copies of the same data and still protect against loss?” https://code.facebook.com/posts/1433093613662262/-under-the-hood-facebook-s-cold-storage-system-/ https://code.facebook.com/posts/1433093613662262/-under-the-...

thesz 10y ago

You should have redundancy even for data in memory, if data is that sensitive.