5 ms·

> Building machines without ECC is a bad idea. I'd say it depends on the use case. For business storage servers, especially "mission-critical", sure. (Then aga

> Building machines without ECC is a bad idea. I'd say it depends on the use case. For business storage servers, especially "mission-critical", sure. (Then again, this data was created somewhere; presumably mainly on laptops without ECC. Why is that not a bad idea?) But for a home storage server, ECC is no more critical than buying a UPS, shielded SATA cables, enterprise-grade drives, a high quality power supply, etc. etc. Together these upgrades will easily double the cost of the storage server. Since you'll want an off-site backup solution anyway in case of disaster, why bother paying for an expensive server that only reduces the chance of needing to restore from backup from 0.001% to 0.00001%?

wyager 10y ago
Any work we do to improve the reliability of consumer software is limited by the reliability of the underlying machine. If everyone used ECC RAM, it would be marginally more expensive than regular RAM, but it would let us (theoretically) cut the number of real-life crashes by a huge degree. I think if every computer system were like this, it would have non-obvious positive effects on society (like the ability to safely give computers increased responsibility).
- semi-extrinsic 10y ago
  > it would let us (theoretically) cut the number of real-life crashes by a huge degree Do you have a source for that claim? I know people have done tests with running large molecular simulations on hundreds of GPUs with/without ECC for days on end, and who've come to the conclusions that ECC probably isn't worth it [1]. And this makes sense. Most of a program's memory usage is data. Thus the rare event of a bitflip is most likely to occur in data. Such a bitflip doesn't cause a crash in a consumer system; maybe it changes the color of one pixel in one frame of a video file, or adds some other tiny noise somewhere. [1] http://dl.acm.org/citation.cfm?id=2484774 http://dl.acm.org/citation.cfm?id=2484774
  - wyager 10y ago
    http://research.google.com/pubs/pub35162.html http://research.google.com/pubs/pub35162.html
    - semi-extrinsic 10y ago
      Interesting, thanks. From what I can tell, the probability of one bit flip per year is about 25% for a machine that's always on and that's under constant load. Reducing the load about halves the probability. If we consider then a work laptop, average life of three years, used 8-5 five days a week, you get a probability of ~10% for a single bitflip happening during the machine's lifetime. The question is then: would reducing one crash per lifetime of the machine on every ten machines be a huge improvement?
      - nkurz 10y ago
        I'm not sure if I'm interpreting the paper correctly, but I don't think those numbers quite match what the authors found. I find their presentation confusing, though, so possibly I'm the one that's wrong. I think you understand this but in case it's not clear to others (I wasn't sure until I got to the end of the paper) all of the machines in their study are using ECC memory. This allows them to measure both the number of Correctable Errors (single bit) and Uncorrectable Errors (more than single). I think you are right that the chances of a single bit flip for a randomly chosen machine are about 25%. In Table 1, they say CE incidence for a randomly chosen machine is 32% per year. But they also show that a machine that has one correctable memory error is much more likely to exhibit many more. I think that "CE Median Affct" (also in Table 1) says that among the machines that have been affected by at least one correctable error, the mean number of errors is 277. Thus rather than one crash per 10 machines over 3 years, it's likely that you'll get hundreds of crashes, but with one (or zero, or two) of those 10 machines responsible for all of them. The Uncorrectable Error rate they found was about 1/20 of the correctable rate, a little over 1% per year. Assuming the same "use time" factor applies, this would mean that with ECC none of the 10 machines would be likely to ever crash due to bit flips, reducing the number of crashes by hundreds. But this doesn't tell the full story either. Maybe it's best to view it as 1000 employees, each of whom is issued a laptop. If you buy them machines without ECC, the initial cost is less but 100 of them will experience frequent crashes during the lifetime of the laptop, leading to frustration, possible lost work, and early machine replacement. If you pay more up front for ECC, the number of affected employees will be reduced from 100 to 5. How much extra one should be willing to pay for this reduction depends on circumstances, but I think the impact is quite different than merely avoiding 100 equally distributed random crashes over the course of 3 years.
erkkie 10y ago
You say based on what information? Zfs in particular, with the safety features (like checksumming) it tries to guarantee, relies on ram not lying to you. http://research.cs.wisc.edu/adsl/Publications/zfs-corruption-fast10.pdf http://research.cs.wisc.edu/adsl/Publications/zfs-corruption...
- jonathankoren 10y ago
  Cost-benefit analysis. These things are relatively expensive. My home NAS basically stores photos, mp3, and some videos. It's backed up offsite. The vast majority of the time, it's in hibernate. It uses ext4, because I didn't get ECC RAM. Why not? I picked the board on price, and didn't realize it until I tried to install FreeNAS[], which uses ZFS, so I switched to OpenMediaVault, and never looked back. What's the worst that's going to happen? Some photo is going to get a single bit error? So what? The data doesn't need super high fidelity. What about a file system crash? Well that's what backups are for. [] I would also like to take time that I found the FreeNAS community completely horrible. They're focused solely on people setting up like HIPPA compliant scientific data, or something. It's ZFS for everything. Backups to USB drives are stupid, and you're stupid for having that as a backup strategy, and instead should drop another $1000 on a second NAS for your house, and then backup across a network. OpenMediaVault's community on the other hand is much more helpful and open to both professional and personal use cases.
  - greedo 10y ago
    What happens if you have a corrupted file (or more) that then gets replicated to your offsite? Are you keeping multiple versions offsite?
    - semi-extrinsic 10y ago
      This of course depends on your data, but for the home user who generally stores photos, videos and music, 99.9% of the changes consist in adding new files. Thus any modifications of an existing file are somewhat suspicious, and keeping also the old version in your offsite backup is a very good idea and essentially won't increase the storage requirement. E.g. with rsync, you use the --backup and --suffix flags.
dap 10y ago
> But for a home storage server, ECC is no more critical than buying a UPS, shielded SATA cables, enterprise-grade drives, a high quality power supply, etc. etc. Together these upgrades will easily double the cost of the storage server. Since you'll want an off-site backup solution anyway in case of disaster, why bother paying for an expensive server that only reduces the chance of needing to restore from backup from 0.001% to 0.00001%? Flipping it around: what's the point of backing up to a remote server if the data you're backing up is itself corrupted? I don't think most of the upgrades you mentioned are really in the same bucket as ECC DRAM because only the ECC DRAM seems likely to prevent data corruption. Failing drives and cables and power loss, even failures that corrupt data, should be survivable events assuming drive redundancy, particularly given a filesystem like ZFS that will detect and repair the corruption.

wyager 10y ago

Any work we do to improve the reliability of consumer software is limited by the reliability of the underlying machine. If everyone used ECC RAM, it would be marginally more expensive than regular RAM, but it would let us (theoretically) cut the number of real-life crashes by a huge degree. I think if every computer system were like this, it would have non-obvious positive effects on society (like the ability to safely give computers increased responsibility).

semi-extrinsic 10y ago
> it would let us (theoretically) cut the number of real-life crashes by a huge degree Do you have a source for that claim? I know people have done tests with running large molecular simulations on hundreds of GPUs with/without ECC for days on end, and who've come to the conclusions that ECC probably isn't worth it [1]. And this makes sense. Most of a program's memory usage is data. Thus the rare event of a bitflip is most likely to occur in data. Such a bitflip doesn't cause a crash in a consumer system; maybe it changes the color of one pixel in one frame of a video file, or adds some other tiny noise somewhere. [1] http://dl.acm.org/citation.cfm?id=2484774 http://dl.acm.org/citation.cfm?id=2484774
- wyager 10y ago
  http://research.google.com/pubs/pub35162.html http://research.google.com/pubs/pub35162.html
  - semi-extrinsic 10y ago
    Interesting, thanks. From what I can tell, the probability of one bit flip per year is about 25% for a machine that's always on and that's under constant load. Reducing the load about halves the probability. If we consider then a work laptop, average life of three years, used 8-5 five days a week, you get a probability of ~10% for a single bitflip happening during the machine's lifetime. The question is then: would reducing one crash per lifetime of the machine on every ten machines be a huge improvement?
    - nkurz 10y ago
      I'm not sure if I'm interpreting the paper correctly, but I don't think those numbers quite match what the authors found. I find their presentation confusing, though, so possibly I'm the one that's wrong. I think you understand this but in case it's not clear to others (I wasn't sure until I got to the end of the paper) all of the machines in their study are using ECC memory. This allows them to measure both the number of Correctable Errors (single bit) and Uncorrectable Errors (more than single). I think you are right that the chances of a single bit flip for a randomly chosen machine are about 25%. In Table 1, they say CE incidence for a randomly chosen machine is 32% per year. But they also show that a machine that has one correctable memory error is much more likely to exhibit many more. I think that "CE Median Affct" (also in Table 1) says that among the machines that have been affected by at least one correctable error, the mean number of errors is 277. Thus rather than one crash per 10 machines over 3 years, it's likely that you'll get hundreds of crashes, but with one (or zero, or two) of those 10 machines responsible for all of them. The Uncorrectable Error rate they found was about 1/20 of the correctable rate, a little over 1% per year. Assuming the same "use time" factor applies, this would mean that with ECC none of the 10 machines would be likely to ever crash due to bit flips, reducing the number of crashes by hundreds. But this doesn't tell the full story either. Maybe it's best to view it as 1000 employees, each of whom is issued a laptop. If you buy them machines without ECC, the initial cost is less but 100 of them will experience frequent crashes during the lifetime of the laptop, leading to frustration, possible lost work, and early machine replacement. If you pay more up front for ECC, the number of affected employees will be reduced from 100 to 5. How much extra one should be willing to pay for this reduction depends on circumstances, but I think the impact is quite different than merely avoiding 100 equally distributed random crashes over the course of 3 years.

erkkie 10y ago

You say based on what information? Zfs in particular, with the safety features (like checksumming) it tries to guarantee, relies on ram not lying to you. http://research.cs.wisc.edu/adsl/Publications/zfs-corruption-fast10.pdf http://research.cs.wisc.edu/adsl/Publications/zfs-corruption...

jonathankoren 10y ago
Cost-benefit analysis. These things are relatively expensive. My home NAS basically stores photos, mp3, and some videos. It's backed up offsite. The vast majority of the time, it's in hibernate. It uses ext4, because I didn't get ECC RAM. Why not? I picked the board on price, and didn't realize it until I tried to install FreeNAS[], which uses ZFS, so I switched to OpenMediaVault, and never looked back. What's the worst that's going to happen? Some photo is going to get a single bit error? So what? The data doesn't need super high fidelity. What about a file system crash? Well that's what backups are for. [] I would also like to take time that I found the FreeNAS community completely horrible. They're focused solely on people setting up like HIPPA compliant scientific data, or something. It's ZFS for everything. Backups to USB drives are stupid, and you're stupid for having that as a backup strategy, and instead should drop another $1000 on a second NAS for your house, and then backup across a network. OpenMediaVault's community on the other hand is much more helpful and open to both professional and personal use cases.
- greedo 10y ago
  What happens if you have a corrupted file (or more) that then gets replicated to your offsite? Are you keeping multiple versions offsite?
  - semi-extrinsic 10y ago
    This of course depends on your data, but for the home user who generally stores photos, videos and music, 99.9% of the changes consist in adding new files. Thus any modifications of an existing file are somewhat suspicious, and keeping also the old version in your offsite backup is a very good idea and essentially won't increase the storage requirement. E.g. with rsync, you use the --backup and --suffix flags.

dap 10y ago

> But for a home storage server, ECC is no more critical than buying a UPS, shielded SATA cables, enterprise-grade drives, a high quality power supply, etc. etc. Together these upgrades will easily double the cost of the storage server. Since you'll want an off-site backup solution anyway in case of disaster, why bother paying for an expensive server that only reduces the chance of needing to restore from backup from 0.001% to 0.00001%? Flipping it around: what's the point of backing up to a remote server if the data you're backing up is itself corrupted? I don't think most of the upgrades you mentioned are really in the same bucket as ECC DRAM because only the ECC DRAM seems likely to prevent data corruption. Failing drives and cables and power loss, even failures that corrupt data, should be survivable events assuming drive redundancy, particularly given a filesystem like ZFS that will detect and repair the corruption.