4 ms·

Any work we do to improve the reliability of consumer software is limited by the reliability of the underlying machine. If everyone used ECC RAM, it would be m

by wyager 10y ago

Any work we do to improve the reliability of consumer software is limited by the reliability of the underlying machine. If everyone used ECC RAM, it would be marginally more expensive than regular RAM, but it would let us (theoretically) cut the number of real-life crashes by a huge degree. I think if every computer system were like this, it would have non-obvious positive effects on society (like the ability to safely give computers increased responsibility).

semi-extrinsic 10y ago
> it would let us (theoretically) cut the number of real-life crashes by a huge degree Do you have a source for that claim? I know people have done tests with running large molecular simulations on hundreds of GPUs with/without ECC for days on end, and who've come to the conclusions that ECC probably isn't worth it [1]. And this makes sense. Most of a program's memory usage is data. Thus the rare event of a bitflip is most likely to occur in data. Such a bitflip doesn't cause a crash in a consumer system; maybe it changes the color of one pixel in one frame of a video file, or adds some other tiny noise somewhere. [1] http://dl.acm.org/citation.cfm?id=2484774 http://dl.acm.org/citation.cfm?id=2484774
- wyager 10y ago
  http://research.google.com/pubs/pub35162.html http://research.google.com/pubs/pub35162.html
  - semi-extrinsic 10y ago
    Interesting, thanks. From what I can tell, the probability of one bit flip per year is about 25% for a machine that's always on and that's under constant load. Reducing the load about halves the probability. If we consider then a work laptop, average life of three years, used 8-5 five days a week, you get a probability of ~10% for a single bitflip happening during the machine's lifetime. The question is then: would reducing one crash per lifetime of the machine on every ten machines be a huge improvement?
    - nkurz 10y ago
      I'm not sure if I'm interpreting the paper correctly, but I don't think those numbers quite match what the authors found. I find their presentation confusing, though, so possibly I'm the one that's wrong. I think you understand this but in case it's not clear to others (I wasn't sure until I got to the end of the paper) all of the machines in their study are using ECC memory. This allows them to measure both the number of Correctable Errors (single bit) and Uncorrectable Errors (more than single). I think you are right that the chances of a single bit flip for a randomly chosen machine are about 25%. In Table 1, they say CE incidence for a randomly chosen machine is 32% per year. But they also show that a machine that has one correctable memory error is much more likely to exhibit many more. I think that "CE Median Affct" (also in Table 1) says that among the machines that have been affected by at least one correctable error, the mean number of errors is 277. Thus rather than one crash per 10 machines over 3 years, it's likely that you'll get hundreds of crashes, but with one (or zero, or two) of those 10 machines responsible for all of them. The Uncorrectable Error rate they found was about 1/20 of the correctable rate, a little over 1% per year. Assuming the same "use time" factor applies, this would mean that with ECC none of the 10 machines would be likely to ever crash due to bit flips, reducing the number of crashes by hundreds. But this doesn't tell the full story either. Maybe it's best to view it as 1000 employees, each of whom is issued a laptop. If you buy them machines without ECC, the initial cost is less but 100 of them will experience frequent crashes during the lifetime of the laptop, leading to frustration, possible lost work, and early machine replacement. If you pay more up front for ECC, the number of affected employees will be reduced from 100 to 5. How much extra one should be willing to pay for this reduction depends on circumstances, but I think the impact is quite different than merely avoiding 100 equally distributed random crashes over the course of 3 years.
      - semi-extrinsic 10y ago
        I agree this point is confusing. However, I think the "10% of employees will experience frequent crashes and possible data loss" sounds way too extreme. I mean, how many companies do you know that give all their workers Xeon-powered laptops? My impression is that 99% of people have i3/5/7 powered machines at work. Are really companies experiencing frequent failures caused by RAM on 10% of their employee's machines? Can anyone with such knowledge weigh in?

by wyager 10y ago

semi-extrinsic 10y ago

> it would let us (theoretically) cut the number of real-life crashes by a huge degree Do you have a source for that claim? I know people have done tests with running large molecular simulations on hundreds of GPUs with/without ECC for days on end, and who've come to the conclusions that ECC probably isn't worth it [1]. And this makes sense. Most of a program's memory usage is data. Thus the rare event of a bitflip is most likely to occur in data. Such a bitflip doesn't cause a crash in a consumer system; maybe it changes the color of one pixel in one frame of a video file, or adds some other tiny noise somewhere. [1] http://dl.acm.org/citation.cfm?id=2484774 http://dl.acm.org/citation.cfm?id=2484774

wyager 10y ago
http://research.google.com/pubs/pub35162.html http://research.google.com/pubs/pub35162.html
- semi-extrinsic 10y ago
  Interesting, thanks. From what I can tell, the probability of one bit flip per year is about 25% for a machine that's always on and that's under constant load. Reducing the load about halves the probability. If we consider then a work laptop, average life of three years, used 8-5 five days a week, you get a probability of ~10% for a single bitflip happening during the machine's lifetime. The question is then: would reducing one crash per lifetime of the machine on every ten machines be a huge improvement?
  - nkurz 10y ago
    I'm not sure if I'm interpreting the paper correctly, but I don't think those numbers quite match what the authors found. I find their presentation confusing, though, so possibly I'm the one that's wrong. I think you understand this but in case it's not clear to others (I wasn't sure until I got to the end of the paper) all of the machines in their study are using ECC memory. This allows them to measure both the number of Correctable Errors (single bit) and Uncorrectable Errors (more than single). I think you are right that the chances of a single bit flip for a randomly chosen machine are about 25%. In Table 1, they say CE incidence for a randomly chosen machine is 32% per year. But they also show that a machine that has one correctable memory error is much more likely to exhibit many more. I think that "CE Median Affct" (also in Table 1) says that among the machines that have been affected by at least one correctable error, the mean number of errors is 277. Thus rather than one crash per 10 machines over 3 years, it's likely that you'll get hundreds of crashes, but with one (or zero, or two) of those 10 machines responsible for all of them. The Uncorrectable Error rate they found was about 1/20 of the correctable rate, a little over 1% per year. Assuming the same "use time" factor applies, this would mean that with ECC none of the 10 machines would be likely to ever crash due to bit flips, reducing the number of crashes by hundreds. But this doesn't tell the full story either. Maybe it's best to view it as 1000 employees, each of whom is issued a laptop. If you buy them machines without ECC, the initial cost is less but 100 of them will experience frequent crashes during the lifetime of the laptop, leading to frustration, possible lost work, and early machine replacement. If you pay more up front for ECC, the number of affected employees will be reduced from 100 to 5. How much extra one should be willing to pay for this reduction depends on circumstances, but I think the impact is quite different than merely avoiding 100 equally distributed random crashes over the course of 3 years.
    - semi-extrinsic 10y ago
      I agree this point is confusing. However, I think the "10% of employees will experience frequent crashes and possible data loss" sounds way too extreme. I mean, how many companies do you know that give all their workers Xeon-powered laptops? My impression is that 99% of people have i3/5/7 powered machines at work. Are really companies experiencing frequent failures caused by RAM on 10% of their employee's machines? Can anyone with such knowledge weigh in?