5 ms·

I don't know if ECC is that important, but reliability of RAM (or any storage) feels pretty crazy to me. 128GB being refreshed every second for a month error re

by phh 6y ago

I don't know if ECC is that important, but reliability of RAM (or any storage) feels pretty crazy to me. 128GB being refreshed every second for a month error requires that the per-bit refresh process has a reliability of 99.9999999999999999% to be flawless. Considering we are dealing with quantum effects (which are inherently probabilistic), I wouldn't trust myself to design anything like that. Now back to ECC, I'll probably be corrected, but I don't think ECC helps gain more than two order of magnitudes, so we still need incredibly reliable RAM. If we move to ECC RAM by default everywhere, aren't we simply going to get less reliable RAM at the end?

bitcharmer 6y ago
A system on Earth, at sea level, with 4 GB of RAM has a 96% percent chance of having a bit error in three days without ECC RAM. With ECC RAM, that goes down to 1.67e-10 or about one chance in six billions. So I'd say ECC is not only important but insanely impactful. There's a reason why many organizations don't even want to hear about getting rigs with non-ECC memory.
- tomxor 6y ago
  I like when people back up their claims with numbers, but would you mind describing roughly what that 96% probability of error is based upon? I understand altitude has some kind of proportionality to cosmic ray exposure, and number of bits will multiply the probability of an error.. I'm presuming there is also an inherent error rate to DRAM separate from environment. But what are those numbers.
  - bitcharmer 6y ago
    Apologies, you're totally right. I should have linked to the source: http://lambda-diode.com/opinion/ecc-memory#:~:text=A%20system%20on%20Earth%2C%20at,one%20chance%20in%20six%20billions http://lambda-diode.com/opinion/ecc-memory#:~:text=A%20syste....
    - tomxor 6y ago
      Great thanks! [edit] Looks like the calculation was revised [0] after criticism: > Under these assumptions, you'll have to wait about 33 to 600 days to get a 96% chance of getting a bit error. What's more worrying is the variance, the above calculation is based on expected well behaved DRAM.. yet some computers just seem to have manufacturing defects that make the incidence of errors high enough to be a regular problem. [0] http://lambda-diode.com/opinion/ecc-memory-2 http://lambda-diode.com/opinion/ecc-memory-2
- dejj 6y ago
  And even higher in the vicinity of radioactive cattle: https://www.jakepoz.com/debugging-behind-the-iron-curtain/ https://www.jakepoz.com/debugging-behind-the-iron-curtain/
- davidw 6y ago
  Could you measure altitude with memory?
  - cyberlurker 6y ago
    If the 96% every 3 days is true, you could approximate based on that. But it would be a really slow measurement.
  - tomxor 6y ago
    :D yes, although I expect you would need either a prohibitively large quantity of memory or a extremely slow rate of change in altitude to effectively measure it.
  - asimpletune 6y ago
    That’s a very interesting idea, and I think you totally could. You run some benchmarks, measure the bit flips, and after enough runs you’d be able to say with a degree of confidence what your altitude is. I wonder though what accuracy could be achieved with this?
- johndough 6y ago
  I ran a memory test for two weeks straight on a consumer laptop with 8 GB RAM and could not get a single bit flip, so your mileage may vary.
  - bitcharmer 6y ago
    How did you run those tests? From what I understand on the topic, for your results to be statistically significant you need at least hundreds of machines and very rigid testing methodology.
    - avian 6y ago
      As someone who also ran a similar test myself and haven't seen a bit flip, I'm also skeptical of the 96% figure. I'm too lazy to run the exact numbers right now, but with "4 GB, 96% percent chance, three days" as the hypothesis, I think you'll find that an experimental result of "8 GB, 0% chance, 14 days" is highly statistically significant. Edit: rough back of napkin estimate - you're not seeing an event in roughly 10x trials (2x number of bits and ~5x number of days). Given hypothesis is true your experimental result has a probability of (1-0.96)^10 = very very small. Conclusion: hypothesis is false.
      - bitcharmer 6y ago
        The 96% figure comes from Google and was obtained in a large scale experiment over many months. I've been in this business long enough to have witnessed adverse effects of cosmic rays an non-ECC memory multiple times myself. I don't think you're sample gets anywhere near statistical significance. Not mentioning testing methodology.
        toast0 6y ago
        My anecdotal evidence is far from rigorous, but the Google data from ten years ago doesn't match up with my experience running thousands of ECC enabled servers up to a few years ago. Their rates seem a lot higher than what my servers experienced; we would page on any ram errors, correctable or not (uncorrectable would halt the machine, so we would have to inspect the console to confirm; when we knowingly tried machines with uncorrectable errors after a halt, they nearly all failed again within 24 hours, so those we didn't inspect the console of probably were counted on their second failure), and while there were pages from time to time, it felt like a lot less than 8% of the machine having a There's a lot of variables that go into RAM errors, including manufacturing quality and condition of the ram, the dimm, the dimm slot, the motherboard generally, the power supply, the wiring, and the temperature of all of those. Google was known for cost cutting in their servers, especially early on; so I wouldn't be surprised if some of that resulted in higher bitflip rate than running in commercially available servers. Things like running bare motherboards, supported only on the edges cause excess strain and can impact resistance and capacitance of traces on the board (and in extreme cases, break the traces).
- mrlala 6y ago
  So, I hear what you are saying. But, on the other hand, I have been using 2 non-ECC desktops for a workstation/server for the past ~6 years.. and I would be hard pressed to come up with a single situation where either of the machines randomly crashed or applications did anything 'unexpected' (to my knowledge, of course). My point is, when you say there is a "96% chance of having an error in THREE DAYS", one would EXPECT to be having issues like.. all the time? So I'm not disagreeing with you, but with the amount of non-ECC machines all over the world and how insanely stable modern machines are, it still seems like a very low risk. Now of course I agree that if you want to take every precaution, go ECC, but simple observation prove that this "problem" can't be as bad as the numbers are saying.
  - bitcharmer 6y ago
    Your questions are perfectly valid. It's just that out of all the random bit flips that happen over a period of time on a non-ECC platform only a miniscule percentage will manifest to you in any noticeable way. Most will escape your attention.
- gzalo 6y ago
  That number is flawed, and the author did a follow-up with better results: http://lambda-diode.com/opinion/ecc-memory-2 http://lambda-diode.com/opinion/ecc-memory-2 "33 to 600 days to get a 96% chance of getting a bit error." Still, it seems way too high. I guess anyone with ECC RAM could confirm that they are getting those sort of recovered error rates?
formerly_proven 6y ago
RAM is not as reliable as you think. Some ECC memory hardly ever finds an error, some machines see them at a very consistent rate, e.g. 50 errors per TB-day. That would translate to 1-2 errors per day in a 32 GB PC. Without ECC you cannot know in which bucket you are.
- trevyn 6y ago
  If true, that seems like... a very straightforward bucket to test if you’re in.
  - toast0 6y ago
    The bucket can change over time though. If you want to be sure, you need to test often, which gets in the way of using the computer.

by phh 6y ago

bitcharmer 6y ago

A system on Earth, at sea level, with 4 GB of RAM has a 96% percent chance of having a bit error in three days without ECC RAM. With ECC RAM, that goes down to 1.67e-10 or about one chance in six billions. So I'd say ECC is not only important but insanely impactful. There's a reason why many organizations don't even want to hear about getting rigs with non-ECC memory.

tomxor 6y ago
I like when people back up their claims with numbers, but would you mind describing roughly what that 96% probability of error is based upon? I understand altitude has some kind of proportionality to cosmic ray exposure, and number of bits will multiply the probability of an error.. I'm presuming there is also an inherent error rate to DRAM separate from environment. But what are those numbers.
- bitcharmer 6y ago
  Apologies, you're totally right. I should have linked to the source: http://lambda-diode.com/opinion/ecc-memory#:~:text=A%20system%20on%20Earth%2C%20at,one%20chance%20in%20six%20billions http://lambda-diode.com/opinion/ecc-memory#:~:text=A%20syste....
  - tomxor 6y ago
    Great thanks! [edit] Looks like the calculation was revised [0] after criticism: > Under these assumptions, you'll have to wait about 33 to 600 days to get a 96% chance of getting a bit error. What's more worrying is the variance, the above calculation is based on expected well behaved DRAM.. yet some computers just seem to have manufacturing defects that make the incidence of errors high enough to be a regular problem. [0] http://lambda-diode.com/opinion/ecc-memory-2 http://lambda-diode.com/opinion/ecc-memory-2
dejj 6y ago
And even higher in the vicinity of radioactive cattle: https://www.jakepoz.com/debugging-behind-the-iron-curtain/ https://www.jakepoz.com/debugging-behind-the-iron-curtain/
davidw 6y ago
Could you measure altitude with memory?
- cyberlurker 6y ago
  If the 96% every 3 days is true, you could approximate based on that. But it would be a really slow measurement.
- tomxor 6y ago
  :D yes, although I expect you would need either a prohibitively large quantity of memory or a extremely slow rate of change in altitude to effectively measure it.
- asimpletune 6y ago
  That’s a very interesting idea, and I think you totally could. You run some benchmarks, measure the bit flips, and after enough runs you’d be able to say with a degree of confidence what your altitude is. I wonder though what accuracy could be achieved with this?
johndough 6y ago
I ran a memory test for two weeks straight on a consumer laptop with 8 GB RAM and could not get a single bit flip, so your mileage may vary.
- bitcharmer 6y ago
  How did you run those tests? From what I understand on the topic, for your results to be statistically significant you need at least hundreds of machines and very rigid testing methodology.
  - avian 6y ago
    As someone who also ran a similar test myself and haven't seen a bit flip, I'm also skeptical of the 96% figure. I'm too lazy to run the exact numbers right now, but with "4 GB, 96% percent chance, three days" as the hypothesis, I think you'll find that an experimental result of "8 GB, 0% chance, 14 days" is highly statistically significant. Edit: rough back of napkin estimate - you're not seeing an event in roughly 10x trials (2x number of bits and ~5x number of days). Given hypothesis is true your experimental result has a probability of (1-0.96)^10 = very very small. Conclusion: hypothesis is false.
    - bitcharmer 6y ago
      The 96% figure comes from Google and was obtained in a large scale experiment over many months. I've been in this business long enough to have witnessed adverse effects of cosmic rays an non-ECC memory multiple times myself. I don't think you're sample gets anywhere near statistical significance. Not mentioning testing methodology.
      - toast0 6y ago
        My anecdotal evidence is far from rigorous, but the Google data from ten years ago doesn't match up with my experience running thousands of ECC enabled servers up to a few years ago. Their rates seem a lot higher than what my servers experienced; we would page on any ram errors, correctable or not (uncorrectable would halt the machine, so we would have to inspect the console to confirm; when we knowingly tried machines with uncorrectable errors after a halt, they nearly all failed again within 24 hours, so those we didn't inspect the console of probably were counted on their second failure), and while there were pages from time to time, it felt like a lot less than 8% of the machine having a There's a lot of variables that go into RAM errors, including manufacturing quality and condition of the ram, the dimm, the dimm slot, the motherboard generally, the power supply, the wiring, and the temperature of all of those. Google was known for cost cutting in their servers, especially early on; so I wouldn't be surprised if some of that resulted in higher bitflip rate than running in commercially available servers. Things like running bare motherboards, supported only on the edges cause excess strain and can impact resistance and capacitance of traces on the board (and in extreme cases, break the traces).
mrlala 6y ago
So, I hear what you are saying. But, on the other hand, I have been using 2 non-ECC desktops for a workstation/server for the past ~6 years.. and I would be hard pressed to come up with a single situation where either of the machines randomly crashed or applications did anything 'unexpected' (to my knowledge, of course). My point is, when you say there is a "96% chance of having an error in THREE DAYS", one would EXPECT to be having issues like.. all the time? So I'm not disagreeing with you, but with the amount of non-ECC machines all over the world and how insanely stable modern machines are, it still seems like a very low risk. Now of course I agree that if you want to take every precaution, go ECC, but simple observation prove that this "problem" can't be as bad as the numbers are saying.
- bitcharmer 6y ago
  Your questions are perfectly valid. It's just that out of all the random bit flips that happen over a period of time on a non-ECC platform only a miniscule percentage will manifest to you in any noticeable way. Most will escape your attention.
gzalo 6y ago
That number is flawed, and the author did a follow-up with better results: http://lambda-diode.com/opinion/ecc-memory-2 http://lambda-diode.com/opinion/ecc-memory-2 "33 to 600 days to get a 96% chance of getting a bit error." Still, it seems way too high. I guess anyone with ECC RAM could confirm that they are getting those sort of recovered error rates?

formerly_proven 6y ago

RAM is not as reliable as you think. Some ECC memory hardly ever finds an error, some machines see them at a very consistent rate, e.g. 50 errors per TB-day. That would translate to 1-2 errors per day in a 32 GB PC. Without ECC you cannot know in which bucket you are.

trevyn 6y ago
If true, that seems like... a very straightforward bucket to test if you’re in.
- toast0 6y ago
  The bucket can change over time though. If you want to be sure, you need to test often, which gets in the way of using the computer.