5 ms·

When people complain about these random OS crashes and freezes it's usually RAM corruption at fault. Per 1gb of RAM, you can expect to see 266 bit errors per m

When people complain about these random OS crashes and freezes it's usually RAM corruption at fault. Per 1gb of RAM, you can expect to see 266 bit errors per month[1-2] if you are using your PC 16h per day. Multiply that by 64GB or 128GB of RAM and it's crazy to think that you won't run into any of the stability issues. [1-2] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf https://static.googleusercontent.com/media/research.google.c... [1-2] https://en.wikipedia.org/wiki/ECC_memory#Research https://en.wikipedia.org/wiki/ECC_memory#Research

ubercow13 4y ago
>266 bit errors per month That seems like an overestimate. How can memtest ever pass on non-ECC RAM if errors are that frequent?
- moloch-hai 4y ago
  Because memtest only looks at values it wrote out very recently, before they have had a chance to flip. Memtest is looking for reliable failures, not evanescent one-off events.
  - deleted 4y ago
    [deleted]
  - willis936 4y ago
    SDRAM is continuously refreshing all cells. How long ago data was written doesn't make a big difference (aside from the case where you're reading data immediately after writing or reading that data).
    - moloch-hai 4y ago
      This does not, of course, make any sense. Any given memory cell will be read once per, say, millisecond, and the value written back. Once it has flipped once, the wrong value will then be written back, and after that the wrong value is read back out and rewritten again, indefinitely. Errors are sticky, and accumulate. (Flipping back again is negligible unlikely.) With ECC in the refresh path, such an error could be corrected and the right value would be overwritten over top of the bad one. Then errors would not accumulate, but would instead be "scrubbed". Mainframe machines scrub their RAM. Disks too.
    - ilyt 4y ago
      > SDRAM is continuously refreshing all cells ...so ? we're not talking about slow deterioration (which is why refresh is needed), we're talking about bit flip from cosmic rays where cell changes state completely.
toast0 4y ago
I don't think this rate is reasonable to use in this manner. I've run thousands of servers, with lots of ECC ram (quite a few servers had 768GB of ram, but most were more like 32 or 64). The vast majority ran for years with zero reported errors. A small handful would have major errors of thousands in an hour, but we would replace for hundreds in an hour. A couple servers developed a periodic report of one or two (correctable) errors per day. If 266 bit errors per month per 1Gb was a usable rate, all of our servers would have been throwing ECC correctable errors all the time. But I didn't have time or desire to publish a study on our experience, so there's no hard numbers.
pflanze 4y ago
> Per 1gb of RAM The Google study you linked says "25,000 to 70,000 errors per billion device hours per Mbit". Assuming bits, {25000 to 70000}/1e9 * 3016 1000 = 12 to 33.6 bit errors per month. Assuming bytes, it's 96 to 268 bit errors per month. Apparently you've meant bytes, not bits. (I'm a pedant, but was also just unsure and interested in the numbers.) FWIW, a comment I ran across that confirms toast0's view[1]: "It's a bimodal distribution - you either have many errors (due to a defect somewhere) or basically zero. If you're on the good side of the distribution, with only extremely rare errors, then you probably don't need ECC. But without ECC, you don't know whether you need ECC!" [1] https://www.realworldtech.com/forum/?threadid=198497&curpostid=198724 https://www.realworldtech.com/forum/?threadid=198497&curpost...

ubercow13 4y ago

>266 bit errors per month That seems like an overestimate. How can memtest ever pass on non-ECC RAM if errors are that frequent?

moloch-hai 4y ago
Because memtest only looks at values it wrote out very recently, before they have had a chance to flip. Memtest is looking for reliable failures, not evanescent one-off events.
- deleted 4y ago
  [deleted]
- willis936 4y ago
  SDRAM is continuously refreshing all cells. How long ago data was written doesn't make a big difference (aside from the case where you're reading data immediately after writing or reading that data).
  - moloch-hai 4y ago
    This does not, of course, make any sense. Any given memory cell will be read once per, say, millisecond, and the value written back. Once it has flipped once, the wrong value will then be written back, and after that the wrong value is read back out and rewritten again, indefinitely. Errors are sticky, and accumulate. (Flipping back again is negligible unlikely.) With ECC in the refresh path, such an error could be corrected and the right value would be overwritten over top of the bad one. Then errors would not accumulate, but would instead be "scrubbed". Mainframe machines scrub their RAM. Disks too.
  - ilyt 4y ago
    > SDRAM is continuously refreshing all cells ...so ? we're not talking about slow deterioration (which is why refresh is needed), we're talking about bit flip from cosmic rays where cell changes state completely.

toast0 4y ago

I don't think this rate is reasonable to use in this manner. I've run thousands of servers, with lots of ECC ram (quite a few servers had 768GB of ram, but most were more like 32 or 64). The vast majority ran for years with zero reported errors. A small handful would have major errors of thousands in an hour, but we would replace for hundreds in an hour. A couple servers developed a periodic report of one or two (correctable) errors per day. If 266 bit errors per month per 1Gb was a usable rate, all of our servers would have been throwing ECC correctable errors all the time. But I didn't have time or desire to publish a study on our experience, so there's no hard numbers.

pflanze 4y ago

> Per 1gb of RAM The Google study you linked says "25,000 to 70,000 errors per billion device hours per Mbit". Assuming bits, {25000 to 70000}/1e9 * 3016 1000 = 12 to 33.6 bit errors per month. Assuming bytes, it's 96 to 268 bit errors per month. Apparently you've meant bytes, not bits. (I'm a pedant, but was also just unsure and interested in the numbers.) FWIW, a comment I ran across that confirms toast0's view[1]: "It's a bimodal distribution - you either have many errors (due to a defect somewhere) or basically zero. If you're on the good side of the distribution, with only extremely rare errors, then you probably don't need ECC. But without ECC, you don't know whether you need ECC!" [1] https://www.realworldtech.com/forum/?threadid=198497&curpostid=198724 https://www.realworldtech.com/forum/?threadid=198497&curpost...