4 ms·
> Beyond differences in component brands, no obvious reasons for the different behavior have presented themselves to me. Google's results show that the error r
by paulmd 4y ago
> Beyond differences in component brands, no obvious reasons for the different behavior have presented themselves to me.
Google's results show that the error rate is strongly driven by bad modules rather than random errors. A module that throws one error is up to 224x more likely to throw another error, and corrected errors are a strong predictor of future uncorrectable errors.
The major exception is cosmic rays, which can flip bits even in a good module. But unless you're at high altitude they don't really dominate the error rate.
http://www.cs.toronto.edu/%7Ebianca/papers/sigmetrics09.pdf http://www.cs.toronto.edu/%7Ebianca/papers/sigmetrics09.pdf
Obviously without ECC you don't know if you have bad modules, but, if a machine passes a memtest scrub for 24+ hours then it's probably fine.
On-die ECC should also catch the majority of these errors, especially if you are not right at the limits of your memory training. JEDEC type timings should be very forgiving to transmission errors in general.
People are very down on the "on-die ECC" on DDR5, they act like ECC is completely useless unless you protect the whole datapath, and of course it would be preferable to do so. But actually the on-die ECC should catch a large number of the errors, because bad modules (dies) are where the errors mostly happen.
- jml7c5 4y agoMy only objection to on-die ECC is that, AFAIK, there's no way to query the number of internal corrections that have occurred. That is important information! Yes, it might freak out some sysadmins that frequent corrected errors are now "normal" (after all, the main reason for on-die ECC is that manufacturers won't have to scrap slightly imperfect DRAM), but those individuals can be re-educated, and I really want to be able to tell if a given module is running within spec yet right at the edge of failure.
- paulmd 4y agoyep definitely, it'd be nice to have a way to communicate this, I totally agree. And yeah they probably didn't because people would freak at seeing errors corrected during normal operation.
- Scramblejams 4y agoYes, I'd like to see these plotted over time and correlated with machine failures.
- monocasa 4y agoAlso, the author of that paper wasn't aware that Google had a stint around that time buying up DRAM chips that had failed manufacturer's QA and sticking them on DIMMs themselves, trying to clock them slow and use ECC to get cheaper, mostly working memory. It was a poor choice and Google stopped the process AFAIK.
- sliken 4y agoI agree with the "if a machine passes a memtest scrub for 24+ hours then it's probably fine" in theory. Sadly in practice I regularly find dimms that generate errors plenty of errors, and test find with memtest. Not sure why, maybe it's the different memory access patterns, maybe bad dimms are more sensitive to repeated accesses below the rowhammer level, that healthy dimms have no problems with. I find things like make -j for large builds MUCH better at finding memory errors than memtest.
- paulmd 4y agoyep also true. maybe you could build some kind of tooling around this, if builds are deterministic (or you do something like pgbench with a deterministic workload/script) then there's no reason you can't check the output in addition to very obvious segfault failures.
- sliken 4y agoYears ago the GCC FAQ had an entry for a strange compiler error, that pretty much was only triggered by overclocking or memory errors. I just looked and couldn't find it.