9 ms·

I have some servers that in many years of service have never logged a single error. I have others that pop a cluster of errors over the course of a few hours ev

I have some servers that in many years of service have never logged a single error. I have others that pop a cluster of errors over the course of a few hours every year or two. Beyond differences in component brands, no obvious reasons for the different behavior have presented themselves to me. FWIW I’m a believer in ECC. But give it some time. Maybe you’ll see errors. Or maybe you’ll get lucky and see none. Either way, you’re covered!

wkat4242 4y ago
You're not covered for all types of errors, only single bit flips.
- Scramblejams 4y ago
  A double bit flip will result in a halt. Ya may not find it convenient, but for my purposes I call that covered. (I choose not to worry about flips of three bits or more. YMMV.)
  - theevilsharpie 4y ago
    > A double bit flip will result in a halt. This is dependent on the system configuration. For Linux, default behavior is to halt only if the uncorrected error occurs in memory used by the kernel. It's more accurate to say that double bit flips will be detected.
    - Scramblejams 4y ago
      Thanks for the clarification. My Google Fu apparently isn't up to the task, can you share where the knobs are to control this behavior?
      - theevilsharpie 4y ago
        https://docs.kernel.org/admin-guide/ras.html#module-parameters https://docs.kernel.org/admin-guide/ras.html#module-paramete... Search for "edac_mc_panic_on_ue"
        Scramblejams 4y ago
        Thanks!
  - anonymousDan 4y ago
    Are there error patterns that could still result in silent failure? I'm not %100 sure how modern ECC RAM is implemented, but I can imagine a simple parity bit would fail silently if different bits fail simultaneously by flipping bits in the opposite direction.
    - Scramblejams 4y ago
      Flipping three or more bits can do this.
paulmd 4y ago
> Beyond differences in component brands, no obvious reasons for the different behavior have presented themselves to me. Google's results show that the error rate is strongly driven by bad modules rather than random errors. A module that throws one error is up to 224x more likely to throw another error, and corrected errors are a strong predictor of future uncorrectable errors. The major exception is cosmic rays, which can flip bits even in a good module. But unless you're at high altitude they don't really dominate the error rate. http://www.cs.toronto.edu/%7Ebianca/papers/sigmetrics09.pdf http://www.cs.toronto.edu/%7Ebianca/papers/sigmetrics09.pdf Obviously without ECC you don't know if you have bad modules, but, if a machine passes a memtest scrub for 24+ hours then it's probably fine. On-die ECC should also catch the majority of these errors, especially if you are not right at the limits of your memory training. JEDEC type timings should be very forgiving to transmission errors in general. People are very down on the "on-die ECC" on DDR5, they act like ECC is completely useless unless you protect the whole datapath, and of course it would be preferable to do so. But actually the on-die ECC should catch a large number of the errors, because bad modules (dies) are where the errors mostly happen.
- jml7c5 4y ago
  My only objection to on-die ECC is that, AFAIK, there's no way to query the number of internal corrections that have occurred. That is important information! Yes, it might freak out some sysadmins that frequent corrected errors are now "normal" (after all, the main reason for on-die ECC is that manufacturers won't have to scrap slightly imperfect DRAM), but those individuals can be re-educated, and I really want to be able to tell if a given module is running within spec yet right at the edge of failure.
  - paulmd 4y ago
    yep definitely, it'd be nice to have a way to communicate this, I totally agree. And yeah they probably didn't because people would freak at seeing errors corrected during normal operation.
  - Scramblejams 4y ago
    Yes, I'd like to see these plotted over time and correlated with machine failures.
pocketarc 4y ago
Oh, absolutely, I wouldn't want to be without ECC, I understand the value of having the additional integrity, and the reassurance that things are OK. I was just hoping to see some errors on a regular basis, so I could feel vindicated and rave to everyone about how useful ECC is and point to my console and say "look at how many errors you might be missing!". ;)

wkat4242 4y ago

You're not covered for all types of errors, only single bit flips.

Scramblejams 4y ago
A double bit flip will result in a halt. Ya may not find it convenient, but for my purposes I call that covered. (I choose not to worry about flips of three bits or more. YMMV.)
- theevilsharpie 4y ago
  > A double bit flip will result in a halt. This is dependent on the system configuration. For Linux, default behavior is to halt only if the uncorrected error occurs in memory used by the kernel. It's more accurate to say that double bit flips will be detected.
  - Scramblejams 4y ago
    Thanks for the clarification. My Google Fu apparently isn't up to the task, can you share where the knobs are to control this behavior?
    - theevilsharpie 4y ago
      https://docs.kernel.org/admin-guide/ras.html#module-parameters https://docs.kernel.org/admin-guide/ras.html#module-paramete... Search for "edac_mc_panic_on_ue"
      - Scramblejams 4y ago
        Thanks!
- anonymousDan 4y ago
  Are there error patterns that could still result in silent failure? I'm not %100 sure how modern ECC RAM is implemented, but I can imagine a simple parity bit would fail silently if different bits fail simultaneously by flipping bits in the opposite direction.
  - Scramblejams 4y ago
    Flipping three or more bits can do this.

paulmd 4y ago

> Beyond differences in component brands, no obvious reasons for the different behavior have presented themselves to me. Google's results show that the error rate is strongly driven by bad modules rather than random errors. A module that throws one error is up to 224x more likely to throw another error, and corrected errors are a strong predictor of future uncorrectable errors. The major exception is cosmic rays, which can flip bits even in a good module. But unless you're at high altitude they don't really dominate the error rate. http://www.cs.toronto.edu/%7Ebianca/papers/sigmetrics09.pdf http://www.cs.toronto.edu/%7Ebianca/papers/sigmetrics09.pdf Obviously without ECC you don't know if you have bad modules, but, if a machine passes a memtest scrub for 24+ hours then it's probably fine. On-die ECC should also catch the majority of these errors, especially if you are not right at the limits of your memory training. JEDEC type timings should be very forgiving to transmission errors in general. People are very down on the "on-die ECC" on DDR5, they act like ECC is completely useless unless you protect the whole datapath, and of course it would be preferable to do so. But actually the on-die ECC should catch a large number of the errors, because bad modules (dies) are where the errors mostly happen.

jml7c5 4y ago
My only objection to on-die ECC is that, AFAIK, there's no way to query the number of internal corrections that have occurred. That is important information! Yes, it might freak out some sysadmins that frequent corrected errors are now "normal" (after all, the main reason for on-die ECC is that manufacturers won't have to scrap slightly imperfect DRAM), but those individuals can be re-educated, and I really want to be able to tell if a given module is running within spec yet right at the edge of failure.
- paulmd 4y ago
  yep definitely, it'd be nice to have a way to communicate this, I totally agree. And yeah they probably didn't because people would freak at seeing errors corrected during normal operation.
- Scramblejams 4y ago
  Yes, I'd like to see these plotted over time and correlated with machine failures.

pocketarc 4y ago

Oh, absolutely, I wouldn't want to be without ECC, I understand the value of having the additional integrity, and the reassurance that things are OK. I was just hoping to see some errors on a regular basis, so I could feel vindicated and rave to everyone about how useful ECC is and point to my console and say "look at how many errors you might be missing!". ;)