5 ms·

I ran into this very tool when setting up my home server some months ago (ECC and ZFS being necessary for integrity’s sake). It works, though I’ve yet to see a

I ran into this very tool when setting up my home server some months ago (ECC and ZFS being necessary for integrity’s sake). It works, though I’ve yet to see a single error, which has only left me thinking that the fuss over ECC RAM and the risk of bit flips was overblown.

lillecarl 4y ago
They don't happen often, but when they do, they do. There's a reason why every datacenter does it. You must be able to trust your machine.
belter 4y ago
"You really DO need ECC Memory" - https://perspectives.mvdirona.com/2009/10/you-really-do-need-ecc-memory/ https://perspectives.mvdirona.com/2009/10/you-really-do-need...
Scramblejams 4y ago
I have some servers that in many years of service have never logged a single error. I have others that pop a cluster of errors over the course of a few hours every year or two. Beyond differences in component brands, no obvious reasons for the different behavior have presented themselves to me. FWIW I’m a believer in ECC. But give it some time. Maybe you’ll see errors. Or maybe you’ll get lucky and see none. Either way, you’re covered!
- wkat4242 4y ago
  You're not covered for all types of errors, only single bit flips.
  - Scramblejams 4y ago
    A double bit flip will result in a halt. Ya may not find it convenient, but for my purposes I call that covered. (I choose not to worry about flips of three bits or more. YMMV.)
    - theevilsharpie 4y ago
      > A double bit flip will result in a halt. This is dependent on the system configuration. For Linux, default behavior is to halt only if the uncorrected error occurs in memory used by the kernel. It's more accurate to say that double bit flips will be detected.
      - Scramblejams 4y ago
        Thanks for the clarification. My Google Fu apparently isn't up to the task, can you share where the knobs are to control this behavior?
        theevilsharpie 4y ago
        https://docs.kernel.org/admin-guide/ras.html#module-parameters https://docs.kernel.org/admin-guide/ras.html#module-paramete... Search for "edac_mc_panic_on_ue"
        Scramblejams 4y ago
        Thanks!
    - anonymousDan 4y ago
      Are there error patterns that could still result in silent failure? I'm not %100 sure how modern ECC RAM is implemented, but I can imagine a simple parity bit would fail silently if different bits fail simultaneously by flipping bits in the opposite direction.
      - Scramblejams 4y ago
        Flipping three or more bits can do this.
- paulmd 4y ago
  > Beyond differences in component brands, no obvious reasons for the different behavior have presented themselves to me. Google's results show that the error rate is strongly driven by bad modules rather than random errors. A module that throws one error is up to 224x more likely to throw another error, and corrected errors are a strong predictor of future uncorrectable errors. The major exception is cosmic rays, which can flip bits even in a good module. But unless you're at high altitude they don't really dominate the error rate. http://www.cs.toronto.edu/%7Ebianca/papers/sigmetrics09.pdf http://www.cs.toronto.edu/%7Ebianca/papers/sigmetrics09.pdf Obviously without ECC you don't know if you have bad modules, but, if a machine passes a memtest scrub for 24+ hours then it's probably fine. On-die ECC should also catch the majority of these errors, especially if you are not right at the limits of your memory training. JEDEC type timings should be very forgiving to transmission errors in general. People are very down on the "on-die ECC" on DDR5, they act like ECC is completely useless unless you protect the whole datapath, and of course it would be preferable to do so. But actually the on-die ECC should catch a large number of the errors, because bad modules (dies) are where the errors mostly happen.
  - jml7c5 4y ago
    My only objection to on-die ECC is that, AFAIK, there's no way to query the number of internal corrections that have occurred. That is important information! Yes, it might freak out some sysadmins that frequent corrected errors are now "normal" (after all, the main reason for on-die ECC is that manufacturers won't have to scrap slightly imperfect DRAM), but those individuals can be re-educated, and I really want to be able to tell if a given module is running within spec yet right at the edge of failure.
    - paulmd 4y ago
      yep definitely, it'd be nice to have a way to communicate this, I totally agree. And yeah they probably didn't because people would freak at seeing errors corrected during normal operation.
    - Scramblejams 4y ago
      Yes, I'd like to see these plotted over time and correlated with machine failures.
- pocketarc 4y ago
  Oh, absolutely, I wouldn't want to be without ECC, I understand the value of having the additional integrity, and the reassurance that things are OK. I was just hoping to see some errors on a regular basis, so I could feel vindicated and rave to everyone about how useful ECC is and point to my console and say "look at how many errors you might be missing!". ;)
greyw 4y ago
On nine servers (each 512 GiB RAM) I have access to one experienced an ECC error back in november. My local workstation (64 GiB RAM) logged two errors back in december.
loeg 4y ago
IIRC, Linux doesn't report corrected errors by default. If you enable that, you'll see a warning periodically (I see a few a year).
- pocketarc 4y ago
  Are you saying that the tool in the article doesn't report corrected errors? It looks to me like that's exactly what it does, but maybe you know more than I do.
  - adql 4y ago
    As in it won't send messages to kernel log and you need the tool to ask kernel about it
jml7c5 4y ago
Bit flips aren't evenly distributed over time or across RAM. In other words, the vast, vast majority of DIMMs are good and will never have a bit flip. For those modules, the money you spent on ECC will be in a sense "wasted". (Except for the value of knowing that you had a good module in the first place.)
sliken 4y ago
Heh, well there's 2 things I wanted from ECC. Sure I wanted the very occasional random bit flip to be caught and fixed. Making the system more reliable, maybe save a few process crashes or reboots over 10 years. Although I was much more interested in ability to find dying hardware. Whenever weird (file corruptions, processes dying, reboots, things not working like they did yesterday) could be from numerous causes. Voltage dips, problems in the power supply, software bugs, OS bugs, security compromise, defective CPU/Ram/motherboard/dimms, etc. It can be very hard to track down, you might well spend weeks and significant $$$ playing the game of swap the part and wait for the problem to reappear ... or not. Much like what recently happened to LinusT. I'll happy pay another 10% system price for a system that has a few less crashes and will tell me when it starts seeing memory errors (while fixing the small ones). I'm typing this from a Xeon E3-1230v5 bought in 2015, that cost LESS then the similar i7. I did spent a bit more on the motherboard ($50) and ram ($75), but it's been rock solid since and if I had a dimm problem it would likely be 10 minutes to troubleshoot and then I'd RMA or buy replacement ram. ECC reduces the chance of writing the wrong data to the right place on disk or, even worse, the right data to the wrong place.

lillecarl 4y ago

They don't happen often, but when they do, they do. There's a reason why every datacenter does it. You must be able to trust your machine.

belter 4y ago

"You really DO need ECC Memory" - https://perspectives.mvdirona.com/2009/10/you-really-do-need-ecc-memory/ https://perspectives.mvdirona.com/2009/10/you-really-do-need...

Scramblejams 4y ago

I have some servers that in many years of service have never logged a single error. I have others that pop a cluster of errors over the course of a few hours every year or two. Beyond differences in component brands, no obvious reasons for the different behavior have presented themselves to me. FWIW I’m a believer in ECC. But give it some time. Maybe you’ll see errors. Or maybe you’ll get lucky and see none. Either way, you’re covered!

wkat4242 4y ago
You're not covered for all types of errors, only single bit flips.
- Scramblejams 4y ago
  A double bit flip will result in a halt. Ya may not find it convenient, but for my purposes I call that covered. (I choose not to worry about flips of three bits or more. YMMV.)
  - theevilsharpie 4y ago
    > A double bit flip will result in a halt. This is dependent on the system configuration. For Linux, default behavior is to halt only if the uncorrected error occurs in memory used by the kernel. It's more accurate to say that double bit flips will be detected.
    - Scramblejams 4y ago
      Thanks for the clarification. My Google Fu apparently isn't up to the task, can you share where the knobs are to control this behavior?
      - theevilsharpie 4y ago
        https://docs.kernel.org/admin-guide/ras.html#module-parameters https://docs.kernel.org/admin-guide/ras.html#module-paramete... Search for "edac_mc_panic_on_ue"
        Scramblejams 4y ago
        Thanks!
  - anonymousDan 4y ago
    Are there error patterns that could still result in silent failure? I'm not %100 sure how modern ECC RAM is implemented, but I can imagine a simple parity bit would fail silently if different bits fail simultaneously by flipping bits in the opposite direction.
    - Scramblejams 4y ago
      Flipping three or more bits can do this.
paulmd 4y ago
> Beyond differences in component brands, no obvious reasons for the different behavior have presented themselves to me. Google's results show that the error rate is strongly driven by bad modules rather than random errors. A module that throws one error is up to 224x more likely to throw another error, and corrected errors are a strong predictor of future uncorrectable errors. The major exception is cosmic rays, which can flip bits even in a good module. But unless you're at high altitude they don't really dominate the error rate. http://www.cs.toronto.edu/%7Ebianca/papers/sigmetrics09.pdf http://www.cs.toronto.edu/%7Ebianca/papers/sigmetrics09.pdf Obviously without ECC you don't know if you have bad modules, but, if a machine passes a memtest scrub for 24+ hours then it's probably fine. On-die ECC should also catch the majority of these errors, especially if you are not right at the limits of your memory training. JEDEC type timings should be very forgiving to transmission errors in general. People are very down on the "on-die ECC" on DDR5, they act like ECC is completely useless unless you protect the whole datapath, and of course it would be preferable to do so. But actually the on-die ECC should catch a large number of the errors, because bad modules (dies) are where the errors mostly happen.
- jml7c5 4y ago
  My only objection to on-die ECC is that, AFAIK, there's no way to query the number of internal corrections that have occurred. That is important information! Yes, it might freak out some sysadmins that frequent corrected errors are now "normal" (after all, the main reason for on-die ECC is that manufacturers won't have to scrap slightly imperfect DRAM), but those individuals can be re-educated, and I really want to be able to tell if a given module is running within spec yet right at the edge of failure.
  - paulmd 4y ago
    yep definitely, it'd be nice to have a way to communicate this, I totally agree. And yeah they probably didn't because people would freak at seeing errors corrected during normal operation.
  - Scramblejams 4y ago
    Yes, I'd like to see these plotted over time and correlated with machine failures.
pocketarc 4y ago
Oh, absolutely, I wouldn't want to be without ECC, I understand the value of having the additional integrity, and the reassurance that things are OK. I was just hoping to see some errors on a regular basis, so I could feel vindicated and rave to everyone about how useful ECC is and point to my console and say "look at how many errors you might be missing!". ;)

greyw 4y ago

On nine servers (each 512 GiB RAM) I have access to one experienced an ECC error back in november. My local workstation (64 GiB RAM) logged two errors back in december.

loeg 4y ago

IIRC, Linux doesn't report corrected errors by default. If you enable that, you'll see a warning periodically (I see a few a year).

pocketarc 4y ago
Are you saying that the tool in the article doesn't report corrected errors? It looks to me like that's exactly what it does, but maybe you know more than I do.
- adql 4y ago
  As in it won't send messages to kernel log and you need the tool to ask kernel about it

jml7c5 4y ago

Bit flips aren't evenly distributed over time or across RAM. In other words, the vast, vast majority of DIMMs are good and will never have a bit flip. For those modules, the money you spent on ECC will be in a sense "wasted". (Except for the value of knowing that you had a good module in the first place.)

sliken 4y ago

Heh, well there's 2 things I wanted from ECC. Sure I wanted the very occasional random bit flip to be caught and fixed. Making the system more reliable, maybe save a few process crashes or reboots over 10 years. Although I was much more interested in ability to find dying hardware. Whenever weird (file corruptions, processes dying, reboots, things not working like they did yesterday) could be from numerous causes. Voltage dips, problems in the power supply, software bugs, OS bugs, security compromise, defective CPU/Ram/motherboard/dimms, etc. It can be very hard to track down, you might well spend weeks and significant $$$ playing the game of swap the part and wait for the problem to reappear ... or not. Much like what recently happened to LinusT. I'll happy pay another 10% system price for a system that has a few less crashes and will tell me when it starts seeing memory errors (while fixing the small ones). I'm typing this from a Xeon E3-1230v5 bought in 2015, that cost LESS then the similar i7. I did spent a bit more on the motherboard ($50) and ram ($75), but it's been rock solid since and if I had a dimm problem it would likely be 10 minutes to troubleshoot and then I'd RMA or buy replacement ram. ECC reduces the chance of writing the wrong data to the right place on disk or, even worse, the right data to the wrong place.