4 ms·

As someone who has been aware of ECC RAM for a long time could you briefly explain a concrete case where that is useful? Like, in all my time using computers I

As someone who has been aware of ECC RAM for a long time could you briefly explain a concrete case where that is useful? Like, in all my time using computers I don't think I've yet encountered an error caused by RAM, or.. Did I?

jiggawatts 5y ago
You probably didn't even realise it was a memory error. A lot of the instability of Windows can be attributed to the use of non-ECC memory in typical home PCs. Linux tends to run on servers with ECC memory, so at least some of the perspective of it being "more stable" comes from this anecdotal evidence. The parent comment saying that they never see ECC errors in the wild is missing a few things: - Server memory tends to be clocked lower than consumer memory, so errors are less frequent to begin with. - The errors are not evenly distributed. Some memory sticks have a high error rate, others are virtually zero. There's batch-to-batch variations. - I've done my own tests on hundreds of servers. We run burn-in tests for about 24-48 hours. About 95% have zero errors of any kind, but 5% have a high enough rate that putting them into production would be a mistake. ECC allows us to catch those bit errors instead of silently accepting them and allowing data corruption to creep in. - I've personally had 3 different personal computers experience high memory error rates, to the point of multiple BSODs per day and data corruption. They all started off "good" and slowly turned "bad". The only reason I knew to look for memory corruption as the root cause is because of my extensive industry experience. A grandma using the same PC would have just blamed Windows for being unstable. - Vendors like Microsoft simply ignore all crash error reports sent back by telemetry with only 1 or 2 samples, because those are virtually guaranteed to be caused by memory corruption, not programmer error. I've done similar memory dump collection and found that easily 30% of all crashes were unique in this way, suggesting that ECC memory could improve PC stability significantly. - Suggesting that ECC memory is not needed because "good" memory doesn't need it and only "bad" memory is a problem is missing the point. All memory is bad, it's just that the bit error rates are different!
- feffe 5y ago
  I recently had an interesting experience with bad non-ECC RAM. The only effect I saw for many months was a frequently crashing web browser. I was very annoyed and downgraded to the firefox LTS release to try to solve the issue. The problem went away for a while, then came back. Seemingly erratic the problem could resurface only to disappear a while later. Eventually I got a corruption in a large Git repository with photos. First I suspected a disk error but reruns of "git fsck" reported different bad objects, run to run. How odd, so I ran memtest86 and it reported a bad 32 MB memory region at offset 4 GB. Never saw any kernel issues or other instability. Booting up the computer does not use up 4 GB of memory (Linux) but starting a bloated web browser does. The computer is stable after mapping away the bad memory region by using GRUB_BADRAM. That was an new takeaway for me, a machine with ECC memory can do this bad blocks mapping automatically. It's not just beneficial in correcting single bit errors. I would love to have ECC memory in my machine. I used it for my previous build in 2012 but I think the situation has gotten worse since then. Bigger price difference and ECC memory is not even available at the same clocks as non-ECC memory.
PragmaticPulp 5y ago
Practically speaking, ECC is extra insurance against faulty memory. It's true that ECC will catch errors caused by cosmic rays, but in practice most memory errors are just from faulty memory. Large studies in Google datacenters showed that the errors were heavily concentrated in a small number of DIMMs: http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf The catch is that you don't really know if you have one of these faulty memory sticks. I always run memtest86 overnight to check for obviously faulty memory, but some of these errors could take months to manifest. I had a laptop with a RAM cell that failed. It would show up in memtest86 in a matter of minutes, yet surprisingly I didn't notice the issue for a very long time in day to day usage. I always wonder if there were random bit flips in anything I worked with during that period, but I'll never know. With ECC, it's just one less thing to worry about.
- glandium 5y ago
  One part of the equation is that a shit-ton amount of memory modules are, in fact, overclocked (e.g. do you think your DDR4-4800 runs at nominal speed?). That surely contributes to memory being faulty.
  - mmis1000 5y ago
    DDR4 Memory these day are rarely runs at their default frequency (2666,2333) unless you buy a really cheap one. Most 3200/3600 memory are in fact overclocked(Just the factory pre-configure the overclock setting with xmp) except for a few ones that are natively 3200(some memory from micron are natively 3200 at 1.2v).
Gigachad 5y ago
How would you ever know? If your computer or a program has ever crashed or acted weird. That's a symptom of memory errors. It's also a symptom of a million more common issues but you would never know. There are some bugs that get raised on linux once every few years that are so obscure they are suspected to be random memory errors.
Lammy 5y ago
Anecdote: I once lost a lot of files over a period of ~two years on a desktop PC running ZFS with bad non-ECC RAM. I figured out what was going on after a bunch of audible corrupt blips kept appearing in all my favorite FLAC rips as they got mapped into the bad memory, checksummed against the contents still on disk, and written back "corrected" as far as ZFS was concerned. There's a better (i.e. not by me) write-up of it in the OP of this thread under "What happens when non-ECC RAM goes bad in a ZFS system?": https://www.truenas.com/community/threads/ecc-vs-non-ecc-ram-and-zfs.15449/ https://www.truenas.com/community/threads/ecc-vs-non-ecc-ram...
- lambdaba 5y ago
  Thank you, this is a perfect illustration and one of the scenarios I feared around keeping data long-term, though I was thinking about disk errors and such. Sidenote I seem to have experienced Cunningham's Law for the first time semi-consciously, as I was aware my claim is most probably wrong. Still, I wonder if data corruption is a more common scenario rather than crashing, because ever since I use Linux I don't recall experiencing many crashes I couldn't attribute to a driver issue or known instability in the software I used, especially on compatible hardware. Windows days were another story... But maybe the memory modules were worse back then and perhaps having less memory made it more likely to crash?
mmis1000 5y ago
You will find it useful if you crashed 1 in a month and realize turn the memory frequency down for a bit fixed it forever.(Technically I am still guessing, without ecc, you can't get the answer) Seriously, the hardware should be able to determine whether the configuration is safe or not instead of by randomly guessing yourself. Even the memory labeled as work as 3200, it isn't necessary the case when you actually use it.
sliken 5y ago
Well there's a bunch of problems. Could be high energy particles flipping bits, but that's very rare. Could be a row hammer attack flipping bits. Could be a flaw in the memory chip, dimm, dimm socket, motherboard traces, CPU socket, or inside the CPU. Without ECC any of the above will crash an app, or even the entire machine. With ECC you'll get a log message, and if it's a single bit error it will be automatically repaired. If it's not fixable and it's in userspace that application is killed (and the kernel logs the error), if it's in kernel space the kernel will panic. So generally it makes your machine more reliable, and easier to debug. Generally repeated errors are a fault of some kind and you can easily tell which dimm it is. Without ECC you end up troubleshooting all causes of crashes, and even if you know it's memory, you can't be sure which dimm it is. So I think it's well worth the minimal premium to make your machine more reliable, and if it's unreliable it's much easier to track and fix the problem. Amusingly without ECC heavy CPU use with parallel gcc compiles causes a particular error. Not sure if it's in the FAQ, but it's well know that the particular error means you have a CPU/RAM problem, common with memory errors, CPUs that are too hot, or overclocked CPUs.

jiggawatts 5y ago

You probably didn't even realise it was a memory error. A lot of the instability of Windows can be attributed to the use of non-ECC memory in typical home PCs. Linux tends to run on servers with ECC memory, so at least some of the perspective of it being "more stable" comes from this anecdotal evidence. The parent comment saying that they never see ECC errors in the wild is missing a few things: - Server memory tends to be clocked lower than consumer memory, so errors are less frequent to begin with. - The errors are not evenly distributed. Some memory sticks have a high error rate, others are virtually zero. There's batch-to-batch variations. - I've done my own tests on hundreds of servers. We run burn-in tests for about 24-48 hours. About 95% have zero errors of any kind, but 5% have a high enough rate that putting them into production would be a mistake. ECC allows us to catch those bit errors instead of silently accepting them and allowing data corruption to creep in. - I've personally had 3 different personal computers experience high memory error rates, to the point of multiple BSODs per day and data corruption. They all started off "good" and slowly turned "bad". The only reason I knew to look for memory corruption as the root cause is because of my extensive industry experience. A grandma using the same PC would have just blamed Windows for being unstable. - Vendors like Microsoft simply ignore all crash error reports sent back by telemetry with only 1 or 2 samples, because those are virtually guaranteed to be caused by memory corruption, not programmer error. I've done similar memory dump collection and found that easily 30% of all crashes were unique in this way, suggesting that ECC memory could improve PC stability significantly. - Suggesting that ECC memory is not needed because "good" memory doesn't need it and only "bad" memory is a problem is missing the point. All memory is bad, it's just that the bit error rates are different!

feffe 5y ago
I recently had an interesting experience with bad non-ECC RAM. The only effect I saw for many months was a frequently crashing web browser. I was very annoyed and downgraded to the firefox LTS release to try to solve the issue. The problem went away for a while, then came back. Seemingly erratic the problem could resurface only to disappear a while later. Eventually I got a corruption in a large Git repository with photos. First I suspected a disk error but reruns of "git fsck" reported different bad objects, run to run. How odd, so I ran memtest86 and it reported a bad 32 MB memory region at offset 4 GB. Never saw any kernel issues or other instability. Booting up the computer does not use up 4 GB of memory (Linux) but starting a bloated web browser does. The computer is stable after mapping away the bad memory region by using GRUB_BADRAM. That was an new takeaway for me, a machine with ECC memory can do this bad blocks mapping automatically. It's not just beneficial in correcting single bit errors. I would love to have ECC memory in my machine. I used it for my previous build in 2012 but I think the situation has gotten worse since then. Bigger price difference and ECC memory is not even available at the same clocks as non-ECC memory.

PragmaticPulp 5y ago

Practically speaking, ECC is extra insurance against faulty memory. It's true that ECC will catch errors caused by cosmic rays, but in practice most memory errors are just from faulty memory. Large studies in Google datacenters showed that the errors were heavily concentrated in a small number of DIMMs: http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf The catch is that you don't really know if you have one of these faulty memory sticks. I always run memtest86 overnight to check for obviously faulty memory, but some of these errors could take months to manifest. I had a laptop with a RAM cell that failed. It would show up in memtest86 in a matter of minutes, yet surprisingly I didn't notice the issue for a very long time in day to day usage. I always wonder if there were random bit flips in anything I worked with during that period, but I'll never know. With ECC, it's just one less thing to worry about.

glandium 5y ago
One part of the equation is that a shit-ton amount of memory modules are, in fact, overclocked (e.g. do you think your DDR4-4800 runs at nominal speed?). That surely contributes to memory being faulty.
- mmis1000 5y ago
  DDR4 Memory these day are rarely runs at their default frequency (2666,2333) unless you buy a really cheap one. Most 3200/3600 memory are in fact overclocked(Just the factory pre-configure the overclock setting with xmp) except for a few ones that are natively 3200(some memory from micron are natively 3200 at 1.2v).

Gigachad 5y ago

How would you ever know? If your computer or a program has ever crashed or acted weird. That's a symptom of memory errors. It's also a symptom of a million more common issues but you would never know. There are some bugs that get raised on linux once every few years that are so obscure they are suspected to be random memory errors.

Lammy 5y ago

Anecdote: I once lost a lot of files over a period of ~two years on a desktop PC running ZFS with bad non-ECC RAM. I figured out what was going on after a bunch of audible corrupt blips kept appearing in all my favorite FLAC rips as they got mapped into the bad memory, checksummed against the contents still on disk, and written back "corrected" as far as ZFS was concerned. There's a better (i.e. not by me) write-up of it in the OP of this thread under "What happens when non-ECC RAM goes bad in a ZFS system?": https://www.truenas.com/community/threads/ecc-vs-non-ecc-ram-and-zfs.15449/ https://www.truenas.com/community/threads/ecc-vs-non-ecc-ram...

lambdaba 5y ago
Thank you, this is a perfect illustration and one of the scenarios I feared around keeping data long-term, though I was thinking about disk errors and such. Sidenote I seem to have experienced Cunningham's Law for the first time semi-consciously, as I was aware my claim is most probably wrong. Still, I wonder if data corruption is a more common scenario rather than crashing, because ever since I use Linux I don't recall experiencing many crashes I couldn't attribute to a driver issue or known instability in the software I used, especially on compatible hardware. Windows days were another story... But maybe the memory modules were worse back then and perhaps having less memory made it more likely to crash?

mmis1000 5y ago

You will find it useful if you crashed 1 in a month and realize turn the memory frequency down for a bit fixed it forever.(Technically I am still guessing, without ecc, you can't get the answer) Seriously, the hardware should be able to determine whether the configuration is safe or not instead of by randomly guessing yourself. Even the memory labeled as work as 3200, it isn't necessary the case when you actually use it.

sliken 5y ago

Well there's a bunch of problems. Could be high energy particles flipping bits, but that's very rare. Could be a row hammer attack flipping bits. Could be a flaw in the memory chip, dimm, dimm socket, motherboard traces, CPU socket, or inside the CPU. Without ECC any of the above will crash an app, or even the entire machine. With ECC you'll get a log message, and if it's a single bit error it will be automatically repaired. If it's not fixable and it's in userspace that application is killed (and the kernel logs the error), if it's in kernel space the kernel will panic. So generally it makes your machine more reliable, and easier to debug. Generally repeated errors are a fault of some kind and you can easily tell which dimm it is. Without ECC you end up troubleshooting all causes of crashes, and even if you know it's memory, you can't be sure which dimm it is. So I think it's well worth the minimal premium to make your machine more reliable, and if it's unreliable it's much easier to track and fix the problem. Amusingly without ECC heavy CPU use with parallel gcc compiles causes a particular error. Not sure if it's in the FAQ, but it's well know that the particular error means you have a CPU/RAM problem, common with memory errors, CPUs that are too hot, or overclocked CPUs.