4 ms·

Check, check, check. I only use ECC DRAM in my desktop these days and I monitor bit flips which on my 64GB desktop happen about once a month[1]. A similar rate

Check, check, check. I only use ECC DRAM in my desktop these days and I monitor bit flips which on my 64GB desktop happen about once a month[1]. A similar rate on my FreeNAS device which also has ECC memory. Like others, I was super happy when I could by a Ryzen chip at retail prices and get ECC rather than buying a Xeon sku which was an extra $300. [1] When I first built the system I was getting about 2 a week which I tracked down to a 'weak' ram stick. Replaced it (Corsair, and they replaced it for free) and now it's about one a month.

hyperpl 6y ago
How did you track these bit flips and how did you ultimately narrow it down to "weak" RAM?
- ChuckMcM 6y ago
  The memory log has both the bit number and the DIMM "row". They were all coming from the same row, so that was the DIMM I replaced.
tzs 6y ago
In the early 2000s for a year or two I had a process running on my Linux desktop without ECC that just filled a 128 MB buffer with a pattern, and then checked every minute to see if the buffer had changed. I ran this for a couple of years and never saw a change. I believe that based on published memory error rates for memory at that time, I should have seen a few errors. I am not sure why I never did. Maybe consumer desktop memory was less dense than server or workstation memory, and the published error rates were for the latter? From 2008 to 2017 I had a 2008 Mac Pro at work with ECC memory, and from 2009 to 2017 I had a 2009 Mac Pro at home with ECC memory. I'd occasional look in the memory section of the System Information report and never saw any ECC corrected errors reported. With the Mac Pros, it is possible that I just never happened to check the report between the time of a corrected error and the time the counter was next reset (assuming it resets every boot...if it is cumulative than I have no idea how I never got an error).
- ChuckMcM 6y ago
  FWIW these are 16GB DIMMs. So 64GB there are roughly 512 times as many bits in that memory as opposed to 128MB of memory. Assuming a linear relationship between bit flips[1] and time (it is a statistical event after all) 1 bit flip a month would be the equivalent of 1 bit flip every 42 years in 128MB. Or put differently, your survey size was quite a bit different than mine has been. Also there is always a chance that the bit flip is in the ECC bits not the main memory bits. ECC DIMMS typically have 72 bits per 64 bit long word, vs 66 bits per 64 data bits for parity memory, and 64 bits for 64 bits on memory with no protection at all. So it's hard to make a 1:1 comparison without knowing the exact layout of the memory and whether or not error detection bits are present. [1] Soft (and Hard) bit errors are generally spec'd as a probability per unit time per bit, so more bits means higher probability of a bit flip per unit time.
- toast0 6y ago
  Your sample size is too small. In my experience with thousands of servers with 64-512GB of ECC ram, not very many of them will have any ECC reports during use. Usually, everything works fine, but sometimes something will go wrong, and it's nice to know. Of course, that's when I'm not paying the bills. I don't have ECC on my home systems, because of added expense, lower performance (no XMP ECC ram when I was shopping), and extra research needed. If a consumer oriented motherboard says it supports ECC, that may mean it lets you use ECC ram, but doesn't enable any of the reporting, which would be mostly useless; not completely useless, because used ECC ram from server retirements is sometimes very inexpensive, but it's often fully buffered, which is less likely to work in consumer platforms.
  - marcan_42 6y ago
    > If a consumer oriented motherboard says it supports ECC, that may mean it lets you use ECC ram, but doesn't enable any of the reporting, which would be mostly useless That is largely a feature of OS software, not the motherboard. Linux on AMD motherboards which support ECC has reporting. There is no reason why it wouldn't. If the ECC is there and active, Linux can get the information (EDAC). On Intel though, they flat out fuse ECC off on consumer CPUs for market segmentation reasons. If you want a consumer motherboard with ECC, you go AMD.
madjam002 6y ago
I also have ECC memory for my Ryzen system, I’m curious what sticks you use as I really struggled to find mine as they have to be ECC UDIMMs?

hyperpl 6y ago

How did you track these bit flips and how did you ultimately narrow it down to "weak" RAM?

ChuckMcM 6y ago
The memory log has both the bit number and the DIMM "row". They were all coming from the same row, so that was the DIMM I replaced.

tzs 6y ago

In the early 2000s for a year or two I had a process running on my Linux desktop without ECC that just filled a 128 MB buffer with a pattern, and then checked every minute to see if the buffer had changed. I ran this for a couple of years and never saw a change. I believe that based on published memory error rates for memory at that time, I should have seen a few errors. I am not sure why I never did. Maybe consumer desktop memory was less dense than server or workstation memory, and the published error rates were for the latter? From 2008 to 2017 I had a 2008 Mac Pro at work with ECC memory, and from 2009 to 2017 I had a 2009 Mac Pro at home with ECC memory. I'd occasional look in the memory section of the System Information report and never saw any ECC corrected errors reported. With the Mac Pros, it is possible that I just never happened to check the report between the time of a corrected error and the time the counter was next reset (assuming it resets every boot...if it is cumulative than I have no idea how I never got an error).

ChuckMcM 6y ago
FWIW these are 16GB DIMMs. So 64GB there are roughly 512 times as many bits in that memory as opposed to 128MB of memory. Assuming a linear relationship between bit flips[1] and time (it is a statistical event after all) 1 bit flip a month would be the equivalent of 1 bit flip every 42 years in 128MB. Or put differently, your survey size was quite a bit different than mine has been. Also there is always a chance that the bit flip is in the ECC bits not the main memory bits. ECC DIMMS typically have 72 bits per 64 bit long word, vs 66 bits per 64 data bits for parity memory, and 64 bits for 64 bits on memory with no protection at all. So it's hard to make a 1:1 comparison without knowing the exact layout of the memory and whether or not error detection bits are present. [1] Soft (and Hard) bit errors are generally spec'd as a probability per unit time per bit, so more bits means higher probability of a bit flip per unit time.
toast0 6y ago
Your sample size is too small. In my experience with thousands of servers with 64-512GB of ECC ram, not very many of them will have any ECC reports during use. Usually, everything works fine, but sometimes something will go wrong, and it's nice to know. Of course, that's when I'm not paying the bills. I don't have ECC on my home systems, because of added expense, lower performance (no XMP ECC ram when I was shopping), and extra research needed. If a consumer oriented motherboard says it supports ECC, that may mean it lets you use ECC ram, but doesn't enable any of the reporting, which would be mostly useless; not completely useless, because used ECC ram from server retirements is sometimes very inexpensive, but it's often fully buffered, which is less likely to work in consumer platforms.
- marcan_42 6y ago
  > If a consumer oriented motherboard says it supports ECC, that may mean it lets you use ECC ram, but doesn't enable any of the reporting, which would be mostly useless That is largely a feature of OS software, not the motherboard. Linux on AMD motherboards which support ECC has reporting. There is no reason why it wouldn't. If the ECC is there and active, Linux can get the information (EDAC). On Intel though, they flat out fuse ECC off on consumer CPUs for market segmentation reasons. If you want a consumer motherboard with ECC, you go AMD.

madjam002 6y ago

I also have ECC memory for my Ryzen system, I’m curious what sticks you use as I really struggled to find mine as they have to be ECC UDIMMs?