5 ms·

Error Correction isn't the important part of ECC: it's Error Checking. There was a very long thread recently about this in the context of row hammer over at th

by bcrl 5y ago

Error Correction isn't the important part of ECC: it's Error Checking. There was a very long thread recently about this in the context of row hammer over at the forums on Real World Tech. While ECC can't fix row hammer, it can let you know that something is wrong and the system operator needs to intervene. The same thing is needed on desktops. Without ECC you won't know that your system is failing and silently corrupting your data. Especially when DRAM vendors are shipping products that don't reliably meet the defined operating parameters in their data sheets. "Trust, but verify". The extra cost for ECC is basically noise in the cost of a desktop system. I buy unbuffered ECC DIMMs for my AMD systems and they're basically maybe 5-10% more than a non-ECC DIMM. The SSD premium is more than that. Registered DIMMs have a higher premium mostly because they need to have additional ICs on the DIMM to handle the higher memory capacities that servers tend to support. I really wish CPU vendors would just flat out require ECC on every system shipped. Knowing that main memory is failing would save so much time speaking as someone who has encountered DIMM failures in otherwise stable systems. Days of debugging what turns out to be a hardware issue is no fun.

deckard1 5y ago
> I buy unbuffered ECC DIMMs for my AMD systems and they're basically maybe 5-10% more than a non-ECC DIMM Where? Are you talking DDR4-3200? These are 40-50% higher than non-ECC UDIMMs. It usually gets worse as capacity increases. Additionally, there are only one or two models of 1x32GB UDIMM in production. These sticks are quite rare. You don't just go to Newegg or Amazon unless you want to pay out the arse for them. And in addition to that, ECC has worse timings which are important for Ryzen. > Registered DIMMs have a higher premium You have this backwards. RDIMM are common and cheaper because that's what servers generally use. UDIMM ECC cost more because, well, they barely exist as a thing.
- bcrl 5y ago
  DDR4-3200 is an expensive boutique product. Drop the speed requirement a little and you'll find the pricing is in line with my comment.
  - boardwaalk 5y ago
    DDR4-3200 is already fairly slow for a new system -- DDR4-3600 f.e. aligns with the 1800 MHz Fabric Clock in Ryzen 3 and makes a not insubstantial difference in performance, and (non-ECC-wise) is not expensive. I have a feeling the cost is not just a money-wise 5-10%.
    - bcrl 5y ago
      Amazon currently lists the 32GB Nemix DDR4-2666 UDIMM for $169.99 while a non-ECC UDIMM is $129.99. So you're right, at $40 it is closer to a 30% premium right now. But that one time cost of $40 per DIMM is worth it if it even saves me 10-15 minutes of time debugging. And I'm willing to pay that small cost adder on every new machine I get, because I have the battle scars that have pro It's not like I'm dealing with hundreds of systems, this is across maybe 2 or 3 dozen machines over the past 30 years.ven it's a worthwhile tradeoff. The battle scars are as follows: - The 2MB expansion card on my Amiga 500 turned out to have a couple of bad bits, and because I was a green techie back then, I just attributed it to general software instability. Years later a RAM test showed there was a problem. Sadly there were no memory test utilities shipped with Amigas back then. - My first 486 system had a bad bank in the SRAM on the motherboard. Took years to figure out as it only really hit when heavy DMA occurred from a VLB SCSI card while the CPU was heavily loaded. Swapped the SRAM out and it was stable for a few more years of service. - One of my K6 systems had a DIMM go bad. At least it failed badly enough that the system couldn't boot. - I've had an Intel Xeon in a colocated server start throwing ECC errors in the L3 cache. Replaced and RMAed the CPU before it caused an outage. - One system started throwing ECC errors because the power supply was marginal. Now, please, show me how the extra cost of ECC is worthless to you when things like this happen in the Real World. Is your time spent debugging hardware failures really not worth the cost of ECC?
  - deckard1 5y ago
    Thanks for the downvotes, folks. Really. > DDR4-3200 is an expensive boutique product. You have to be fucking kidding me. On a desktop Ryzen?? Most desktop PC builders are gamers. That's just a fact. They drive the market. That's why every motherboard looks like a gamery stealth bomber. Outside of Asrock Rack and one or two Asus Pro boards, there is simply nothing out there that matches what the server world is seeing with SuperMicro. I have 1x32GB DDR4-3200 UDIMM ECC on my Ryzen. But I have no delusions about the cost. It was expensive as fuck compared to non-ECC RAM.
- dcow 5y ago
  Nexmix is what most vendors use: https://nemixram.com/64gb-4x16gb-ddr4-3200-pc4-25600-ecc-udimm-memory https://nemixram.com/64gb-4x16gb-ddr4-3200-pc4-25600-ecc-udi.... Pricing seems on par…
- namibj 5y ago
  UDIMMs don't have worse timings. The only bad thing is that they don't offer binned dies with XMP profiles on their SPD. My Kingston Server Premier 32G sticks with Micron Rev. E 16Gbit dies (KSM32ES8/16ME) run these timings very reliably at 1.4V: BankGroupSwap: Disabled BankGroupSwapAlt: Enabled Memory Clock: 1800 MHz GDM: Enabled CR: 1T Tcl: 18 Tras: 39 Trcdrd: 22 Trcdwr: 22 Trc: 83 Trp: 22 Trrds: 5 Trrdl: 9 Trtp: 14 Tfaw: 38 Tcwl: 18 Twtrs: 5 Twtrl: 14 Twr: 26 Trdrddd: 4 Trdrdsd: 5 Trdrdsc: 1 Trdrdscl: 5 Twrwrdd: 6 Twrwrsd: 7 Twrwrsc: 1 Twrwrscl: 5 Twrrd: 3 Trdwr: 9 Tcke: 0 Trfc: 630 Trfc2: 468 Trfc4: 288 Yes, they are somehow really hard to get in the US for decent pricing, but in the EU they are quite affordable (according to geizhals.eu ).
  - deckard1 5y ago
    OC'ing ECC is generally a terrible idea, if that's what you're doing. Also, no one on Ryzen is considering CAS CL22 timings "good" on DDR4-3200 sticks. I think you need to go do research on non-ECC RAM. It really feels like everyone on this thread is so far outside the normal PC market for RAM that they are talking out of their ass here.
    - namibj 5y ago
      The notably part is that I operate it at above 1.2V, which increases power consumption in a way that enterprise generally doesn't deem worthwhile. I run DDR4-3600 CL18. I do question though how OC'ing non-ECC is a better idea than OC'ing ECC. Also, just for the record, I based my timings off of an XMP profile for a non-ECC version of the same die/capacity configuration (32G 2Rx8 w/ Micron Rev.E 16Gbit). I also haven't seen 32G sticks that do better than CL22 at 3200 with just 1.2V.
    - thulle 5y ago
      > OC'ing ECC is generally a terrible idea, if that's what you're doing. How so? For example I can see when the higher summer temperatures becomes an issue and reduce my timings, then tighten them again as it gets colder. No more random crashes where I wonder if it's my RAM or something else. Currently doing 3333MT/s CL14 on 4x16GB of 2666MT/s ECC. I use CL16 in the summer. Since the memory sticks barely are affected by the heat, and that ECC errors occur on the same slot of the motherboard regardless of which stick is in there, I assume its the memory controller on the CPU in combination with four sticks of memory that's the limiting factor here.
zymhan 5y ago
> Without ECC you won't know that your system is failing and silently corrupting your data. The likelihood that a RAM error is corrupting data silently is slim, given the same corruption could affect the RAM storing the OS kernel or your application stack. If you have bad non-ECC RAM, you will notice issues that warrant further investigation (i.e. memtest). But that assumes you're savvy enough to understand what failing memory looks like.
- mmis1000 5y ago
  The real pain is you have no way to sort out the reason of crash. Without ecc, you don't know whether it is the driver you installed last day cause the crash, or the memory itself. Or even driver installed yesterday used the broken region of the memory. Who knows? And memtest won't even come close to 100% reproduce the problem. If it happens continually, then it probably work. For for something that probably crash 1 in a month? No, unless you are lucky enough. And that happen all the time if you overclock. That is basically why ddrm5 mandate in memory ecc. Because in that high frequency, data corruption is more likely to happen.
  - bcrl 5y ago
    The most painful case I had was actually bad memory in a FIFO in an ethernet switch. A few months before one of the Red Hat releases, IT had installed a brand new shiny enterpricey Ethernet switch with lots of line cards and many ports. This was one of the early VLAN capable switches. During the release validation cycle, QA found that one of our stress test workloads was occasionally failing with single bit flips in filesystem data from time to time. Being kernel developers, we were majorly concerned. There were lots of changes to filesystem, VM and other parts of the kernel, and we couldn't rule out that we had a bug where code was using a stray pointer or some other problem. The stress tests had successfully passed last release... Ultimately we tracked it down to network data being corrupted. The shiny new ethernet switch was rarely flipping a bit in packets, then happily fixing up the packet's CRC and IP checksum so that the computers on either end of the network link had no idea that the data was mangled in the network fabric. Oh the hours of brain wracking pain that caused!
rstuart4133 5y ago
Same thought from a slightly different perspective. Decades ago I built a networking stack from scratch. It's wasn't wonderfully reliable. Years later I figured out why. I was wet behind the ears back then, and I put my faith in ARCnet's 16 bit CRC. Had I put aside that blind faith for just a few seconds and looked at how many packets per second that we being sent versus the number that would get through then - well I didn't. I'm going to claim it wasn't entirely my fault. ARCnet did automatic resends for you so if fixed what the CRC detected, and silently what it didn't detect through. I had no idea the errors were happening. ECC is nice, but as you say I don't need fancy hardware to correct a malfunctioning RAM chip. I can do that perfectly well with a hammer. What I need is the hardware to tell when and where to deploy the hammer. Things were actually better 30 years ago - back then PC RAM had parity. It's cheap, fast and effective enough. I have no idea what went wrong.

by bcrl 5y ago

deckard1 5y ago

> I buy unbuffered ECC DIMMs for my AMD systems and they're basically maybe 5-10% more than a non-ECC DIMM Where? Are you talking DDR4-3200? These are 40-50% higher than non-ECC UDIMMs. It usually gets worse as capacity increases. Additionally, there are only one or two models of 1x32GB UDIMM in production. These sticks are quite rare. You don't just go to Newegg or Amazon unless you want to pay out the arse for them. And in addition to that, ECC has worse timings which are important for Ryzen. > Registered DIMMs have a higher premium You have this backwards. RDIMM are common and cheaper because that's what servers generally use. UDIMM ECC cost more because, well, they barely exist as a thing.

bcrl 5y ago
DDR4-3200 is an expensive boutique product. Drop the speed requirement a little and you'll find the pricing is in line with my comment.
- boardwaalk 5y ago
  DDR4-3200 is already fairly slow for a new system -- DDR4-3600 f.e. aligns with the 1800 MHz Fabric Clock in Ryzen 3 and makes a not insubstantial difference in performance, and (non-ECC-wise) is not expensive. I have a feeling the cost is not just a money-wise 5-10%.
  - bcrl 5y ago
    Amazon currently lists the 32GB Nemix DDR4-2666 UDIMM for $169.99 while a non-ECC UDIMM is $129.99. So you're right, at $40 it is closer to a 30% premium right now. But that one time cost of $40 per DIMM is worth it if it even saves me 10-15 minutes of time debugging. And I'm willing to pay that small cost adder on every new machine I get, because I have the battle scars that have pro It's not like I'm dealing with hundreds of systems, this is across maybe 2 or 3 dozen machines over the past 30 years.ven it's a worthwhile tradeoff. The battle scars are as follows: - The 2MB expansion card on my Amiga 500 turned out to have a couple of bad bits, and because I was a green techie back then, I just attributed it to general software instability. Years later a RAM test showed there was a problem. Sadly there were no memory test utilities shipped with Amigas back then. - My first 486 system had a bad bank in the SRAM on the motherboard. Took years to figure out as it only really hit when heavy DMA occurred from a VLB SCSI card while the CPU was heavily loaded. Swapped the SRAM out and it was stable for a few more years of service. - One of my K6 systems had a DIMM go bad. At least it failed badly enough that the system couldn't boot. - I've had an Intel Xeon in a colocated server start throwing ECC errors in the L3 cache. Replaced and RMAed the CPU before it caused an outage. - One system started throwing ECC errors because the power supply was marginal. Now, please, show me how the extra cost of ECC is worthless to you when things like this happen in the Real World. Is your time spent debugging hardware failures really not worth the cost of ECC?
- deckard1 5y ago
  Thanks for the downvotes, folks. Really. > DDR4-3200 is an expensive boutique product. You have to be fucking kidding me. On a desktop Ryzen?? Most desktop PC builders are gamers. That's just a fact. They drive the market. That's why every motherboard looks like a gamery stealth bomber. Outside of Asrock Rack and one or two Asus Pro boards, there is simply nothing out there that matches what the server world is seeing with SuperMicro. I have 1x32GB DDR4-3200 UDIMM ECC on my Ryzen. But I have no delusions about the cost. It was expensive as fuck compared to non-ECC RAM.
dcow 5y ago
Nexmix is what most vendors use: https://nemixram.com/64gb-4x16gb-ddr4-3200-pc4-25600-ecc-udimm-memory https://nemixram.com/64gb-4x16gb-ddr4-3200-pc4-25600-ecc-udi.... Pricing seems on par…
namibj 5y ago
UDIMMs don't have worse timings. The only bad thing is that they don't offer binned dies with XMP profiles on their SPD. My Kingston Server Premier 32G sticks with Micron Rev. E 16Gbit dies (KSM32ES8/16ME) run these timings very reliably at 1.4V: BankGroupSwap: Disabled BankGroupSwapAlt: Enabled Memory Clock: 1800 MHz GDM: Enabled CR: 1T Tcl: 18 Tras: 39 Trcdrd: 22 Trcdwr: 22 Trc: 83 Trp: 22 Trrds: 5 Trrdl: 9 Trtp: 14 Tfaw: 38 Tcwl: 18 Twtrs: 5 Twtrl: 14 Twr: 26 Trdrddd: 4 Trdrdsd: 5 Trdrdsc: 1 Trdrdscl: 5 Twrwrdd: 6 Twrwrsd: 7 Twrwrsc: 1 Twrwrscl: 5 Twrrd: 3 Trdwr: 9 Tcke: 0 Trfc: 630 Trfc2: 468 Trfc4: 288 Yes, they are somehow really hard to get in the US for decent pricing, but in the EU they are quite affordable (according to geizhals.eu ).
- deckard1 5y ago
  OC'ing ECC is generally a terrible idea, if that's what you're doing. Also, no one on Ryzen is considering CAS CL22 timings "good" on DDR4-3200 sticks. I think you need to go do research on non-ECC RAM. It really feels like everyone on this thread is so far outside the normal PC market for RAM that they are talking out of their ass here.
  - namibj 5y ago
    The notably part is that I operate it at above 1.2V, which increases power consumption in a way that enterprise generally doesn't deem worthwhile. I run DDR4-3600 CL18. I do question though how OC'ing non-ECC is a better idea than OC'ing ECC. Also, just for the record, I based my timings off of an XMP profile for a non-ECC version of the same die/capacity configuration (32G 2Rx8 w/ Micron Rev.E 16Gbit). I also haven't seen 32G sticks that do better than CL22 at 3200 with just 1.2V.
  - thulle 5y ago
    > OC'ing ECC is generally a terrible idea, if that's what you're doing. How so? For example I can see when the higher summer temperatures becomes an issue and reduce my timings, then tighten them again as it gets colder. No more random crashes where I wonder if it's my RAM or something else. Currently doing 3333MT/s CL14 on 4x16GB of 2666MT/s ECC. I use CL16 in the summer. Since the memory sticks barely are affected by the heat, and that ECC errors occur on the same slot of the motherboard regardless of which stick is in there, I assume its the memory controller on the CPU in combination with four sticks of memory that's the limiting factor here.

zymhan 5y ago

> Without ECC you won't know that your system is failing and silently corrupting your data. The likelihood that a RAM error is corrupting data silently is slim, given the same corruption could affect the RAM storing the OS kernel or your application stack. If you have bad non-ECC RAM, you will notice issues that warrant further investigation (i.e. memtest). But that assumes you're savvy enough to understand what failing memory looks like.

mmis1000 5y ago
The real pain is you have no way to sort out the reason of crash. Without ecc, you don't know whether it is the driver you installed last day cause the crash, or the memory itself. Or even driver installed yesterday used the broken region of the memory. Who knows? And memtest won't even come close to 100% reproduce the problem. If it happens continually, then it probably work. For for something that probably crash 1 in a month? No, unless you are lucky enough. And that happen all the time if you overclock. That is basically why ddrm5 mandate in memory ecc. Because in that high frequency, data corruption is more likely to happen.
- bcrl 5y ago
  The most painful case I had was actually bad memory in a FIFO in an ethernet switch. A few months before one of the Red Hat releases, IT had installed a brand new shiny enterpricey Ethernet switch with lots of line cards and many ports. This was one of the early VLAN capable switches. During the release validation cycle, QA found that one of our stress test workloads was occasionally failing with single bit flips in filesystem data from time to time. Being kernel developers, we were majorly concerned. There were lots of changes to filesystem, VM and other parts of the kernel, and we couldn't rule out that we had a bug where code was using a stray pointer or some other problem. The stress tests had successfully passed last release... Ultimately we tracked it down to network data being corrupted. The shiny new ethernet switch was rarely flipping a bit in packets, then happily fixing up the packet's CRC and IP checksum so that the computers on either end of the network link had no idea that the data was mangled in the network fabric. Oh the hours of brain wracking pain that caused!

rstuart4133 5y ago

Same thought from a slightly different perspective. Decades ago I built a networking stack from scratch. It's wasn't wonderfully reliable. Years later I figured out why. I was wet behind the ears back then, and I put my faith in ARCnet's 16 bit CRC. Had I put aside that blind faith for just a few seconds and looked at how many packets per second that we being sent versus the number that would get through then - well I didn't. I'm going to claim it wasn't entirely my fault. ARCnet did automatic resends for you so if fixed what the CRC detected, and silently what it didn't detect through. I had no idea the errors were happening. ECC is nice, but as you say I don't need fancy hardware to correct a malfunctioning RAM chip. I can do that perfectly well with a hammer. What I need is the hardware to tell when and where to deploy the hammer. Things were actually better 30 years ago - back then PC RAM had parity. It's cheap, fast and effective enough. I have no idea what went wrong.