6 ms·

> Without ECC you won't know that your system is failing and silently corrupting your data. The likelihood that a RAM error is corrupting data silently is sli

by zymhan 5y ago

> Without ECC you won't know that your system is failing and silently corrupting your data. The likelihood that a RAM error is corrupting data silently is slim, given the same corruption could affect the RAM storing the OS kernel or your application stack. If you have bad non-ECC RAM, you will notice issues that warrant further investigation (i.e. memtest). But that assumes you're savvy enough to understand what failing memory looks like.

mmis1000 5y ago
The real pain is you have no way to sort out the reason of crash. Without ecc, you don't know whether it is the driver you installed last day cause the crash, or the memory itself. Or even driver installed yesterday used the broken region of the memory. Who knows? And memtest won't even come close to 100% reproduce the problem. If it happens continually, then it probably work. For for something that probably crash 1 in a month? No, unless you are lucky enough. And that happen all the time if you overclock. That is basically why ddrm5 mandate in memory ecc. Because in that high frequency, data corruption is more likely to happen.
- bcrl 5y ago
  The most painful case I had was actually bad memory in a FIFO in an ethernet switch. A few months before one of the Red Hat releases, IT had installed a brand new shiny enterpricey Ethernet switch with lots of line cards and many ports. This was one of the early VLAN capable switches. During the release validation cycle, QA found that one of our stress test workloads was occasionally failing with single bit flips in filesystem data from time to time. Being kernel developers, we were majorly concerned. There were lots of changes to filesystem, VM and other parts of the kernel, and we couldn't rule out that we had a bug where code was using a stray pointer or some other problem. The stress tests had successfully passed last release... Ultimately we tracked it down to network data being corrupted. The shiny new ethernet switch was rarely flipping a bit in packets, then happily fixing up the packet's CRC and IP checksum so that the computers on either end of the network link had no idea that the data was mangled in the network fabric. Oh the hours of brain wracking pain that caused!
  - rstuart4133 5y ago
    > The shiny new ethernet switch was rarely flipping a bit in packets, then happily fixing up the packet's CRC and IP checksum so that the computers on either end of the network link had no idea that the data was mangled in the network fabric. shudder
bcrl 5y ago
It's not about "likelihood", it's about being able to trust the hardware. The CPU running in your system probably has ECC protected caches, and many RAS features that happen to be designed in because most CPU vendors design the features into their cores that are shared between consumer gear and enterprise servers. These features are a necessary part of building hardware that can be trusted with atomic scale features. Furthermore, your comment reads like someone that has never spent weeks debugging issues that only happen under extreme load stress tests. A single marginal bit is not so simple to track down when memory allocation patterns are non-deterministic. I have no desire to go through that again if it can be avoided by spending an extra $20 per DIMM. My time is worth more than that. Hardware is not perfect, and anyone who trusts hardware blindly should really take a look at what's going on behind the curtain. Fwiw, the nature of DRAM errors is different now compared to the studies published decades ago when cosmic rays would only disturb a single cell. These days minute physical manufacturing defects and electrical disturbances are the dominant failure modes. Educate yourself: read the papers published about row hammer. Hammering a row with reads can flip bits in adjacent rows, and even DRAMs that have supposed mitigation features can suffer from disturbances a row or two further away. It has already been proven that the DRAM vendors' hardware cannot be trusted! What more proof do you need that this is a real world requirement in 2021?

by zymhan 5y ago

mmis1000 5y ago

The real pain is you have no way to sort out the reason of crash. Without ecc, you don't know whether it is the driver you installed last day cause the crash, or the memory itself. Or even driver installed yesterday used the broken region of the memory. Who knows? And memtest won't even come close to 100% reproduce the problem. If it happens continually, then it probably work. For for something that probably crash 1 in a month? No, unless you are lucky enough. And that happen all the time if you overclock. That is basically why ddrm5 mandate in memory ecc. Because in that high frequency, data corruption is more likely to happen.

bcrl 5y ago
The most painful case I had was actually bad memory in a FIFO in an ethernet switch. A few months before one of the Red Hat releases, IT had installed a brand new shiny enterpricey Ethernet switch with lots of line cards and many ports. This was one of the early VLAN capable switches. During the release validation cycle, QA found that one of our stress test workloads was occasionally failing with single bit flips in filesystem data from time to time. Being kernel developers, we were majorly concerned. There were lots of changes to filesystem, VM and other parts of the kernel, and we couldn't rule out that we had a bug where code was using a stray pointer or some other problem. The stress tests had successfully passed last release... Ultimately we tracked it down to network data being corrupted. The shiny new ethernet switch was rarely flipping a bit in packets, then happily fixing up the packet's CRC and IP checksum so that the computers on either end of the network link had no idea that the data was mangled in the network fabric. Oh the hours of brain wracking pain that caused!
- rstuart4133 5y ago
  > The shiny new ethernet switch was rarely flipping a bit in packets, then happily fixing up the packet's CRC and IP checksum so that the computers on either end of the network link had no idea that the data was mangled in the network fabric. shudder

bcrl 5y ago

It's not about "likelihood", it's about being able to trust the hardware. The CPU running in your system probably has ECC protected caches, and many RAS features that happen to be designed in because most CPU vendors design the features into their cores that are shared between consumer gear and enterprise servers. These features are a necessary part of building hardware that can be trusted with atomic scale features. Furthermore, your comment reads like someone that has never spent weeks debugging issues that only happen under extreme load stress tests. A single marginal bit is not so simple to track down when memory allocation patterns are non-deterministic. I have no desire to go through that again if it can be avoided by spending an extra $20 per DIMM. My time is worth more than that. Hardware is not perfect, and anyone who trusts hardware blindly should really take a look at what's going on behind the curtain. Fwiw, the nature of DRAM errors is different now compared to the studies published decades ago when cosmic rays would only disturb a single cell. These days minute physical manufacturing defects and electrical disturbances are the dominant failure modes. Educate yourself: read the papers published about row hammer. Hammering a row with reads can flip bits in adjacent rows, and even DRAMs that have supposed mitigation features can suffer from disturbances a row or two further away. It has already been proven that the DRAM vendors' hardware cannot be trusted! What more proof do you need that this is a real world requirement in 2021?