5 ms·

Judging from Jeff Atwood's post only three days ago, a lot of people converted or seem to believe now that bit flips in RAMs are a thing of the past and ECC is

Judging from Jeff Atwood's post only three days ago, a lot of people converted or seem to believe now that bit flips in RAMs are a thing of the past and ECC is just an "enterprisey" thing. https://news.ycombinator.com/item?id=10598629 https://news.ycombinator.com/item?id=10598629 He didn't dare himself to not use ECC for the database... :)

mud_dauber 11y ago
Seeing that DRAM pricing is so dirt cheap, it's difficult to persuade any chip manufacturer to devote another 10-15% (my estimate based on previous memory design projects) of die area to ECC bits & decode circuitry when they won't get another dime for the capability.
- feld 11y ago
  Why can't they charge 15% more for ECC to recoup the costs?
  - mud_dauber 11y ago
    It depends on the end customer. PC manufacturers have zero incentive to add ECC when 99.99% of consumers don't care or even know. Networking: a mixed big. Some designs manage ECC inside the processor, others use memories with ECC for specialized functions like packet QoS. Commmodity DRAM errors are a known irritant and planned for. Servers: I'm not quite sure of my facts here, but a DRAM manufacturer would have to persuade a commmodity manager at Dell, etc, that ECC is worth it. A commodity manager won't give a d* unless his systems guys say that it's needed.
    - feld 11y ago
      I don't think you get the point. Just stop making non-ECC RAM for all future DDR releases. Let's just say, for example, "DDR6 has ECC baked into the standard" and have everyone on board with this. The memory manufacturers bump prices a bit to cover the costs and the entire world moves on unabashed.
      - mud_dauber 11y ago
        Ah, now I get it. I'll step back and let economics take over. :)
      - PhantomGremlin 11y ago
        Just stop making non-ECC RAM That's what's currently done with flash, but for DRAM that's not smart from a system point of view. A long time ago ECC was done with Hamming Codes[1]; now there are similar but more sophisticated versions. These codes share a property such that the wider the word, the fewer the additional bits required. So, e.g. 8 bits + 5 bits 16 bits + 6 bits 32 bits + 7 bits 64 bits + 8 bits That is enough for a minimum "hamming distance" of 4 between valid words. That gets you SEC (single error correction) plus DED (double error detection). You can save 1 bit if you're willing to forego the latter. Many decades ago (but I couldn't find a reference with a quick google) Micron made a chip that had on-board ECC. IIRC it was 8+4, which meant that it would internally correct a 1 bit error, but there was a possibility it would mis-correct some 2+ bit errors (which were rare). But that meant that Micron was putting 50% more bits onto a die over a non-ECC device. They probably did this not for system reliability but to cover up the failings of their chips at the time. Yes you could make each bit smaller, but no way would it pay off if you needed to put 50% more bits onto a die. ECC is best done with wide words. For example, a SIMM that presents a 72-bit interface can easily be built with 9 chips, each 8-bits wide. The system (CPU, memory controller, whatever) takes in 72 bits (or some multiple thereof) from DRAM and does ECC internally. Note that as words get wider the time to compute ECC goes up. So a system could even speculatively execute using the uncorrected data, while in parallel checking to make sure that it was OK. This requires recovery in case of error, but since errors are infrequent it could be a big win overall. It doesn't make sense for a DDRx spec to require ECC. It's cheaper and smarter to do ECC at a system level. [1] https://en.wikipedia.org/wiki/Hamming_code https://en.wikipedia.org/wiki/Hamming_code
  - pkaye 11y ago
    PC manufacturers are incredibly cost sensitive right now. It is kind of like the airline industry where they strip away a $4 meal and $1 blanket on a $250 ticket to save money.
    - sufiyan 11y ago
      True for most of the world of consumer electronics. 10 cent saving on a TV - the person who proposes the technique stands to make a quarters salary as a bonus.
- jacquesm 11y ago
  ECC is not just a DRAM thing, you can't put ECC DRAM into a machine that does not support it. So even if there was a demand for it the CPU/Chipset/Motherboard combo would have to allow for the option to use ECC DRAM first before you could make that choice.
- monocasa 11y ago
  You don't have to add anything to the DRAM chips for ECC. It's just another chip on the DIMM to store the extra ECC bits. It's the memory controller that puts the pieces back together.
- deleted 11y ago
  [deleted]
feld 11y ago
His logic is beyond broken. "Why do I need ECC when I can just have more servers in my cluster in case something bad happens" Umm, I don't know -- perhaps data integrity? Having multiple servers doesn't solve the "I just wrote bad data to disk" problem.
- lmm 11y ago
  If the client calls the cluster nodes individually then yes it does, provided there's enough overlap. Say the client considers a write committed when it's written to 5/7 of the cluster, and a read is only valid when you've heard from 5/7 of the cluster. Then whenever you read you're guaranteed to read at least three saved values, so if they disagree you can take the majority value (and write a correction). And that's the kind of system you already need to be safe against hard drive failures (just with the numbers at 4/7 and 4/7), so it's cheap to enable. In a modern distributed system you're already effectively doing ECC at a cross-machine level, so there's little value in doing it again within each individual machine.
  - Retric 11y ago
    Depends on how much processing you’re doing. It's not hard to get to the point where 7/7 machines all give different answers.
  - xorcist 11y ago
    Do you end clients really write to all nodes? I know pretty much no application for which this is the case.
    - lmm 11y ago
      Sometimes. Cassandra and similar datastores follow this model. I've been out of the web game a while but it seems like a sensible way to do a "single page application" - indeed it seems like the only way to have true high availability, if the client only makes a single call then whatever initially receives that call (e.g. the load balancer) becomes a single point of failure.
  - 11y ago

mud_dauber 11y ago

Seeing that DRAM pricing is so dirt cheap, it's difficult to persuade any chip manufacturer to devote another 10-15% (my estimate based on previous memory design projects) of die area to ECC bits & decode circuitry when they won't get another dime for the capability.

feld 11y ago
Why can't they charge 15% more for ECC to recoup the costs?
- mud_dauber 11y ago
  It depends on the end customer. PC manufacturers have zero incentive to add ECC when 99.99% of consumers don't care or even know. Networking: a mixed big. Some designs manage ECC inside the processor, others use memories with ECC for specialized functions like packet QoS. Commmodity DRAM errors are a known irritant and planned for. Servers: I'm not quite sure of my facts here, but a DRAM manufacturer would have to persuade a commmodity manager at Dell, etc, that ECC is worth it. A commodity manager won't give a d* unless his systems guys say that it's needed.
  - feld 11y ago
    I don't think you get the point. Just stop making non-ECC RAM for all future DDR releases. Let's just say, for example, "DDR6 has ECC baked into the standard" and have everyone on board with this. The memory manufacturers bump prices a bit to cover the costs and the entire world moves on unabashed.
    - mud_dauber 11y ago
      Ah, now I get it. I'll step back and let economics take over. :)
    - PhantomGremlin 11y ago
      Just stop making non-ECC RAM That's what's currently done with flash, but for DRAM that's not smart from a system point of view. A long time ago ECC was done with Hamming Codes[1]; now there are similar but more sophisticated versions. These codes share a property such that the wider the word, the fewer the additional bits required. So, e.g. 8 bits + 5 bits 16 bits + 6 bits 32 bits + 7 bits 64 bits + 8 bits That is enough for a minimum "hamming distance" of 4 between valid words. That gets you SEC (single error correction) plus DED (double error detection). You can save 1 bit if you're willing to forego the latter. Many decades ago (but I couldn't find a reference with a quick google) Micron made a chip that had on-board ECC. IIRC it was 8+4, which meant that it would internally correct a 1 bit error, but there was a possibility it would mis-correct some 2+ bit errors (which were rare). But that meant that Micron was putting 50% more bits onto a die over a non-ECC device. They probably did this not for system reliability but to cover up the failings of their chips at the time. Yes you could make each bit smaller, but no way would it pay off if you needed to put 50% more bits onto a die. ECC is best done with wide words. For example, a SIMM that presents a 72-bit interface can easily be built with 9 chips, each 8-bits wide. The system (CPU, memory controller, whatever) takes in 72 bits (or some multiple thereof) from DRAM and does ECC internally. Note that as words get wider the time to compute ECC goes up. So a system could even speculatively execute using the uncorrected data, while in parallel checking to make sure that it was OK. This requires recovery in case of error, but since errors are infrequent it could be a big win overall. It doesn't make sense for a DDRx spec to require ECC. It's cheaper and smarter to do ECC at a system level. [1] https://en.wikipedia.org/wiki/Hamming_code https://en.wikipedia.org/wiki/Hamming_code
- pkaye 11y ago
  PC manufacturers are incredibly cost sensitive right now. It is kind of like the airline industry where they strip away a $4 meal and $1 blanket on a $250 ticket to save money.
  - sufiyan 11y ago
    True for most of the world of consumer electronics. 10 cent saving on a TV - the person who proposes the technique stands to make a quarters salary as a bonus.
jacquesm 11y ago
ECC is not just a DRAM thing, you can't put ECC DRAM into a machine that does not support it. So even if there was a demand for it the CPU/Chipset/Motherboard combo would have to allow for the option to use ECC DRAM first before you could make that choice.
monocasa 11y ago
You don't have to add anything to the DRAM chips for ECC. It's just another chip on the DIMM to store the extra ECC bits. It's the memory controller that puts the pieces back together.
deleted 11y ago
[deleted]

feld 11y ago

His logic is beyond broken. "Why do I need ECC when I can just have more servers in my cluster in case something bad happens" Umm, I don't know -- perhaps data integrity? Having multiple servers doesn't solve the "I just wrote bad data to disk" problem.

lmm 11y ago
If the client calls the cluster nodes individually then yes it does, provided there's enough overlap. Say the client considers a write committed when it's written to 5/7 of the cluster, and a read is only valid when you've heard from 5/7 of the cluster. Then whenever you read you're guaranteed to read at least three saved values, so if they disagree you can take the majority value (and write a correction). And that's the kind of system you already need to be safe against hard drive failures (just with the numbers at 4/7 and 4/7), so it's cheap to enable. In a modern distributed system you're already effectively doing ECC at a cross-machine level, so there's little value in doing it again within each individual machine.
- Retric 11y ago
  Depends on how much processing you’re doing. It's not hard to get to the point where 7/7 machines all give different answers.
- xorcist 11y ago
  Do you end clients really write to all nodes? I know pretty much no application for which this is the case.
  - lmm 11y ago
    Sometimes. Cassandra and similar datastores follow this model. I've been out of the web game a while but it seems like a sensible way to do a "single page application" - indeed it seems like the only way to have true high availability, if the client only makes a single call then whatever initially receives that call (e.g. the load balancer) becomes a single point of failure.
- 11y ago