6 ms·

The problem isn't RowHammer, the problem is lack of ECC. This is mostly Intel's fault, as they cripple their consumer CPU lines to disable ECC for market segmen

The problem isn't RowHammer, the problem is lack of ECC. This is mostly Intel's fault, as they cripple their consumer CPU lines to disable ECC for market segmentation purposes. The entire premise that we can store tens of gigabytes of information reliably in tiny silicon capacitors without any error detection or correction whatsoever is ludicrous. Every other storage technology uses advanced error correction - NAND Flash, HDDs, optical media, etc. We've been doing this for decades. Even floppies at least had error detection (checksums). All high speed transmission protocols also do error detection and re-transmission, except HDMI (because I guess nobody cares about the odd corrupted pixel). Same for anything going over radio. Once you have ECC, your error rate margin goes up by many orders of magnitude. Worst case these attacks turn into a DoS as ECC fails to correct a badly corrupted word, and most of the time it will just work. Plus you can detect attack attempts as the memory controller will report increased error rates. It's not possible to design modern RAM that is perfectly reliable. That's just physics. We know how to solve this problem. We just aren't doing it because Intel thinks consumers don't deserve reliable RAM. The only other option is doing ECC on-chip in the RAM itself, and that has other efficiency trade-offs which probably make it not worth it, at least not with DDR style interfaces (it might've made sense in an FB-DIMM style world where the memory controller is in the RAM)

userbinator 5y ago
ECC is at best a temporary workaround, and at worst an implicit approval of the creation of defective products. It's purpose is to prevent transient bitflips due to cosmic rays and such, not consistently reproducible errors from what is otherwise perfectly normal operation. Plus you can detect attack attempts as the memory controller will report increased error rates. Now you've turned specific access patterns into "attack attempts", and by discriminating against them as such, entirely destroyed a fundamental part of general-purpose computing. What's next, CPUs that are only guaranteed to do certain operations correctly, and everything else is considered an "attack"? That is NOT a direction that we should be heading. Hell fucking no!!! RAM should always hold what was last written to it, under all conditions of software accesses. To expect or imply that anything less is acceptable, is to destroy one of the foundations on which general-purpose computing is built. NAND Flash Where manufacturers are trying to sell 2-4x the capacity with 1/4 to 1/16th of the endurance? That's a rant for another thread... We know how to solve this problem. This problem didn't exist 15 years ago. It only got created because the RAM companies have forsaken correctness, and somehow managed to convince everyone else that it's not their own fault.
- staticassertion 5y ago
  > Now you've turned specific access patterns into "attack attempts", We already have error correction detection available to us and it's in use today. > RAM should always hold what was last written to it, under all conditions of software accesses. OK? That's what anyone who is advocating for ECC is already advocating for. ECC is error correcting up to 1 bit and error detecting up to 2 bits. The goal is to ensure that the property you've described holds. > It only got created because the RAM companies have forsaken correctness, This problem exists because... physics. Do you actually have a design for RAM that can match today's cell density while somehow providing perfect isolation? Saying this problem didn't exist 15 years ago is like saying that malware didn't impact the abacus - RAM 15 years ago was designed completely differently, and had considerably lower capabilities in terms of storage and latency. And for reference, hamming codes (which back ECC) were created in the 1950s for error correction in punch cards. This is not a new problem. ECC is the existing, obvious solution to this problem, it just wasn't built for this threat - but it's an excellent, proven method for correcting bit flips. Even without being designed to prevent rowhammer it already makes the attack much slower, less likely to succeed, and easier to detect (just monitor for a massive spike in ecc corrections). Trying to design high density RAM that maintains perfect isolation seems pointless. Improving ECC or ECC-like approaches is likely going to be much more effective.
  - beebeepka 5y ago
    > RAM 15 years ago was designed completely differently, and had considerably lower capabilities in terms of storage and latency. You got that backwards, chief. RAM latencies have increased tremendously over the last 15 years.
    - namibj 5y ago
      Actual latency has improved, but clock-relative latency hasn't. So what?
      - beebeepka 5y ago
        No, actual latencies have been climbing since ddr2. Thanks for the downvote though
Laforet 5y ago
Not here to defend Intel (their decision to remove ECC support from X299 chipset pretty much tanked their own HEDT platform), but I often wonder how much of the current disregard of ECC is the fault of the system integrators. As of now they would gladly ship configurations with single channel memory and under-par cooling to save a few bucks, while these cut corners could hurt performance by as much as 30% due to insufficient memory bandwidth and thermal throttling. Even if ECC is available everywhere I doubt any OEM will use it without charging their users a premium. This is kind of what has been going on with AMD MSDT platforms: ECC support is in the CPU but many motherboard does not even implement the necessary memory traces to enable it, and software support is even messier to the point that most people can't be bothered with it.
- staticassertion 5y ago
  Maybe while both ECC and non-ECC are both being sold, but that would be a temporary situation. The goal would be to have non-ECC memory be effectively deprecated, so the next revision of the spec enforces ECC as a requirement.
- marcan_42 5y ago
  This is a self-fulfilling issue; since Intel does not support ECC, ECC is not mainstream, so mainstream platforms don't care about ECC. It would cost motherboard manufacturers ~nothing to support this properly, so if it were accepted as a thing they would, and it would eventually be a requirement for newer RAM technologies. ECC should've been mandatory since about the DDR2 era, practically speaking. That it isn't is a collective failure of the computer industry, and Intel takes a big part of the blame. At least I had no issues getting ECC to work on my Threadripper box, but sure, that's HEDT.
6ue7nNMEEbHcM 5y ago
Thanks for posting this. I think there isn't much knowledge about ecc among typical customers. I wish at some point ecc would be the norm. The cost doesn't seem high but for any professional PC usage it seems it's necessary.
Zenst 5y ago
> The problem isn't RowHammer, the problem is lack of ECC. This is mostly Intel's fault, as they cripple their consumer CPU lines to disable ECC for market segmentation purposes. Whilst Intel did play a part, when they offered it as standard the uptake in the consumer market was low and the price factor and need at most consumer level was not there. Today it is there, though the consumer mindset is unaware of it and I do foresee a time in which either some in-the-wild exploit that makes media traction and they roll with it event. Then there will be a sudden panic rush with media rhetoric akin to the run-up to Y2K and drama headlines how everybody needs to buy a new computer. Now if that happened - Intel on the desktop would sure take a wack as it is currently. Then the whole supply and demand aspect that factors into the cost and alas - until there is some mass consumer awareness. Well, until then for the majority consumers, computer memory is just computer memory and for most of those, they don't even look into things like memory speed or dual-channel memory even. Which see's your DELL's et all selling systems with one large stick of RAM even today as your average consumer just don't know any better. So the move to ECC for consumers will need en event or large player marketing it. Maybe a balance of the two. Just the aspect of size of memory today in computers, let alone density and speeds, bit errors become far more likely and that is without the security aspects at play. So for stability alone ECC is becoming more and more justifiable.
- rasz 5y ago
  Please, The ONLY time Intel offered ECC as standard on its products was during brief 486 period when all chipsets the offered were considered highend and their chipset market share was in single digits. ECC was standard on first IBM PC 5150, on PS/2 line, on pretty much all 286 clones etc. Intel market segmentation shenanigans started around 1995 with Pentium line, with ECC absent from all consumer oriented product lines.
  - Zenst 5y ago
    True, the whole dip into RAMBUS kinda saw them shift things and when they came back to less niched out RAM, they split things business and consumer. however, been some odd glimmer of hope for the cheaper end for ECC suport from Intel - https://www.truenas.com/community/threads/surprise-surprise-kaby-lake-pentiums-support-ecc.54748/ https://www.truenas.com/community/threads/surprise-surprise-... What really needs to happen is all these AMD Ryzen users to buy into ECC, help get the price down and more so, push Intel in another tickbox catchup in that market.
femiagbabiaka 5y ago
I thought ECC wasn’t effective protection against Rowhammer? Agreed that it is incredibly useful otherwise. https://www.vusec.net/projects/eccploit/ https://www.vusec.net/projects/eccploit/
- marcan_42 5y ago
  ECC helps a lot, and combined with memory encryption (which is another thing we should be doing) it effectively mitigates the issue, as then you can't target specific bits any more, so your chances of getting past ECC without causing hard errors are negligible.
formerly_proven 5y ago
Doing ECC on-die (as in DDR5) does solve issues related to the storage itself, and the access timing of the array, but still leaves data vulnerable to corruption on the memory bus itself. Memory is indeed the only place were we are moving and storing data and it is commonplace to use zero layers of data protection; everywhere else we have at least one, often multiple, checksums and storage always has had forward error correction for a very long time, regardless of medium. But of course, generally speaking, all data processed by a computer moves through the main memory.

userbinator 5y ago

ECC is at best a temporary workaround, and at worst an implicit approval of the creation of defective products. It's purpose is to prevent transient bitflips due to cosmic rays and such, not consistently reproducible errors from what is otherwise perfectly normal operation. Plus you can detect attack attempts as the memory controller will report increased error rates. Now you've turned specific access patterns into "attack attempts", and by discriminating against them as such, entirely destroyed a fundamental part of general-purpose computing. What's next, CPUs that are only guaranteed to do certain operations correctly, and everything else is considered an "attack"? That is NOT a direction that we should be heading. Hell fucking no!!! RAM should always hold what was last written to it, under all conditions of software accesses. To expect or imply that anything less is acceptable, is to destroy one of the foundations on which general-purpose computing is built. NAND Flash Where manufacturers are trying to sell 2-4x the capacity with 1/4 to 1/16th of the endurance? That's a rant for another thread... We know how to solve this problem. This problem didn't exist 15 years ago. It only got created because the RAM companies have forsaken correctness, and somehow managed to convince everyone else that it's not their own fault.

staticassertion 5y ago
> Now you've turned specific access patterns into "attack attempts", We already have error correction detection available to us and it's in use today. > RAM should always hold what was last written to it, under all conditions of software accesses. OK? That's what anyone who is advocating for ECC is already advocating for. ECC is error correcting up to 1 bit and error detecting up to 2 bits. The goal is to ensure that the property you've described holds. > It only got created because the RAM companies have forsaken correctness, This problem exists because... physics. Do you actually have a design for RAM that can match today's cell density while somehow providing perfect isolation? Saying this problem didn't exist 15 years ago is like saying that malware didn't impact the abacus - RAM 15 years ago was designed completely differently, and had considerably lower capabilities in terms of storage and latency. And for reference, hamming codes (which back ECC) were created in the 1950s for error correction in punch cards. This is not a new problem. ECC is the existing, obvious solution to this problem, it just wasn't built for this threat - but it's an excellent, proven method for correcting bit flips. Even without being designed to prevent rowhammer it already makes the attack much slower, less likely to succeed, and easier to detect (just monitor for a massive spike in ecc corrections). Trying to design high density RAM that maintains perfect isolation seems pointless. Improving ECC or ECC-like approaches is likely going to be much more effective.
- beebeepka 5y ago
  > RAM 15 years ago was designed completely differently, and had considerably lower capabilities in terms of storage and latency. You got that backwards, chief. RAM latencies have increased tremendously over the last 15 years.
  - namibj 5y ago
    Actual latency has improved, but clock-relative latency hasn't. So what?
    - beebeepka 5y ago
      No, actual latencies have been climbing since ddr2. Thanks for the downvote though

Laforet 5y ago

Not here to defend Intel (their decision to remove ECC support from X299 chipset pretty much tanked their own HEDT platform), but I often wonder how much of the current disregard of ECC is the fault of the system integrators. As of now they would gladly ship configurations with single channel memory and under-par cooling to save a few bucks, while these cut corners could hurt performance by as much as 30% due to insufficient memory bandwidth and thermal throttling. Even if ECC is available everywhere I doubt any OEM will use it without charging their users a premium. This is kind of what has been going on with AMD MSDT platforms: ECC support is in the CPU but many motherboard does not even implement the necessary memory traces to enable it, and software support is even messier to the point that most people can't be bothered with it.

staticassertion 5y ago
Maybe while both ECC and non-ECC are both being sold, but that would be a temporary situation. The goal would be to have non-ECC memory be effectively deprecated, so the next revision of the spec enforces ECC as a requirement.
marcan_42 5y ago
This is a self-fulfilling issue; since Intel does not support ECC, ECC is not mainstream, so mainstream platforms don't care about ECC. It would cost motherboard manufacturers ~nothing to support this properly, so if it were accepted as a thing they would, and it would eventually be a requirement for newer RAM technologies. ECC should've been mandatory since about the DDR2 era, practically speaking. That it isn't is a collective failure of the computer industry, and Intel takes a big part of the blame. At least I had no issues getting ECC to work on my Threadripper box, but sure, that's HEDT.

6ue7nNMEEbHcM 5y ago

Thanks for posting this. I think there isn't much knowledge about ecc among typical customers. I wish at some point ecc would be the norm. The cost doesn't seem high but for any professional PC usage it seems it's necessary.

Zenst 5y ago

> The problem isn't RowHammer, the problem is lack of ECC. This is mostly Intel's fault, as they cripple their consumer CPU lines to disable ECC for market segmentation purposes. Whilst Intel did play a part, when they offered it as standard the uptake in the consumer market was low and the price factor and need at most consumer level was not there. Today it is there, though the consumer mindset is unaware of it and I do foresee a time in which either some in-the-wild exploit that makes media traction and they roll with it event. Then there will be a sudden panic rush with media rhetoric akin to the run-up to Y2K and drama headlines how everybody needs to buy a new computer. Now if that happened - Intel on the desktop would sure take a wack as it is currently. Then the whole supply and demand aspect that factors into the cost and alas - until there is some mass consumer awareness. Well, until then for the majority consumers, computer memory is just computer memory and for most of those, they don't even look into things like memory speed or dual-channel memory even. Which see's your DELL's et all selling systems with one large stick of RAM even today as your average consumer just don't know any better. So the move to ECC for consumers will need en event or large player marketing it. Maybe a balance of the two. Just the aspect of size of memory today in computers, let alone density and speeds, bit errors become far more likely and that is without the security aspects at play. So for stability alone ECC is becoming more and more justifiable.

rasz 5y ago
Please, The ONLY time Intel offered ECC as standard on its products was during brief 486 period when all chipsets the offered were considered highend and their chipset market share was in single digits. ECC was standard on first IBM PC 5150, on PS/2 line, on pretty much all 286 clones etc. Intel market segmentation shenanigans started around 1995 with Pentium line, with ECC absent from all consumer oriented product lines.
- Zenst 5y ago
  True, the whole dip into RAMBUS kinda saw them shift things and when they came back to less niched out RAM, they split things business and consumer. however, been some odd glimmer of hope for the cheaper end for ECC suport from Intel - https://www.truenas.com/community/threads/surprise-surprise-kaby-lake-pentiums-support-ecc.54748/ https://www.truenas.com/community/threads/surprise-surprise-... What really needs to happen is all these AMD Ryzen users to buy into ECC, help get the price down and more so, push Intel in another tickbox catchup in that market.

femiagbabiaka 5y ago

I thought ECC wasn’t effective protection against Rowhammer? Agreed that it is incredibly useful otherwise. https://www.vusec.net/projects/eccploit/ https://www.vusec.net/projects/eccploit/

marcan_42 5y ago
ECC helps a lot, and combined with memory encryption (which is another thing we should be doing) it effectively mitigates the issue, as then you can't target specific bits any more, so your chances of getting past ECC without causing hard errors are negligible.

formerly_proven 5y ago

Doing ECC on-die (as in DDR5) does solve issues related to the storage itself, and the access timing of the array, but still leaves data vulnerable to corruption on the memory bus itself. Memory is indeed the only place were we are moving and storing data and it is commonplace to use zero layers of data protection; everywhere else we have at least one, often multiple, checksums and storage always has had forward error correction for a very long time, regardless of medium. But of course, generally speaking, all data processed by a computer moves through the main memory.