10 ms·

From the article: > We demonstrate that it is possible to trigger Rowhammer bit flips on all DRAM devices today despite deployed mitigations on commodity off-th

by nickcw 5y ago

From the article: > We demonstrate that it is possible to trigger Rowhammer bit flips on all DRAM devices today despite deployed mitigations on commodity off-the-shelf systems with little effort. The fact that user space code can cause bit flips in your RAM is a hardware bug. I'd love to see this code in memory testers like memtest86 so I could send the RAM back if it ever caught a problem like this. I guess it shows just how close to the edge of not working our modern computing environment is.

StillBored 5y ago
Pretty sure the passmark version does https://www.memtest86.com/compare.html https://www.memtest86.com/compare.html Except that the distro's are still shipping an older version due to license issues (or something like that AFAIK).
temac 5y ago
The problem is right now DDR4 devices that work correctly in that regard, do not exist. Likely the best mitigation for any sensitive application, even so lightly, is DDR4 with ECC (even if it may not be enough, it is vastly better than nothing, and not just because of rowhammer) And I have no idea if the internal "ECC" of standard DDR5 helps or not. It is not intended for regular ECC level of reliability anyway. (And I have seen discussion about likely bitflips detected in crash dumps of M1 Pro devices) So, as much as I would like to return defective devices, I would probably be left with no computer, no smartphone, etc.
- jandrese 5y ago
  Maybe a paranoid app could try to allocate the memory adjacent to any sensitive bits as a buffer? That would be pretty difficult to do but might be possible if you are tremendously paranoid and have a good way to examine the hardware.
  - pas 5y ago
    Or use different banks of DRAM for different security contexts, managed via NUMA?
  - to11mtm 5y ago
    I wonder how easy that would be to reason about though. I really don't know much about modern DRAM circuitry I feel like it might be abstracted away to some level, also there are bios-tweaky settings that might make a difference (i.e. things like channel configuration and/or bank interleaving, if that's still a thing.) IDK though. Maybe there's a way. I've worked on code that uses padding around a data structure to ensure it has it's own cache line. Maybe if you allocate a large enough contiguous block you'll be OK?
  - mnw21cam 5y ago
    You'd have to have the operating system's cooperation, because it may be mapping individual 4k pages all over the place, and it's the only thing that has a chance of knowing how the memory is laid out on the actual chips.
    - AshamedCaptain 5y ago
      And the CPU and the chipset's cooperation. In many cases it is even a trade secret how data is striped across the different slots/banks.
      - xxpor 5y ago
        Doesn't the same thing to exploiting rowhammer though? If you're after specific data, how would you know what physical address to use, even if you knew the physical address of the target data?
        jandrese 5y ago
        Yes, but there is a difference between an attack being run by a low level hardware hacker vs. the software being run by average users. This is why I said it would be hard, you would need to encode a lot of low level hardware knowledge into your application. This might only be useful in very specialized circumstances, like with kernel support on only a handful of carefully chosen hardware platforms. However, someone like Apple could do this on their systems, as they control both the kernel and the hardware. Sensitive memory locations could be cordoned off in special zones with buffering to prevent Rowhammer type attacks.
      - StillBored 5y ago
        <i> it is even a trade secret how data is striped across the different slots/banks </i> If that is true, vs just the usual "you don't need to know" trash common these days, its crazy. A pretty good picture can be built up with just software timing analysis, but its hardly rocket science to hook a logic analyzer/scope up and determine bit swizzling and page/bank interleave. Plus, many of the more open BIOS vendors have tended to provide page/row/controller/socket interleave options for quite a while, partially because it can mean a 5-10% uplift for some applications to have it set one way or the other. Its been one of those how do you tune your memory system options for a couple decades now.
  - staticassertion 5y ago
    If your app is particularly paranoid you could just double your memory usage, and verify errors against the clone.
- buran77 5y ago
  According to older paper [0] ECC can also be bypassed after reverse-engineering the mitigation in DDR3 DIMMs. Also: > “DDR4 systems with ECC will likely be more exploitable, after reverse-engineering the ECC functions,” researchers Razavi and Jattke said > What if I have ECC-capable DIMMs? Previous work [1] showed that due to the large number of bit flips in current DDR4 devices, ECC cannot provide complete protection against Rowhammer but makes exploitation harder. [0] https://cs.vu.nl/~lcr220/ecc/ecc-rh-paper-eccploit-press-preprint.pdf https://cs.vu.nl/~lcr220/ecc/ecc-rh-paper-eccploit-press-pre... [1] https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8835222 https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8835222
  - FpUser 5y ago
    While it can not protect from bit flips it can most likely warn about attack in progress due to a large number of bit flips.
- myself248 5y ago
  Yep, though I agree with the parent, if a RAM returns data from a location that is different from what is stored in that location, assuming all recommended timings have been followed, then that RAM is defective. If that means they should be recommending a refresh after every single operation, then that's what it means. In other words, the whole industry sits on a bed of lies at this point and it's only because government is technically incompetent that we haven't seen the world's biggest class-action.
  - ygjb 5y ago
    > In other words, the whole industry sits on a bed of lies at this point and it's only because government is technically incompetent that we haven't seen the world's biggest class-action. That's a pretty bold claim - if it's accurate, can you point at the government regulation that precludes a class action lawsuit? I would assume it's something in the T&C or EULA for the hardware rather than a regulation?
    - userbinator 5y ago
      Bold but sadly correct. The industry runs on misdirection and deception. If users knew better, we'd have the equivalent of another Pentium FDIV bug.
  - rzerda 5y ago
    The whole industry built this bed of lies by their own hands, I don’t see how it’s government’s fault.
- indymike 5y ago
  > The problem is right now DDR4 devices that work correctly in that regard, do not exist. Yes, and this is actually the problem. Without safe hardware, it is almost impossible to write safe software.
  - userbinator 5y ago
    Meanwhile you can still buy DDR3 --- used --- from various shops in China, and it'll be 100% perfect and Rowhammer-free because it was made before the insane process shrinkages that caused it. (Have done this a few times. Was hesitant initially because of the low price and used nature, but when the first stick passed MemTest86+ including RH perfectly, I was convinced. Half a dozen more sticks later and still good. But maybe they've started to run out of the good old stuff now...)
    - inetknght 5y ago
      > you can still buy DDR3 --- used --- from various shops in China, and it'll be 100% perfect and Rowhammer-free because it was made before the insane process shrinkages that caused it That's demonstrably false. I remember rowhammer being thought of and demonstrated on DDR3 modules. [0] https://hammertux.github.io/rowhammer-ffs-ddr3 https://hammertux.github.io/rowhammer-ffs-ddr3
      - userbinator 5y ago
        Rowhammer started appearing in newer DDR3; see the original paper (figure 3): https://news.ycombinator.com/item?id=8713411 https://news.ycombinator.com/item?id=8713411 The tl;dr is that everything 2009 and older is good, 2010-2011 is possibly bad, 2012 and newer is almost certainly bad.
  - throwaway81523 5y ago
    > Yes, and this is actually the problem. Without safe hardware, it is almost impossible to write safe software. Software that attempts a hostile rowhammer attack aims to be unsafe by definition. It's not a matter of trying and failing to be safe. Rather, we're seeing that it's very difficult to build hardware and software that prevents hostile code from running amuck: if the attacker can run hostile code on your computer at all, you have already lost. That's bad news for cloud hosts and confirms what we already knew about web scripting, but doesn't seem all THAT bad for the rest of us. Of course it's possible to build computers with no DRAM at all (i.e. only SRAM, which costs more). That is not economically sane for big servers, but might be the right thing to do for security sensitive components.
- oopsyDoodl 5y ago
  I think this illustrates the cloud is unacceptable for anything more than storage and retrieval. All computed results from data science must include steps and code to verify locally. It calls into question privacy on federated networks and crypto networks; any node can be manipulated locally to change payload outputs on delivery, reveal secrets, disrupt workloads. This makes sense to a lot of folks in computer engineering and physics, versus abstract software. No physical theory I know of offers any guarantee our arbitrary computing machines will ever be securable. We put fart pipes on Hondas. Science proves it’s titillating smoke and mirrors once again. Still waiting for nuclear rocket cars. I think this proves further as well why general computing chips need to be replaced with workload specific designs, where the anticipated inputs are well known and no vague logic paths to intentionally allow software monkey patching ever ship.
  - Ysraes 5y ago
    > I think this illustrates the cloud is unacceptable for anything more than storage and retrieval. You can run in dedicated tenancy where you have the whole machine or a metal configuration where you also have the whole machine.
    - frankzinger 5y ago
      That would ruin providers' user:hardware ratios, one of the foundational principals of cloud computing.
      - Terretta 5y ago
        On the contrary, certain CSPs did this already (quietly) and further, had already developed hardware mitigations for things like meltdown, spectre and rowhammer.
        frankzinger 5y ago
        But did they manage to keep their prices low? (And those who employed hardware mitigations wouldn't have a problem with user:hardware ratios, would they?)
jdmichal 5y ago
> I'd love to see this code in memory testers like memtest86 so I could send the RAM back if it ever caught a problem like this. Is this an individual stick bug, or a design bug akin to Spectre? Given that they tested 40 different sticks and seemed to find effective patterns for all of them make me think it's the latter. In which case, just return all your computers?
- jandrese 5y ago
  Yeah, this is a problem with high density DRAM. Fixes are possible but difficult and expensive. In the cutthroat DRAM market nobody seems willing to put in the engineering effort to develop a solution and then manufacture it. It would have to sell at a premium, in a market where it's difficult to even find ECC memory at times. The article states that ECC isn't sufficient to solve Rowhammer, but it does make the attack harder. If you are looking for a hardware solution it is your only option currently, even if it is imperfect.
  - seiferteric 5y ago
    I imagine you are right that consumers don't care or know enough to put pressure on manufacturers, but what about big companies like the FANGs? Or Government agencies like NSA etc? What are they doing about it? I imagine intelligence agencies are well aware of the issue and have or are trying to get solutions to protect their own infra.
    - codetrotter 5y ago
      Off topic but shouldn’t FAANG really be referred to as MAAAN now? Google -> Alphabet Facebook -> Meta Meta, Apple, Amazon, Alphabet, Netflix. MAAAN.
      - somethingwitty1 5y ago
        Google still exists. Facebook still exists. Meta/Alphabet are mainly holding companies. For example, I received reach outs from recruiters for Google and Facebook today. Not from Meta and Alphabet. Maybe once Meta and Alphabet deliver on things and build brand recognition, it should change. Until then, I'd vote "no".
tbob22 5y ago
MemTest86 already has a rowhammer test, not sure how it compares to blacksmith but some sticks I have do fail that test and can only be mitigated by setting tREFI extremely low (while also taking a large performance hit). Most of the higher end hynix/micron/samsung sticks I've tried do not fail at JEDEC or XMP after 7+ passes.
mithro 5y ago
Google is working on platforms to make this easier to explore this problem, see https://opensource.googleblog.com/2021/11/Open%20source%20DDR%20controller%20framework%20for%20mitigating%20Rowhammer.html?m=1 https://opensource.googleblog.com/2021/11/Open%20source%20DD...
gjs278 5y ago
did you read the fucking headline? all ram is vulnerable to this. you don’t need to run the test or return anything. you won’t get a stick that isn’t vulnerable. jesus you are dumb
userbinator 5y ago
The problem is that apparently the authors of memory testing programs were somehow convinced to hide the severity of this issue when it first appeared, claiming things like it being a rare edge case and that too much RAM would fail. I believe newer versions of MemTest86+ have Rowhammer tests but it's disabled by default. See my previous comment on this matter: https://news.ycombinator.com/item?id=12410274 https://news.ycombinator.com/item?id=12410274
jacquesm 5y ago
You'll be sending back all of your RAM then.
usr1106 5y ago
> I'd love to see this code in memory testers like memtest86 so I could send the RAM back if it ever caught a problem like this. How do you run a memory tester after you have returned all CPUs after Meltdown and all the subsequent microarchicture bugs? I'd call all modern CPUs faulty products. Some more than others.
- fy20 5y ago
  If it's a problem with chips becoming higher density, I wonder if older hardware will become popular in circles that care more about security than performance per rack unit.

by nickcw 5y ago

StillBored 5y ago

Pretty sure the passmark version does https://www.memtest86.com/compare.html https://www.memtest86.com/compare.html Except that the distro's are still shipping an older version due to license issues (or something like that AFAIK).

temac 5y ago

The problem is right now DDR4 devices that work correctly in that regard, do not exist. Likely the best mitigation for any sensitive application, even so lightly, is DDR4 with ECC (even if it may not be enough, it is vastly better than nothing, and not just because of rowhammer) And I have no idea if the internal "ECC" of standard DDR5 helps or not. It is not intended for regular ECC level of reliability anyway. (And I have seen discussion about likely bitflips detected in crash dumps of M1 Pro devices) So, as much as I would like to return defective devices, I would probably be left with no computer, no smartphone, etc.

jandrese 5y ago
Maybe a paranoid app could try to allocate the memory adjacent to any sensitive bits as a buffer? That would be pretty difficult to do but might be possible if you are tremendously paranoid and have a good way to examine the hardware.
- pas 5y ago
  Or use different banks of DRAM for different security contexts, managed via NUMA?
- to11mtm 5y ago
  I wonder how easy that would be to reason about though. I really don't know much about modern DRAM circuitry I feel like it might be abstracted away to some level, also there are bios-tweaky settings that might make a difference (i.e. things like channel configuration and/or bank interleaving, if that's still a thing.) IDK though. Maybe there's a way. I've worked on code that uses padding around a data structure to ensure it has it's own cache line. Maybe if you allocate a large enough contiguous block you'll be OK?
- mnw21cam 5y ago
  You'd have to have the operating system's cooperation, because it may be mapping individual 4k pages all over the place, and it's the only thing that has a chance of knowing how the memory is laid out on the actual chips.
  - AshamedCaptain 5y ago
    And the CPU and the chipset's cooperation. In many cases it is even a trade secret how data is striped across the different slots/banks.
    - xxpor 5y ago
      Doesn't the same thing to exploiting rowhammer though? If you're after specific data, how would you know what physical address to use, even if you knew the physical address of the target data?
      - jandrese 5y ago
        Yes, but there is a difference between an attack being run by a low level hardware hacker vs. the software being run by average users. This is why I said it would be hard, you would need to encode a lot of low level hardware knowledge into your application. This might only be useful in very specialized circumstances, like with kernel support on only a handful of carefully chosen hardware platforms. However, someone like Apple could do this on their systems, as they control both the kernel and the hardware. Sensitive memory locations could be cordoned off in special zones with buffering to prevent Rowhammer type attacks.
    - StillBored 5y ago
      <i> it is even a trade secret how data is striped across the different slots/banks </i> If that is true, vs just the usual "you don't need to know" trash common these days, its crazy. A pretty good picture can be built up with just software timing analysis, but its hardly rocket science to hook a logic analyzer/scope up and determine bit swizzling and page/bank interleave. Plus, many of the more open BIOS vendors have tended to provide page/row/controller/socket interleave options for quite a while, partially because it can mean a 5-10% uplift for some applications to have it set one way or the other. Its been one of those how do you tune your memory system options for a couple decades now.
- staticassertion 5y ago
  If your app is particularly paranoid you could just double your memory usage, and verify errors against the clone.
buran77 5y ago
According to older paper [0] ECC can also be bypassed after reverse-engineering the mitigation in DDR3 DIMMs. Also: > “DDR4 systems with ECC will likely be more exploitable, after reverse-engineering the ECC functions,” researchers Razavi and Jattke said > What if I have ECC-capable DIMMs? Previous work [1] showed that due to the large number of bit flips in current DDR4 devices, ECC cannot provide complete protection against Rowhammer but makes exploitation harder. [0] https://cs.vu.nl/~lcr220/ecc/ecc-rh-paper-eccploit-press-preprint.pdf https://cs.vu.nl/~lcr220/ecc/ecc-rh-paper-eccploit-press-pre... [1] https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8835222 https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8835222
- FpUser 5y ago
  While it can not protect from bit flips it can most likely warn about attack in progress due to a large number of bit flips.
myself248 5y ago
Yep, though I agree with the parent, if a RAM returns data from a location that is different from what is stored in that location, assuming all recommended timings have been followed, then that RAM is defective. If that means they should be recommending a refresh after every single operation, then that's what it means. In other words, the whole industry sits on a bed of lies at this point and it's only because government is technically incompetent that we haven't seen the world's biggest class-action.
- ygjb 5y ago
  > In other words, the whole industry sits on a bed of lies at this point and it's only because government is technically incompetent that we haven't seen the world's biggest class-action. That's a pretty bold claim - if it's accurate, can you point at the government regulation that precludes a class action lawsuit? I would assume it's something in the T&C or EULA for the hardware rather than a regulation?
  - userbinator 5y ago
    Bold but sadly correct. The industry runs on misdirection and deception. If users knew better, we'd have the equivalent of another Pentium FDIV bug.
- rzerda 5y ago
  The whole industry built this bed of lies by their own hands, I don’t see how it’s government’s fault.
indymike 5y ago
> The problem is right now DDR4 devices that work correctly in that regard, do not exist. Yes, and this is actually the problem. Without safe hardware, it is almost impossible to write safe software.
- userbinator 5y ago
  Meanwhile you can still buy DDR3 --- used --- from various shops in China, and it'll be 100% perfect and Rowhammer-free because it was made before the insane process shrinkages that caused it. (Have done this a few times. Was hesitant initially because of the low price and used nature, but when the first stick passed MemTest86+ including RH perfectly, I was convinced. Half a dozen more sticks later and still good. But maybe they've started to run out of the good old stuff now...)
  - inetknght 5y ago
    > you can still buy DDR3 --- used --- from various shops in China, and it'll be 100% perfect and Rowhammer-free because it was made before the insane process shrinkages that caused it That's demonstrably false. I remember rowhammer being thought of and demonstrated on DDR3 modules. [0] https://hammertux.github.io/rowhammer-ffs-ddr3 https://hammertux.github.io/rowhammer-ffs-ddr3
    - userbinator 5y ago
      Rowhammer started appearing in newer DDR3; see the original paper (figure 3): https://news.ycombinator.com/item?id=8713411 https://news.ycombinator.com/item?id=8713411 The tl;dr is that everything 2009 and older is good, 2010-2011 is possibly bad, 2012 and newer is almost certainly bad.
- throwaway81523 5y ago
  > Yes, and this is actually the problem. Without safe hardware, it is almost impossible to write safe software. Software that attempts a hostile rowhammer attack aims to be unsafe by definition. It's not a matter of trying and failing to be safe. Rather, we're seeing that it's very difficult to build hardware and software that prevents hostile code from running amuck: if the attacker can run hostile code on your computer at all, you have already lost. That's bad news for cloud hosts and confirms what we already knew about web scripting, but doesn't seem all THAT bad for the rest of us. Of course it's possible to build computers with no DRAM at all (i.e. only SRAM, which costs more). That is not economically sane for big servers, but might be the right thing to do for security sensitive components.
oopsyDoodl 5y ago
I think this illustrates the cloud is unacceptable for anything more than storage and retrieval. All computed results from data science must include steps and code to verify locally. It calls into question privacy on federated networks and crypto networks; any node can be manipulated locally to change payload outputs on delivery, reveal secrets, disrupt workloads. This makes sense to a lot of folks in computer engineering and physics, versus abstract software. No physical theory I know of offers any guarantee our arbitrary computing machines will ever be securable. We put fart pipes on Hondas. Science proves it’s titillating smoke and mirrors once again. Still waiting for nuclear rocket cars. I think this proves further as well why general computing chips need to be replaced with workload specific designs, where the anticipated inputs are well known and no vague logic paths to intentionally allow software monkey patching ever ship.
- Ysraes 5y ago
  > I think this illustrates the cloud is unacceptable for anything more than storage and retrieval. You can run in dedicated tenancy where you have the whole machine or a metal configuration where you also have the whole machine.
  - frankzinger 5y ago
    That would ruin providers' user:hardware ratios, one of the foundational principals of cloud computing.
    - Terretta 5y ago
      On the contrary, certain CSPs did this already (quietly) and further, had already developed hardware mitigations for things like meltdown, spectre and rowhammer.
      - frankzinger 5y ago
        But did they manage to keep their prices low? (And those who employed hardware mitigations wouldn't have a problem with user:hardware ratios, would they?)

jdmichal 5y ago

> I'd love to see this code in memory testers like memtest86 so I could send the RAM back if it ever caught a problem like this. Is this an individual stick bug, or a design bug akin to Spectre? Given that they tested 40 different sticks and seemed to find effective patterns for all of them make me think it's the latter. In which case, just return all your computers?

jandrese 5y ago
Yeah, this is a problem with high density DRAM. Fixes are possible but difficult and expensive. In the cutthroat DRAM market nobody seems willing to put in the engineering effort to develop a solution and then manufacture it. It would have to sell at a premium, in a market where it's difficult to even find ECC memory at times. The article states that ECC isn't sufficient to solve Rowhammer, but it does make the attack harder. If you are looking for a hardware solution it is your only option currently, even if it is imperfect.
- seiferteric 5y ago
  I imagine you are right that consumers don't care or know enough to put pressure on manufacturers, but what about big companies like the FANGs? Or Government agencies like NSA etc? What are they doing about it? I imagine intelligence agencies are well aware of the issue and have or are trying to get solutions to protect their own infra.
  - codetrotter 5y ago
    Off topic but shouldn’t FAANG really be referred to as MAAAN now? Google -> Alphabet Facebook -> Meta Meta, Apple, Amazon, Alphabet, Netflix. MAAAN.
    - somethingwitty1 5y ago
      Google still exists. Facebook still exists. Meta/Alphabet are mainly holding companies. For example, I received reach outs from recruiters for Google and Facebook today. Not from Meta and Alphabet. Maybe once Meta and Alphabet deliver on things and build brand recognition, it should change. Until then, I'd vote "no".

tbob22 5y ago

MemTest86 already has a rowhammer test, not sure how it compares to blacksmith but some sticks I have do fail that test and can only be mitigated by setting tREFI extremely low (while also taking a large performance hit). Most of the higher end hynix/micron/samsung sticks I've tried do not fail at JEDEC or XMP after 7+ passes.

mithro 5y ago

Google is working on platforms to make this easier to explore this problem, see https://opensource.googleblog.com/2021/11/Open%20source%20DDR%20controller%20framework%20for%20mitigating%20Rowhammer.html?m=1 https://opensource.googleblog.com/2021/11/Open%20source%20DD...

gjs278 5y ago

did you read the fucking headline? all ram is vulnerable to this. you don’t need to run the test or return anything. you won’t get a stick that isn’t vulnerable. jesus you are dumb

userbinator 5y ago

The problem is that apparently the authors of memory testing programs were somehow convinced to hide the severity of this issue when it first appeared, claiming things like it being a rare edge case and that too much RAM would fail. I believe newer versions of MemTest86+ have Rowhammer tests but it's disabled by default. See my previous comment on this matter: https://news.ycombinator.com/item?id=12410274 https://news.ycombinator.com/item?id=12410274

jacquesm 5y ago

You'll be sending back all of your RAM then.

usr1106 5y ago

> I'd love to see this code in memory testers like memtest86 so I could send the RAM back if it ever caught a problem like this. How do you run a memory tester after you have returned all CPUs after Meltdown and all the subsequent microarchicture bugs? I'd call all modern CPUs faulty products. Some more than others.

fy20 5y ago
If it's a problem with chips becoming higher density, I wonder if older hardware will become popular in circles that care more about security than performance per rack unit.