4 ms·

So. Time to move to ECC by default?

by ori_b 11y ago

greglindahl 11y ago
ECC just makes the attack take longer.
- tdullien 11y ago
  Co-author of the original exploit writing here. ECC doesn't "make the attack take longer", it turns it from an exploitable security issue into a denial-of-service issue (as odds are the ECC will detect the bitflip and the kernel will shut down the box).
  - nickpsecurity 11y ago
    I noticed supercomputer vendors and Oracle are using ChipKill plus ECC. Does adding that change the results any or still reduce to DOS?
    - tdullien 11y ago
      So I do not know (in the mathematical sense) that ChipKill won't create a problem, but I think it is unlikely. In general, if the ECC infrastructure works (e.g. DRAM detecting bit corruption and reporting it to the OS), the odds of an attacker finding a piece of RAM that flips bits "just right" to bypass the checksumming without triggering a OS panic are low enough that I'd rather worry about software bugs. DoS may still be a pretty viable attack, though.
    - baruch 11y ago
      It can mitigate the DoS if you have enough spare RAM. if you are right on the edge that removed RAM will turn into an OOM condition and will become a DoS.
    - darkr 11y ago
      Not just supercomputer vendors - Chipkill has been a standard option on IBM X-Series (their standard server range) for years now. I remember deploying a whole bunch of servers back 2011 or so that had it.
    - nickpsecurity 11y ago
      Appreciate everyone's insightful replies. :)
  - greglindahl 11y ago
    Then why's the paper[1] say different? The odds of a double-flip are smaller than s single-flip; a single flip gets corrected, a double stops the box (depending on the bios etc), an even rarer triple may not get detected... the odds are exactly why it just takes longer. This paper reports observing it happening. 1: http://www.thirdio.com/rowhammer.pdf http://www.thirdio.com/rowhammer.pdf
    - nkurz 11y ago
      I presume 'tdullien' assumes that "attack" refers to privilege escalation. I think he's claiming you are unlikely to get an undetected and exploitable triple bit flip before you crash the machine with the more single and double bit flips. Thus he says that ECC "turns it from an exploitable security issue into a denial-of-service issue". Does the paper claim that in fact triple bit flips are sufficiently more frequent than one would expect, and thus an exploitable security issue still exists with ECC? This would be interesting news. But if you are defining "attack" to include denial-of-service by crashing the machine, I presume the disagreement is only about terminology.
  - rodgerd 11y ago
    That could still be extraordinarily entertaining for an attacker to execute on a shared hosting environment, though.
SixSigma 11y ago
from TFA The researchers were also able to flip the bits inside DDR3 DIMMs installed on an enterprise-grade server. The tests succeeded even though all of the DDR3 modules included a protection known as ECC, the servers completely locked up or spontaneously rebooted, usually within three minutes of the tests commencing.
- ori_b 11y ago
  Yes; Locking up or rebooting seems like a reasonable thing to do if the memory has multiple unrecoverable errors. It certainly beats silent corruption.
  - roywiggins 11y ago
    But the failure mode for ECC is silent corruption, followed (maybe) by a crash, right? ECC can detect and correct one or two (or a few more?) bit-flips at a time, but if there's too many it won't notice that something's gone wrong. The system then goes merrily along and either crashes (because you've flipped some bits at random) or recovers (because those bits weren't important) or hands the attacker root (because you've managed to flip some particular target bits). So if the computer crashes or locks up, you can probably conclude that you've defeated the ECC and flipped some bits, and the system is vulnerable to Rowhammer. It's (maybe) just a matter of time before someone engineers a way to flip the specific bits needed to hand the attacker root.
    - GauntletWizard 11y ago
      The memory can recover one, detect two, and will throw up loud warning bells to any competent admin as long as it is doing so. If you see a sudden spike of ECC errors from a server, the memory is probably bad, and you replace it. If you see a sudden spike of ECC errors from a large number of servers, something is seriously wrong, and you investigate - I'd be more likely to blame it on bad power myself, or a EM Field/radiation source. Even when trying to trigger rowhammer, the graph of bit flips is going to look like a logarithmic curve - Single-bit flips will be common, two-bits will be rare, and undetected flips will be real but ultimately not much to talk about compared to the discovered flips. You know they're there by the unrecoverable rate, and you replace the ram, discard any replicated data on that machine, and move on.
deleted 11y ago
[deleted]

So. Time to move to ECC by default?

by ori_b 11y ago

So. Time to move to ECC by default?

greglindahl 11y ago

ECC just makes the attack take longer.

tdullien 11y ago
Co-author of the original exploit writing here. ECC doesn't "make the attack take longer", it turns it from an exploitable security issue into a denial-of-service issue (as odds are the ECC will detect the bitflip and the kernel will shut down the box).
- nickpsecurity 11y ago
  I noticed supercomputer vendors and Oracle are using ChipKill plus ECC. Does adding that change the results any or still reduce to DOS?
  - tdullien 11y ago
    So I do not know (in the mathematical sense) that ChipKill won't create a problem, but I think it is unlikely. In general, if the ECC infrastructure works (e.g. DRAM detecting bit corruption and reporting it to the OS), the odds of an attacker finding a piece of RAM that flips bits "just right" to bypass the checksumming without triggering a OS panic are low enough that I'd rather worry about software bugs. DoS may still be a pretty viable attack, though.
  - baruch 11y ago
    It can mitigate the DoS if you have enough spare RAM. if you are right on the edge that removed RAM will turn into an OOM condition and will become a DoS.
  - darkr 11y ago
    Not just supercomputer vendors - Chipkill has been a standard option on IBM X-Series (their standard server range) for years now. I remember deploying a whole bunch of servers back 2011 or so that had it.
  - nickpsecurity 11y ago
    Appreciate everyone's insightful replies. :)
- greglindahl 11y ago
  Then why's the paper[1] say different? The odds of a double-flip are smaller than s single-flip; a single flip gets corrected, a double stops the box (depending on the bios etc), an even rarer triple may not get detected... the odds are exactly why it just takes longer. This paper reports observing it happening. 1: http://www.thirdio.com/rowhammer.pdf http://www.thirdio.com/rowhammer.pdf
  - nkurz 11y ago
    I presume 'tdullien' assumes that "attack" refers to privilege escalation. I think he's claiming you are unlikely to get an undetected and exploitable triple bit flip before you crash the machine with the more single and double bit flips. Thus he says that ECC "turns it from an exploitable security issue into a denial-of-service issue". Does the paper claim that in fact triple bit flips are sufficiently more frequent than one would expect, and thus an exploitable security issue still exists with ECC? This would be interesting news. But if you are defining "attack" to include denial-of-service by crashing the machine, I presume the disagreement is only about terminology.
- rodgerd 11y ago
  That could still be extraordinarily entertaining for an attacker to execute on a shared hosting environment, though.

SixSigma 11y ago

from TFA The researchers were also able to flip the bits inside DDR3 DIMMs installed on an enterprise-grade server. The tests succeeded even though all of the DDR3 modules included a protection known as ECC, the servers completely locked up or spontaneously rebooted, usually within three minutes of the tests commencing.

ori_b 11y ago
Yes; Locking up or rebooting seems like a reasonable thing to do if the memory has multiple unrecoverable errors. It certainly beats silent corruption.
- roywiggins 11y ago
  But the failure mode for ECC is silent corruption, followed (maybe) by a crash, right? ECC can detect and correct one or two (or a few more?) bit-flips at a time, but if there's too many it won't notice that something's gone wrong. The system then goes merrily along and either crashes (because you've flipped some bits at random) or recovers (because those bits weren't important) or hands the attacker root (because you've managed to flip some particular target bits). So if the computer crashes or locks up, you can probably conclude that you've defeated the ECC and flipped some bits, and the system is vulnerable to Rowhammer. It's (maybe) just a matter of time before someone engineers a way to flip the specific bits needed to hand the attacker root.
  - GauntletWizard 11y ago
    The memory can recover one, detect two, and will throw up loud warning bells to any competent admin as long as it is doing so. If you see a sudden spike of ECC errors from a server, the memory is probably bad, and you replace it. If you see a sudden spike of ECC errors from a large number of servers, something is seriously wrong, and you investigate - I'd be more likely to blame it on bad power myself, or a EM Field/radiation source. Even when trying to trigger rowhammer, the graph of bit flips is going to look like a logarithmic curve - Single-bit flips will be common, two-bits will be rare, and undetected flips will be real but ultimately not much to talk about compared to the discovered flips. You know they're there by the unrecoverable rate, and you replace the ram, discard any replicated data on that machine, and move on.

deleted 11y ago

[deleted]