4 ms·

from TFA The researchers were also able to flip the bits inside DDR3 DIMMs installed on an enterprise-grade server. The tests succeeded even though all of the

from TFA The researchers were also able to flip the bits inside DDR3 DIMMs installed on an enterprise-grade server. The tests succeeded even though all of the DDR3 modules included a protection known as ECC, the servers completely locked up or spontaneously rebooted, usually within three minutes of the tests commencing.

ori_b 11y ago
Yes; Locking up or rebooting seems like a reasonable thing to do if the memory has multiple unrecoverable errors. It certainly beats silent corruption.
- roywiggins 11y ago
  But the failure mode for ECC is silent corruption, followed (maybe) by a crash, right? ECC can detect and correct one or two (or a few more?) bit-flips at a time, but if there's too many it won't notice that something's gone wrong. The system then goes merrily along and either crashes (because you've flipped some bits at random) or recovers (because those bits weren't important) or hands the attacker root (because you've managed to flip some particular target bits). So if the computer crashes or locks up, you can probably conclude that you've defeated the ECC and flipped some bits, and the system is vulnerable to Rowhammer. It's (maybe) just a matter of time before someone engineers a way to flip the specific bits needed to hand the attacker root.
  - GauntletWizard 11y ago
    The memory can recover one, detect two, and will throw up loud warning bells to any competent admin as long as it is doing so. If you see a sudden spike of ECC errors from a server, the memory is probably bad, and you replace it. If you see a sudden spike of ECC errors from a large number of servers, something is seriously wrong, and you investigate - I'd be more likely to blame it on bad power myself, or a EM Field/radiation source. Even when trying to trigger rowhammer, the graph of bit flips is going to look like a logarithmic curve - Single-bit flips will be common, two-bits will be rare, and undetected flips will be real but ultimately not much to talk about compared to the discovered flips. You know they're there by the unrecoverable rate, and you replace the ram, discard any replicated data on that machine, and move on.
    - nkurz 11y ago
      The memory can recover one, detect two, and will throw up loud warning bells to any competent admin as long as it is doing so. I don't have personal experience here, but one of the important claims in the paper is that this warning is not given on all servers: Unfortunately, server vendors routinely use a technique called ECC threshold or the 'leaky bucket' algorithm where they count ECC errors for a period of time and report them only if they reach certain levels of failure. From what we understand, this threshold is commonly above 100 per hour, but this remains a trade secret and varies based on the server vendor. So, to see ECC errors (MCE in Linux or WHEA in Windows), there generally needs to be 100 bit flips per hour or greater. This makes “seeing” Rowhammer on server error logs more difficult. In addition, we have observed some server vendors will NEVER report ECC events back to the OS,although they might get logged into IPMI. Typically, users expect to see correctable ECC errors logged directly to the OS or that halt the system when they cannot be corrected. During our investigation into this phenomenon, we even encountered one server that neither reported ECC events to the OS nor halted when bit flips were not correctable. The end result was data corruption at the application level. This is something, in our opinion, that should never happen on an ECC protected server system. http://www.thirdio.com/rowhammer.pdf http://www.thirdio.com/rowhammer.pdf
      - baruch 11y ago
        On the servers that I use the server will log to ipmi sel only after a threshold but the Linux OS will get an MCE for each error so a monitoring tool can detect and alert even before the BIOS logs anything.
      - BuildTheRobots 11y ago
        I really wish they'd have expanded on that somewhere and actually stated what manufacturers/models of server they tested and which ones don't report ECC errors.
        brainfire 11y ago
        Really, without details like that it's just unverifiable FUD.
  - snuxoll 11y ago
    To be fair, the expected failure mode of unrecoverable memory corruption in kernel space should probably be a panic followed by a reboot. Unfortunately, as you said, if you manage to corrupt more than two bits in a row it's a crapshoot if ECC DRAM will even detect the corruption or not.
  - cesarb 11y ago
    > ECC can detect and correct one or two (or a few more?) bit-flips at a time, but if there's too many it won't notice that something's gone wrong. ECC can normally detect and correct a single bit flip, and detect but not correct a double bit flip, in the "line" of bits protected by it. The ECC protection is local, not global; if there are many flipped bits but they are far enough from each other, each "line" will only have a correctable single-bit flip.

ori_b 11y ago

Yes; Locking up or rebooting seems like a reasonable thing to do if the memory has multiple unrecoverable errors. It certainly beats silent corruption.

roywiggins 11y ago
But the failure mode for ECC is silent corruption, followed (maybe) by a crash, right? ECC can detect and correct one or two (or a few more?) bit-flips at a time, but if there's too many it won't notice that something's gone wrong. The system then goes merrily along and either crashes (because you've flipped some bits at random) or recovers (because those bits weren't important) or hands the attacker root (because you've managed to flip some particular target bits). So if the computer crashes or locks up, you can probably conclude that you've defeated the ECC and flipped some bits, and the system is vulnerable to Rowhammer. It's (maybe) just a matter of time before someone engineers a way to flip the specific bits needed to hand the attacker root.
- GauntletWizard 11y ago
  The memory can recover one, detect two, and will throw up loud warning bells to any competent admin as long as it is doing so. If you see a sudden spike of ECC errors from a server, the memory is probably bad, and you replace it. If you see a sudden spike of ECC errors from a large number of servers, something is seriously wrong, and you investigate - I'd be more likely to blame it on bad power myself, or a EM Field/radiation source. Even when trying to trigger rowhammer, the graph of bit flips is going to look like a logarithmic curve - Single-bit flips will be common, two-bits will be rare, and undetected flips will be real but ultimately not much to talk about compared to the discovered flips. You know they're there by the unrecoverable rate, and you replace the ram, discard any replicated data on that machine, and move on.
  - nkurz 11y ago
    The memory can recover one, detect two, and will throw up loud warning bells to any competent admin as long as it is doing so. I don't have personal experience here, but one of the important claims in the paper is that this warning is not given on all servers: Unfortunately, server vendors routinely use a technique called ECC threshold or the 'leaky bucket' algorithm where they count ECC errors for a period of time and report them only if they reach certain levels of failure. From what we understand, this threshold is commonly above 100 per hour, but this remains a trade secret and varies based on the server vendor. So, to see ECC errors (MCE in Linux or WHEA in Windows), there generally needs to be 100 bit flips per hour or greater. This makes “seeing” Rowhammer on server error logs more difficult. In addition, we have observed some server vendors will NEVER report ECC events back to the OS,although they might get logged into IPMI. Typically, users expect to see correctable ECC errors logged directly to the OS or that halt the system when they cannot be corrected. During our investigation into this phenomenon, we even encountered one server that neither reported ECC events to the OS nor halted when bit flips were not correctable. The end result was data corruption at the application level. This is something, in our opinion, that should never happen on an ECC protected server system. http://www.thirdio.com/rowhammer.pdf http://www.thirdio.com/rowhammer.pdf
    - baruch 11y ago
      On the servers that I use the server will log to ipmi sel only after a threshold but the Linux OS will get an MCE for each error so a monitoring tool can detect and alert even before the BIOS logs anything.
    - BuildTheRobots 11y ago
      I really wish they'd have expanded on that somewhere and actually stated what manufacturers/models of server they tested and which ones don't report ECC errors.
      - brainfire 11y ago
        Really, without details like that it's just unverifiable FUD.
- snuxoll 11y ago
  To be fair, the expected failure mode of unrecoverable memory corruption in kernel space should probably be a panic followed by a reboot. Unfortunately, as you said, if you manage to corrupt more than two bits in a row it's a crapshoot if ECC DRAM will even detect the corruption or not.
- cesarb 11y ago
  > ECC can detect and correct one or two (or a few more?) bit-flips at a time, but if there's too many it won't notice that something's gone wrong. ECC can normally detect and correct a single bit flip, and detect but not correct a double bit flip, in the "line" of bits protected by it. The ECC protection is local, not global; if there are many flipped bits but they are far enough from each other, each "line" will only have a correctable single-bit flip.