9 ms·
Flipping bits in memory without accessing them [pdf]
- edwintorok 12y agoWhy doesn't it mention the manufacturers' names?
- mud_dauber 12y agoThe work may have been funded by the manufacturers in return for the analysis and discretion. There's not that many DRAM manufacturers in the world. Micron, Samsung, Elpida, Hynix are all excellent bets.
- tsukikage 12y agoSo, busywaiting on spinlocks considered dangerous?
- nitrogen 12y agoThat would probably read from cache most of the time.
- BetaCygni 12y agoExcellent article! The fact that they can reliably produce errors in most ram chips is worrying. They also provide a solution (probabilistic refresh of neighboring lines).
- userbinator 12y agoThe scary thing is that such a "solution" could be silently disabled, with effectively no signs of any problem (and in fact it would probably increase performance a little!) - I mentioned this in the previous discussion here: https://news.ycombinator.com/item?id=8716977 https://news.ycombinator.com/item?id=8716977 I know there are other parameters of the memory controller that could be changed to cause corruption, e.g. reducing the refresh rate or tweaking the timings, but that is likely to yield random corruptions in normal use instead of this precise one. To me, the real solution seems to be stop making DRAM with such high density processes until some design changes can be done that make them as reliable as before, because at some point it just stops behaving like real memory anymore and turns into a crude approximation of it; memory should be reliable and store the data it holds, without any corruption regardless of access pattern.
- jadc 12y agoDiscussion from a couple of weeks ago: https://news.ycombinator.com/item?id=8713411 https://news.ycombinator.com/item?id=8713411
- ColinWright 12y agoMissed that - thanks - although to be fair there doesn't actually seem to be that much discussion. Maybe most people missed it. Edit: Just checking, it was at 120, looks like it got artificially promoted to 25, then sank quickly. http://hnrankings.info/8713411/ http://hnrankings.info/8713411/
- wdewind 12y ago> looks like it got artificially promoted to 25 How does that happen?
- ColinWright 12y agoThe mods have in place mechanisms to find submissions they think have slipped through the net and provide a boost for them. It's experimental and undocumented, but I saw it mentioned a while ago[0]. I think it's a good thing, even if it's somewhat arbitrary and has multiple false-positives and false-negatives. I've always been a fan of stochastic processes - they exhibit lots of good behavior, and comparatively few pathological flaws. [0] https://news.ycombinator.com/item?id=8157698 https://news.ycombinator.com/item?id=8157698
- wdewind 12y agoHonestly I'm bummed to see a lot of the recent aggressive moderation. I know the community is far from perfect, but seeing arbitrary posts moved to the front page, existing stuff on the front page change not only in title but also in the article its linked to after existing discussion have happened etc. all strike me as obviously worse than just letting the chaos of the community sort itself out.
- bhouston 12y agoHow long until someone uses this as the basis of an exploit? Maybe not root access, but if you can figure out an OS call that replicates the access pattern, you can corrupt machines just by interacting with them.
- ajross 12y agoThere's no need for an OS call. Just userspace access to the same mapped memory is going to stay in the same physical page for far, far, longer than a few hundred thousand DRAM cycles. Obviously the hard part to an exploit would be locating those corrupt bits elsewhere in the system. That's going to depend entirely on the hardware layout of the DRAM chip.
- witty_username 12y agoAnd I guess ASLR would make it even harder.
- johnsmith108959 12y agoASLR only affects the virtual address space. The physical memory allocations are all probably unaffected by ASLR.
- userbinator 12y agoNot really; if the blocks of memory you allocate are not the desired distance apart, just try again... or allocate a block big enough to guarantee it, then start the alternate read sequence to trigger corruption. Of course this assumes you can already run your code on the machine e.g. in a VM.
- im3w1l 12y agoI'd start looking at this for ideas. It's a paper on using memory erros to break out of a java virtual machine. They didn't have a good way for generating them, so resorted to waiting. https://www.cs.princeton.edu/~appel/papers/memerr.pdf https://www.cs.princeton.edu/~appel/papers/memerr.pdf
- 12y ago
- diydsp 12y agoFrom abstract: High speed DRAM reads influence nearby cells. Reproduced on 110 of 139 mem modules after 139k reads on intel and amd. 1 in 1.7k cells affected.
- kabdib 12y agoWhat goes on at the chip level is terrifying. "You have to understand," a hardware engineer once said to me, when we shipped a consumer computer that was clocking its memory system 15% faster than the chips were supposed to go, "that DRAMs are essentially analog devices." He was pushing them to the limit, but he knew what it was, and we never had a problem with the memory system. There was a great TR from IBM describing memory system design for one of their PowerPC chips. Summary: Do your board layout, follow all these rules [a big list], then plan to spend six months in a lab twiddling transmission line parameters and re-doing layout until you're sure it works . . .
- hga 12y agoTrue horror story: The first board I was involved with when working at Lucent in 2001 was a monster modem card, provided something like 300 modems plus or minus. For a long time in the development process we had a weird reliability problem which just could not be tracked down. Until in desperation a hardware engineer started putting his scope probe everywhere ... and found a line (or set of them) where the signal was messed up in the middle but not at the end!!! Analog is a black art.
- Kliment 12y agoThere's a lovely book on high-speed signaling subtitled "a handbook of black magic". I always found it very fitting. It explains the rules and heuristics and methods of working around and exploiting the analog nature of digital signals.
- mbell 12y agoI highly recommend that book as well, I learned more about signal integrity from reading it twice than I did during all of EE undergrad. Link: http://www.amazon.com/High-Speed-Digital-Design-Handbook/dp/0133957241 http://www.amazon.com/High-Speed-Digital-Design-Handbook/dp/...
- mud_dauber 12y ago
- pera 12y agoSo basically you just make a couple memory reads a few hundred thousands times and this will alter some near cell? why manufacturers didn't test this? it looks like a pretty obvious thing to test while working at these scales.
- mud_dauber 12y agob/c the applications that care the most about these types of errors (big iron networking, and mil/aero) implement software error correction in the processor. It's not worth spending the extra money in final test when commodity DRAM is, well, a commodity.
- johnsmith108959 12y agoYes, ECC memory, although it's mostly implemented by the chipset and DRAM modules, with only a small part of it being in the processor.
- mud_dauber 12y agoThanks for the clarification John.
- userbinator 12y agocommodity DRAM is, well, a commodity. You're essentially saying that producing DRAM that doesn't work like memory should, regardless of access pattern, is excusable because it's "commodity"?
- blazespin 12y agoSounds like a great technique for ddos.
- hammer_test 12y agoNote: the regular memtest+ doesn't have this test. Use the researcher fork: https://github.com/CMU-SAFARI/rowhammer https://github.com/CMU-SAFARI/rowhammer in Ubuntu 14.04, run this to bring all the dependencies for building: sudo apt-get build-dep memtest86+ Update: just finished running the test on my cheap Lenovo laptop. Not affected. phew! :)
- deleted 12y ago[deleted]
- Tobu 12y agoIt's been upstreamed, hasn't it? Announcement http://www.passmark.com/forum/showthread.php?4836-MemTest86-v6-0-Beta http://www.passmark.com/forum/showthread.php?4836-MemTest86-... Previous discussion https://news.ycombinator.com/item?id=8713411 https://news.ycombinator.com/item?id=8713411
- deleted 12y ago[deleted]
- markbnj 12y agoThis article actually contains one of the better-written fundamental explanations of DRAM operation that I've read. Thanks for the post.
- blinkingled 12y agoHopefully this doesn't affect ECC DRAM? Also does the problem get worse with increased density - i.e. 16GB modules are more vulnerable than say the 8GB ones?
- twotwotwo 12y agoIt helps, but they point out you can get two-bit errors ECC can merely detect (or, much more rarely, 3+-bit errors ECC isn't guaranteed to even detect). Mighty tricky to work out that exploit that flips just the right two security-relevant bits, though.
- phkahler 12y agoProblem: They propose a solution and calculate the reliability of the solution. Why not test it with their FPGA based memory controller and demonstrate an improvement? Second: While the problem looks real enough, the tests to demonstrate it are not realistic. Hammering the same rows with consecutive reads does not happen in the real world due to caches which the get around via flushes. I'd like to see more data on how bad the abuse needs to be to cause the problem. Will 2 reads in a row cause errors? 5? 10? 100? They never address how likely this is to be a real-world problem. I don't doubt that it is, but how often?
- hammer_test 12y agoIt's quite easy to bypass cache without explicit flush. Most caches are 4-way or 8-way: just hit 4 or 8 addresses sharing the same cache line, and you got your original data evicted from cache.
- xenadu02 12y agoThey address all of these in the paper. It takes ~180,000 "hammered" reads to trigger. The problem is even the least-privileged code can do it because it's just reading without using the cache - a perfectly valid thing that must be allowed for multi-threaded code to even work correctly. Secondly, the DRAM makers don't currently provide enough information to reliably know what neighbors to refresh. I suppose they could have used their guesses to test on the FPGA rig but given the rest of the paper I'm reasonably satisfied that they have correctly identified the problem and that their solution would work.
- kabdib 12y agoSometimes an exploit simply needs to make more trusted code take an exception (often after "paving" memory with known values). Since an exception can be theoretically be induced by user-mode code flipping bits in structures it doesn't own . . . I can see "exploit resistant memory" being a selling point, maybe soon.
- caf 12y agoMulti-threaded code does not need cache flushes / invalidating reads to work correctly, on any halfway modern architecture that includes a hardware cache coherence protocol.
- zanethomas 12y agoThe success of this approach to corrupting memory depends upon knowing the geometry of the memory chip. Naive calculations of which addresses correspond to an adjacent row may be incorrect. It's interesting to see this issue addressed in 2015. In 1980 I worked at Alpha Microsystems and designed a memory chip test program which used translation tables based upon information we required chip manufacturers to give us in order for their chips to be used in the systems we sold. That approach required us to only put one type of memory chip on a memory board. But back in the day microsytems were expensive and customers expected them to be well-tested.
- hga 12y agoMy family thanks you and your colleagues. Back in mid-1979, just before I went to college, I looked at the current alternatives and picked Alpha Micro as the company to go with for a system to computerize a bunch of doctor's offices. It worked very well, and then another one a few years later when they used one to help systematize a company providing satellite TV gear.
- mseaborn 12y agoActually, triggering rowhammer-induced bit flips does not require knowing the memory geometry. It is possible to trigger bit flips just by picking random pairs of addresses to hammer. See the README file at https://github.com/mseaborn/rowhammer-test https://github.com/mseaborn/rowhammer-test.
- rebootthesystem 12y agoI've completed many multi-gigahertz product designs during my career. If you take the time to study and understand the physics involved and bother to do a bit of math none of it is particularly difficult. I reject the characterization of this as some kind of a black art. It's not magic. Yes, of course, experience helps, but it isn't magic. One problem is that some in the industry are still using people who do layout based on how things look rather than through a scientific process. Yes, it's analog electronics. When was it anything else? Want to wrap yourself around another challenging aspect of high-speed design? Power distribution system design (PDS). You can design perfect boards based on solid transmission line and RF theory and have them fail to work due to issues such as frequency-dependent impedances and resonance in the PDS.
- jhowe 12y agoI work in the chip industry. This was a good paper. 1. Note that chip-kill/Extended ECC/Advanced ECC/Chipspare which are all similar server vendor methods for 4-bit correction will prevent this problem. These methods are enabled on the better reliability server systems. 2. This failure mode has been known by the DRAM industry for a couple years now and the newest DRAM parts being produced have this problem solved. The exact solution varies by DRAM vendor. I wish I could go into specifics but I am unaware of any vendor that has stated publicly their fix.
- userbinator 12y agothe newest DRAM parts being produced have this problem solved How new exactly? The newest tested in the paper is from July 2014 and that still has the problem.
- rab_oof 12y agoYou're more than right. In fact, the paper explicitly mentions 4 correct 5 detect (but fail) doesn't solve the issue because each victim row on either side of the attacker row attains varying levels of multibit (5+) errors. It doesn't fix systems deployed right now, and could be used for attacking hypervisors and other multitenant systems. Might make an interesting class of local privilege escalation attacks to try probabilistically of otherwise correctly-secured systems too.
- kazinator 12y agoInterestingly, the researchers used a Xilinx FPGA, not just an off-the-shelf AMD or Intel PC. Why not? If the attack can only be reproduced by custom hardware, why should anyone care? Also, precise patterns of access to DRAM would require disabling the L1 and L2 caches. Doesn't that sort of thing require privileged instructions? With caching in place, memory accesses are indirect. You have to be able to reproduce the attack using only patterns of cache line loads and spills.
- kmowery 12y agoNo, they reproduced on "Intel (Sandy Bridge, Ivy Bridge, and Haswell) and AMD (Piledriver) systems using a 2GB DDR3 module." (see Section 4) They evict cache lines using the CLFLUSH x86 instruction, which I believe is unprivileged.
- dllthomas 12y agoCLFLUSH is definitely unprivileged - I made use of it on a recent project (evicting outbound messages from a core's cache cut cash misses meaningfully).
- jhallenworld 12y agoI think row hammer is basically a DRAM design defect and wish it was fixed in the DRAM instead of on the controller side. At the very least the DRAM vendors should document this access pattern limitation in their datasheets.
- mseaborn 12y agoHere is a program for testing for the DRAM rowhammer problem which runs as a normal userland process: https://github.com/mseaborn/rowhammer-test https://github.com/mseaborn/rowhammer-test Note that for the test to do row hammering effectively, it must pick two addresses that are in different rows but in the same bank. A good way of doing that is just to pick random pairs of addresses. If your machine has 16 banks of DRAM, for example (as various machines I've tested do), there should be a 1/16 chance that the two addresses are in the same bank. This is what the test above does. (Actually, it picks >2 addresses to hammer per iteration.) Be careful about running the test, because on machines that are susceptible to rowhammer, it could cause bit flips that crash the machine (or worse, bit flips in data that gets written back to disc).
- rab_oof 12y agoAm I wrong (if you happen to work on processor microcode) or could microcode patches per processor insert a minimum delay where needed based on RAM parameters and organization to prevent this?