4 ms·
It's my long standing opinion[1] that we shouldn't accept any RAM that can be subject to random bit flips. Most of the mitigations are just security through obs
by mikewarot 2mo ago
It's my long standing opinion[1] that we shouldn't accept any RAM that can be subject to random bit flips. Most of the mitigations are just security through obscurity.
[1] https://news.ycombinator.com/item?id=33860477 https://news.ycombinator.com/item?id=33860477
- e2le 2mo agoI was shocked to learn that around 10% of Firefox crashes are caused by bit flips. https://news.ycombinator.com/item?id=47252971 https://news.ycombinator.com/item?id=47252971
- inigyou 2mo agoMost software is like this. Factorio had a weird one where some routers dropped UDP packets with zero checksum.
- Retr0id 2mo agoIntel MEE addressed this (https://eprint.iacr.org/2016/204.pdf https://eprint.iacr.org/2016/204.pdf), by treating DRAM as entirely untrusted (assuming an attacker with arbitrary physical read/write capabilities), but then they discontinued it. Regarding your linked comment: > If we had properly tested and validated RAM, RowHammer wouldn't work, ever. While it's exacerbated by physical defects and tight timings, it's really a fundamental problem with how DRAM works. It's frankly a miracle it works in the first place.
- inigyou 2mo agoNo. If it was designed really well, it would trigger extra refreshes as needed. It would know how much accessing a row disturbs nearby rows and how much disturbance a row can suffer before it needs a refresh to prevent data corruption. It would know how long a row can be held open for, too. There is no fundamental theorem that rowhammer must work. It was an engineering tradeoff. I don't think we can blame DRAM designers for flying a teensy bit too close to the sun here, since this is no problem in normal operation and only appears under adversarial scenarios. We can blame them for not fixing it once it was discovered. And we can definitely blame Intel for making ECC RAM a market segmentation feature.
- Retr0id 2mo agoWhat you describe is the Target Row Refresh (TRR) mitigation, and it can be bypassed via "half-double" attack.
- inigyou 2mo agoWhere did I imply that they should use an incorrect underestimation of row disturbance? If they actually measured these things, they'd know how far away a disturbance can be created before it becomes so weak the normal refresh cycle fixes it.
- Retr0id 2mo agoIt is not feasible for DRAM (or the memory controller) to maintain a fully accurate simulation of DRAM. If it was easy they'd have done it.
- inigyou 2mo agoOkay then refresh the whole thing and not just a single row whenever TRR is triggered. Or refresh the whole thing after a certain number of row activations. Or interleave activations with refreshes. Say, after every 5 activations, refresh the next row in the refresh cycle. I'm not a DRAM expert. They can figure it out. I promise if you refresh the whole chip after every row activation you won't have rowhammer. It'll be too slow though. Somewhere in between is the fastest point where there isn't rowhammer.
- deleted 2mo ago[deleted]
- reliabilityguy 2mo ago> Okay then refresh the whole thing and not just a single row whenever TRR is triggered. How would you refresh the whole DRAM? Refresh is simply reading the row and writing it back, it is sequential in nature. > Or interleave activations with refreshes. Say, after every 5 activations, refresh the next row in the refresh cycle. It makes zero sense. Refresh is disturbance in itself. You are entering a recursion here where the mere act of refreshing increases the counter values of victim rows making refresh even more frequent. At the end you have DRAM that you cannot read from or write to at all because it’s always refreshes itself. > I'm not a DRAM expert Yes
- anon_rh_guy 2mo agoLittle known fun fact: the MEE's checksum updates... were one of the most successful Rowhammer patterns... :-p
- Retr0id 2mo agoInteresting - but presumably used to attack rows outside of SGX?
- anon_rh_guy 2mo agoboth, inside and outside
- inigyou 2mo agoMy computer with a lot of ECC DDR5 sometimes catches several bitflips in a day. I know this because they are reported to dmesg and sometimes I look at dmesg. Intel decided long ago that you would need to pay more to not be subject to random bitflips. So it's an AMD system.
- userbinator 2mo agoMy computer with a lot of ECC DDR5 sometimes catches several bitflips in a day. Replace your RAM or find what's causing this disturbance in the environment. Sooner or later you'll get a pattern the ECC won't be able to correct. Ironically if you didn't have ECC you would instantly RMA such RAM as defective because even a simple memtest would quickly discover the problem, and "several bitflips in a day" would cause very obvious crashes all the time.
- execution 2mo agoAgreed, my DIY NAS with ECC DDR4 has ran for over 5 years and I have 0 ECC errors in that time. Something must be very wrong with OP's setup. Perhaps the RAM has been overclocked too far.
- inigyou 2mo agoWell that's DDR4. Apparently DDR5 was made with tighter tolerances and less safety margin, which is why many people argue ECC should be mandatory in DDR5 - it actually relies on it.
- userbinator 2mo agoThere are plenty of DDR5 machines without ECC[1], and yet they're not constantly crashing in the way that frequent bitflips would suggest. [1] DDR5's built-in ECC is not the same as traditional "end to end" ECC, and is meant to correct bitflips on the die, not between the RAM and the CPU, so ECC errors that you see are actually suggestive of a bad connection between the RAM and CPU.
- reliabilityguy 2mo ago> It's my long standing opinion[1] that we shouldn't accept any RAM that can be subject to random bit flips. Every system fails eventually.
- anonymous_user9 2mo agoYes things fail, but currently random bit flips are not considered a failure.
- reliabilityguy 2mo agoThey are, that’s why ECC exists, and those who are willing to pay for it, can get it.
- adrian_b 2mo agoThe big companies consider them a failure in their own computers, so they use appropriate memory, but they consider that "consumers" either must accept them as normal and not a failure, or pay a great overprice in comparison with what a big company pays, in order to have memory devices with error detection.
- adrian_b 2mo agoThat is why every subsystem must have means to detect the failures, to allow the replacement of the bad components. ECC in DIMMs achieves this. For myself, this has been the greatest advantage of always using ECC, that I have been warned early about the great increase in the frequency of errors in some DIMMs, after many years (over 5 years) of continuous usage, which has allowed me to replace the offending DIMMs and continue to use those systems for some more years.
- userbinator 2mo ago100% agree. This is a pure case of defective product. Memory not behaving like memory. It's very telling that when Rowhammer first appeared, authors of popular memory testing utilities added tests for it, and then quickly hid and disabled them by default because "too much RAM would test defective". I have no idea how they were convinced to do so.
- alightsoul 2mo agoMaybe they don't want to be seen as the boy who cried wolf? Or people would think it's a lie because it would still work for web browsing?
- SideQuark 2mo agoSingle event upsets are forced on you by physics. There is no possible way to stop them all.
- Dylan16807 2mo ago> Most of the mitigations are just security through obscurity. When manufacturers are sufficiently lazy. If you put a refresh counter into each memory row at like <1% overhead, you can actually guarantee that a thousand or a hundred accesses forces nearby cells to refresh and stay well inside their safety margin.