10 ms·
DRAM chip failures reveal surprising hardware vulnerabilities
- PhantomGremlin 11y agoThey make an interesting claim: Between 12 percent and 45 percent of machines at Google experience at least one DRAM error per year. This is orders of magnitude more frequent than earlier estimates had suggested. We just had a big discussion on this topic a few days ago. https://news.ycombinator.com/item?id=10598629 https://news.ycombinator.com/item?id=10598629
- mud_dauber 11y agoI'm not sure if this is the cited source, but the following Google paper is one of my bookmarks: http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf http://static.googleusercontent.com/media/research.google.co...
- mschuster91 11y agoDoes anyone know of a consumer-grade laptop with ECC RAM?
- listic 11y agoI don't think there are any. For such laptops to exist, there should exist a memory controller that supports ECC RAM. Intel's chips have memory controller built-in, since Nehalem (2008) [1] and Intel has no mobile CPUs with ECC support. Apart from the brand-new mobile Xeon's, the first of which are released in September and have 45W TDP; [2][3] I think the laptops with them are not out yet. [1] https://en.wikipedia.org/wiki/Nehalem_%28microarchitecture%29 https://en.wikipedia.org/wiki/Nehalem_%28microarchitecture%2... [2] http://ark.intel.com/products/family/88210/Intel-Xeon-Processor-E3-v5-Family#@Mobile http://ark.intel.com/products/family/88210/Intel-Xeon-Proces... [3] https://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors#.22Skylake-H.22_.2814_nm.29 https://en.wikipedia.org/wiki/List_of_Intel_Xeon_microproces...
- jkot 11y agoSkylake in Xeon version. So far there is Dell XPS 15 and something from Lenovo.
- mschuster91 11y agoThanks. Though, the downside: both companies got caught red-handed with filling their computers pre-loaded with spyware, malware and other bullshit :'(
- jacquesm 11y agoIf you care enough to put ECC in your machine you probably should start with a clean install as well.
- mschuster91 11y agoDoesn't help a single bit (pun intended) against the bullshit in the BIOS data tables which installs the malware no matter what :(
- acqq 11y agoIt's Dell Precision 15 with the Xeon option, but I doubt it can use ECC RAM until someone provides the reliable link. http://www.dell.com/au/business/p/precision-m7510-workstation/pd http://www.dell.com/au/business/p/precision-m7510-workstatio... "Memory 8GB (1x8G) 2133MHz DDR4 Memory Non ECC 16GB (1x16G) 2133MHz DDR4 Memory Non ECC 16GB (2x8GB) 2133MHz DDR4 ECC Memory ECC 16GB (2x8G) 2133MHz DDR4 Memory Non ECC 32GB (4x8GB) 2133MHz DDR4 Memory ECC 32GB (4x8GB) 2133MHz DDR4 Memory Non ECC 32GB (2x16GB) 2133MHz DDR4 Memory Non ECC 64GB (4x16G) 2133MHz DDR4 Memory Non ECC"
- EvanAnderson 11y agoThe Lenovo discussed here purports to have ECC capability: https://news.ycombinator.com/item?id=10039306 https://news.ycombinator.com/item?id=10039306 I don't know if you'd call that "consumer grade" or not, though.
- parma 11y agoCheck http://www.intelligentmemory.com/ECC-DRAM/; http://www.intelligentmemory.com/ECC-DRAM/; maybe you can fit these on cots laptops. Heard it mentioned in relation to Purism Librem laptops (https://puri.sm/librem-15/); https://puri.sm/librem-15/); better ask them if it's still an option.
- mozumder 11y agoSo how do we enable page retirement in popular operating systems on machines with ECC?
- tempodox 11y agoCan I pay the publisher to omit those blinking distractions? If I want to watch a movie while reading an article, I bring one myself.
- moconnor 11y ago"DRAM chips are a little like people: Their faults are not so much in their stars as in themselves. And like so many people, they can function perfectly well once they compensate for a few small flaws." A beautifully whimsical way to end a fascinating technical article!
- crististm 11y agoJudging from Jeff Atwood's post only three days ago, a lot of people converted or seem to believe now that bit flips in RAMs are a thing of the past and ECC is just an "enterprisey" thing. https://news.ycombinator.com/item?id=10598629 https://news.ycombinator.com/item?id=10598629 He didn't dare himself to not use ECC for the database... :)
- mud_dauber 11y agoSeeing that DRAM pricing is so dirt cheap, it's difficult to persuade any chip manufacturer to devote another 10-15% (my estimate based on previous memory design projects) of die area to ECC bits & decode circuitry when they won't get another dime for the capability.
- feld 11y agoWhy can't they charge 15% more for ECC to recoup the costs?
- mud_dauber 11y agoIt depends on the end customer. PC manufacturers have zero incentive to add ECC when 99.99% of consumers don't care or even know. Networking: a mixed big. Some designs manage ECC inside the processor, others use memories with ECC for specialized functions like packet QoS. Commmodity DRAM errors are a known irritant and planned for. Servers: I'm not quite sure of my facts here, but a DRAM manufacturer would have to persuade a commmodity manager at Dell, etc, that ECC is worth it. A commodity manager won't give a d* unless his systems guys say that it's needed.
- feld 11y agoI don't think you get the point. Just stop making non-ECC RAM for all future DDR releases. Let's just say, for example, "DDR6 has ECC baked into the standard" and have everyone on board with this. The memory manufacturers bump prices a bit to cover the costs and the entire world moves on unabashed.
- nano_o 11y agoThe article seems to be based on the following paper, published in 2012: "Cosmic Rays Don't Strike Twice: Understanding the Nature of DRAM Errors and the Implications for System Design". Newer research on the topic has since been published, for example this year: "Revisiting Memory Errors in Large-Scale Production Data Centers: Analysis and Modeling of New Trends from the Field" (from Facebook) and "Memory Errors in Modern Systems: The Good, The Bad, and The Ugly".
- fivesigma 11y agoThank you Intel for disabling ECC capabilities on consumer grade CPUs. With smaller and more dense DRAM chips every year, it takes an even smaller amount of disturbance to flip a bit. ECC needs to go mainsteam yesterday.
- AnthonyMouse 11y agoAt least AMD is doing the right thing. Obviously the problem is that Intel has twice the single thread performance, but all modern CPUs are so fast that it's almost irrelevant. And on top of that most of the "actually needs performance" applications are being updated to use the GPU instead of the CPU anyway.
- witty_username 11y agoIf the application utilizes all the cores, then AMD is generally superior (performance/$)
- hollerith 11y agoare you including the price of electricity in your "$"? it usually costs more to power a CPU over its lifetime than it does to buy it, and Intel's been aggressively optimizing the Core architecture's performance per watt to prevent ARM from taking over on products where battery life is important.
- AnthonyMouse 11y agoI was curious so I actually measured it with a watt meter. AMD FX-8350 consumes ~39 watts at idle and ~180 at full load. Intel Core i5 (Ivy Bridge) consumes ~27 watts at idle and ~90 at full load. They both consume ~1 watt in standby. Also FWIW if you're holding onto an older chip because it's still "good enough" then you might want to think about upgrading to either of the above. Dual socket Intel Xeon 5160 uses 300 watts at the plug at idle.
- 11y ago
- gima 11y agoSource paper appears to be: http://www.cs.toronto.edu/~bianca/papers/ASPLOS2012.pdf http://www.cs.toronto.edu/~bianca/papers/ASPLOS2012.pdf Based on the list of authors at the beginning of the article and the credited source given in the figures ("Ioan Stefanovici, Andy Hwang & Bianca Schroeder" and "Source: Hwang, Stefanovici, and Schroeder, Proceedings of ASPLOS XVII, 2012")
- sevensor 11y agoThe idea that cosmic rays are responsible for DRAM failures seems laughable. I'm surprised anybody ever thought that. I worked in a DRAM fab; I saw how it was made and tested. It's a month-long process involving dozens of machines. Particulate contamination, watermarks, process variation -- there are tons of things that can go wrong. And then testing. There's just no way to test thoroughly enough to catch all of the possible issues with timing and crosstalk. Not to mention electromigration and dielectric breakdown. You can do burn-in, but it's not going to catch everything. We even know which die are most failure-prone (usually edge die, due to process uniformity issues.) If they pass test, they get shipped! Of course these issues are manufacturing related.
- josaka 11y agoThis is true, re. manufacturing defects. I used to work in a DRAM fab, too, and observed similar patterns. Re. cosmic rays, it's surprising, but I think they can cause errors: http://stackoverflow.com/questions/2580933/cosmic-rays-what-is-the-probability-they-will-affect-a-program http://stackoverflow.com/questions/2580933/cosmic-rays-what-...
- mud_dauber 11y agoThe risk is significant: JEDEC has a standard testing methodology for them: http://www.jedec.org/sites/default/files/docs/jesd89a.pdf http://www.jedec.org/sites/default/files/docs/jesd89a.pdf
- sevensor 11y agoInteresting. Shows how my point of view colors the facts -- I never once heard one of our test engineers complain about cosmic rays, but then it was their job to blame process when defects came up. We used to try to blame dirty test heads, but that's another story. I'm sure in the real world cosmic rays actually do flip bits sometimes, but I'm equally sure that a significant proportion of the DRAM in production has latent defects. Given our observed defect rates and the limits of test coverage, it's unavoidable.