17 ms·
Should I buy ECC memory? (2015)
- deleted 10y ago[deleted]
- godzillabrennus 10y agoDo you use ZFS? If yes then you should use ECC memory. Do you have a use case where you would want your computer to alert you when the ram is failing? If yes then you should use ECC memory. Otherwise it's a nicitey and probably not worth the money.
- static_noise 10y agoDo you use ZFS? If no then you should use ECC memory. Now the half truth becomes full-truth.
- mikeash 10y agoI don't get this association between ZFS and ECC. The recommendation to use ECC with ZFS basically comes down to "all that fancy data integrity checking that ZFS does won't protect you from memory errors, so you'll effectively lose that feature." Are you OK with silent data corruption? If so, don't bother with ECC. If not, use it.
- Spooky23 10y agoOne of the major reasons for using ZFS is ensuring data integrity. If you implement ZFS for that propose and cheap out on RAM, you're at odds for that purpose.
- __jal 10y agoHistory. The ZFS folks, back when, were the only folks making much noise about the association between non-ECC RAM and corrupt data landing on disk. The truth is, if you care about the notion that your disk should return the same data that software thought it was writing, you should use ECC with any file system. But The ZFS folks made noise about the issue, I think lots of people assumed the reason was that there was something special about ZFS that needed it, and now you have something sort of like an urban legend.
- dom0 10y ago> should return the same data that software thought it was writing Hint: In an OS using a page cache (=every OS) I/O errors are not reliably propagated to applications unless they explicitly sync their dirty pages.
- __jal 10y agoI'm aware of that, but I'm not sure what I'm supposed to take away from it in this context.
- dom0 10y agoThat it's difficult to accurately define "what the application thought it wrote" when considering corruption at various abstraction layers; somewhat similar to calculating checksums over already corrupted data.
- boomboomsubban 10y agoKind of, there are two main things that give ZFS this false reputation. First is an academic paper testing if modern filesystems still needed ECC RAM. They tested ZFS and concluded horrible things could happen to your data without ECC RAM. They found the same about ext2, but that was just a small paragraph people overlooked. So nothing new, but many people are unaware that other FS have the same issue. Second is a moderator on the FreeNAS forums coming up with a scenario where a ZFS scrub would wipe out your data. Developers and other people that have read the code said it couldn't happen as described, but the story was perpetrated on the FreeNAS forums and spread across the net.
- dogma1138 10y agoBecause with ZFS bit rot can be cumulative, with most file systems a memory error will corrupt a file if the format can't handle errors, with ZFS overtime the entire volume can get corrupted especially when you are doing recovery or expansion, even in normal operation data is moved around quite a bit. For them most part with other common file systems when a file is written it stays there even in RAID.
- Dylan16807 10y agoI don't know, I wouldn't say ZFS moves files around any more than a typical filesystem.
- blackflame7000 10y agoCorrect. ZFS won't do unnecessary file reordering unless a scrub has been initiated.
- wtallis 10y agoThe only detailed explanations I've ever seen for how memory errors can snowball into whole-filesystem loss on ZFS have relied on the assumption that you have a deterministically stuck bit in a region of memory that the OS is re-using for different parts of the FS data structures but never anything that could cause the machine itself to crash (thereby clueing you in to a hardware reliability issue). Do you have a source for a more plausible analysis that takes into account how memory actually tends to fail?
- bsder 10y ago> I don't get this association between ZFS and ECC. Because ZFS was the ONLY file system that would actually catch some memory failures even if you didn't have ECC. So, ZFS got a reputation for being snotty when in reality the hardware it was running on was broken.
- blackflame7000 10y agoHere is an article detailing how ZFS is virtually unaffected by random bit flips because it would have to occur in such a way as to cause a sha256 collision between one block and its parity block in order for it to repair a valid block with a corrupt one during a scrub. Furthermore it goes on to argue that only a highly specific large scale ram corruption could possibly cause corruption and by that time it's almost certain the OS wouldn't boot up. http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-yo...
- saganus 10y ago"<pubDate>Fri, 27 Nov 2015 00:00:00 +0000</pubDate>" Needs (2015) added to the Title I think.
- sctb 10y agoThanks! Updated.
- ddingus 10y agoYes. Bit errors are uncommon and range from benign to crash. Your storage has them, memory has them, network has them. Non error correcting memory very significantly increases risk. And this is the kind of risk you don't notice, until you do and when you do, it's often subtle, insidious, impossible to track down. Servers absolutely. It's debatable on desktop, but we have huge RAM now. Might as well error correct. The bit error risk is small. Bigger RAM only adds to that possibility.
- VT_Drew 10y agoI wouldn't bother on a desktop or laptop. Servers absolutely.
- hashhar 10y agoI won't bother on a desktop. I've been using 4 machines for the last 17 years with storage varying from 10GB to 2TB and RAM varying from 128MB to 16GB and haven't personally seen any kind of data corruption in motion (or at rest for that matter). Only had 2 mechanical drives fail (though predictably). ECC is costly. The memory modules itself and the board required to support it properly.
- planteen 10y agoHave you not had a computer do something unexpected in the last 17 years? A bit flip might look like a kernel or application crash. Have you ever saved a file that couldn't later be opened by an application? You probably blamed the application for being buggy, but it could have been an upset. Bit flips / upsets cause all sorts of odd behavior.
- falcolas 10y ago> haven't personally seen any kind of data corruption in motion Ever had a program crash, hang, or act oddly? That's how data corruption in memory surfaces. Of course, non-perfect programs (i.e. all of them) act the same way, which means that differentiating memory corruption from misbehaving programs is hard. Fixing the memory errors will result in more stable system, but it still won't be perfect.
- anonymous_iam 10y agoThe article makes no mention of single event upsets (SEUs). These occur randomly when cosmic rays can cause a bit flip anywhere in the chip. ECC is a good way to mitigate SEU effects.
- jbmorgado 10y agoSorry for nitpicking, but it's not the cosmic rays, it's the cosmic rays secondaries cascade shower (produced high up in the atmosphere when a cosmic ray interacts with a particle there).
- nostrademons 10y agoWhile I was at Google, someone asked one of the very early Googlers (I think it was Craig Silverstein, but it may've been Jeff Dean) what was the biggest mistake in their Google career, and they said "Not using ECC memory on early servers." If you look through the source code & postmortems from that era of Google, there are all sorts of nasty hacks and system design constraints that arose from the fact that you couldn't trust the bits that your RAM gave back to you. It saved a few bucks in a time period where Google's hardware costs were rising rapidly, but the ripple-on effects on system design cost much more than that in lost engineer time. Data integrity is one engineering constraint that should be pushed as low down in the stack as is reasonably possible, because as you get higher up the stack, the potential causes of corrupted data multiple exponentially.
- deleted 10y ago[deleted]
- lomnakkus 10y ago> If you look through the source code & postmortems from that era of Google, there are all sorts of nasty hacks and system design constraints that arose from the fact that you couldn't trust the bits that your RAM gave back to you. Details of this would be very interesting, but obviously I understand if you cannot provide such details due to NDAs, etc. I mean, I can imagine a few mitigations (pervasive checksumming, etc), but ultimately there's very little you can actually do reliably if your memory is lying to you[1]. I can imagine that probabilistic programming would be an option, but it's hardly "mainstream" nor particularly performant :) I'm also somewhat dismayed at the price premium that Intel are charging for basic ECC support. This is a case where AMD really is a no-brainer for commodity servers unless you're looking for single-CPU performance. [1] Incidentally also true of humans.
- mentat 10y agoPervasive checksumming is going to cost a lot of CPU and touch a lot of memory. The data could be right, the checksum wrong as well. ECC double bit errors are recognized and you can handle them how you'd like, including killing the affected process.
- deleted 10y ago[deleted]
- VA3FXP 10y agoDepends on what you are doing. ZFS storage servers: Hell yes High-value data in my DB? Hell yes email server: Nope super cool gaming rig: Nope * Cluster: Hell yes General office workstation: maybe. I don't have the budget for 20 redundant copies. I do have the budget for slightly more expensive RAM. Especially on my ZFS storage arrays. ECC memory is like Insurance. You hope you never need it. One real downside that I have found, is finding out _when_ that memory correction has saved your ass. RAID arrays can alert you when a disk is dead. SMART mostly tells you when disks are failing. I haven't found a reliable tool to notify me when I am getting ECC errors/corrections.
- rocqua 10y agoThere is a hidden cost of ECC with regards to the chipset. None of the cheap chipsets support it, so on any home build, it's going to be expensive.
- binarycrusader 10y agoNot true with Ryzen, as long as you find unregistered ECC acceptable. Somewhat not true with Intel, as some of the lower end Xeons now support it.
- dogma1138 10y agoOnly ASROCK currently has BIOS/UEFI support for ECC.
- binarycrusader 10y agoBIOSTAR specs list ECC support, but I don't know if it has specific BIOS/UEFI options for it.
- snuxoll 10y agoAFAIK all of the Xeon chips support ECC, every Xeon E3 chip (which uses the desktop socket) I've looked at includes it.
- ori_b 10y agoIf you can afford it, sure. That's one reason why I'm so happy Ryzen supports it on consumer processors: It makes ECC cheap.
- deleted 10y ago[deleted]
- blackflame7000 10y agoAltitude also plays a factor in random memory corruption. From the wikipedia article on ECC Ram, "Hence, the error rates increase rapidly with rising altitude; for example, compared to the sea level, the rate of neutron flux is 3.5 times higher at 1.5 km and 300 times higher at 10–12 km (the cruising altitude of commercial airplanes).[3] As a result, systems operating at high altitudes require special provision for reliability."
- glup 10y agoso Numpy feature request: airplane mode
- gizmodo59 10y agoI wonder if this has anything to do with Microsoft's plans to build an underwater data center.
- rorosaurus 10y agoIf I remember correctly, that research venture was mostly due to the potential of easy heat exchange and "free" energy via geothermal/tidal. Now that you mention this, though, it's clear that such a datacenter would also be naturally shielded from many things!
- nine_k 10y agoCan adequate shielding help?
- brandmeyer 10y agoShielding against high-energy radiation is heavy, no matter what you use. The tenth-thickness (ie, the thickness to attenuate the radiation flux by a factor of 10) of lead is 2 inches. For water and other light materials, its about a foot. Commercial power reactors use lots of concrete, just because its cheaper to make, form, etc. So to go from "rare" to "almost never" you need maybe three or four tenth-thicknesses of shielding material. That's an impractical amount of mass to suspend around your datacenter (rack isles, whatever). Remember too, that you've already got ~30 feet of water equivalent in shielding (the atmosphere).
- mjevans 10y agoA better question is why /shouldn't/ you use ECC memory? Generally the answer to this is any context where you legitimately do NOT care about your data at all, but you still care about costs. This predominately devolves in to consumption only gaming systems. In all other cases everyone would be better served (in the long run) by buying ECC RAM.
- blackflame7000 10y agoa common network topology is to have a load balancer distribute load to a number of cheap Http servers which internally connect to a centralized and powerful database server. In this case only the database server really needs ECC ram. The system is designed to be fault tolerant for any individual HTTP server node so the increased cost vs the problem it solves doesn't make sense. I guess you could argue that a random bit flip could somehow make the HTTP server vulnerable and able to compromise the network however that risk is impossibly small. If we take IBMs estimation that a bit flip occurs at an approximate rate of (3.7 × 10-9) bytes/month and then divide it by the number of bytes in the system you can see that the odds of randomly corrupting a byte in memory that triggers a vulnerability is too small.
- fulafel 10y agoWhat about memory-error corrupted application data (or application logic) where the corruption occurred on load balancers or web application servers? There's more to data integrity than security holes.
- blackflame7000 10y agoIf you write code that detects stack smashing and illegal dereferences then you can terminate the webservice and either have a watchdog restart it or if it crashes multiple times have it taken out of service by the load balancer. There are plenty of ways to handle hardware errors without throwing out the hardware and getting "better" hardware. Technically, You could have a faulty component somewhere between the Ram and CPU and then what is your expensive ram going to do? What if the CPU cache has errors? For many small businesses often the difference between success and failure is their ability to make things work without throwing cash at the problem.
- olavgg 10y agoCan people here please stop posting that ZFS needs ECC memory. Every filesystem, with any name like FAT, NTFS, EXT4 runs more safe with ECC memory. ZFS is actually one of the few that can still be safer if you don't run with ECC memory. Source: Matthew Ahrens himself: https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=26303271#p26303271 https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=...
- lomnakkus 10y agoIndeed. It's true that the data may be corrupted before hitting any disk[1], but once it has hit the disks (>1), it's extremely unlikely that you'll ever hit a similar bit error where it'll mistakenly choose the wrong disk block to recover from. The main point of e.g. ZFS or Btrfs checksumming is that a) at least it isn't getting worse, and b) I can tell if it's getting worse. [1] ... but if the bits are not generated by the machine that actually saving to disk, how do you know they weren't corrupted along the way? The number of people who religiously check PGP signatures/SHA256sums or whatever is miniscule.
- ruleabidinguser 10y agoWhat are you doing where you're actually checking checksums periodically and detcting when things get worse? That seems like a lot of work to set up.
- boomboomsubban 10y agoThey are using ZFS, scrubbing is one command.
- lomnakkus 10y agozpool scrub (This may be a myth: It's not something you should actually do that often because actually reading the media may dregrade it.)
- derefr 10y ago
- exabrial 10y agoYes. Everyone does.
- Splendor 10y agoRelevant post by Jeff Atwood: https://blog.codinghorror.com/to-ecc-or-not-to-ecc/ https://blog.codinghorror.com/to-ecc-or-not-to-ecc/
- sbierwagen 10y agoThat post is linked in the first sentence of the submission.
- spullara 10y agoI reproduced this by bit-squatting cloudfront.net after reading about it. So many memory errors! http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html Loved the variety as well. Sometimes though requests came to me the Host header was correct!
- esun 10y agoFascinating article. Did you ever find a reason for the different ios results?
- spullara 10y agoBased on source IPs I would say that cheaper RAM = more error prone RAM.
- SolarNet 10y agoSome macs do use ECC memory (specifically some of the most popular varieties, like the Mac Pro) which is probably why you saw lower numbers on bit squatted domains.
- NamTaf 10y agoThis Defcon 21 presentation from Robert Stucke did something similar with google's domains, plus other stuff. A great watch if you've got a spare 40 minutes! https://www.youtube.com/watch?v=yQqWzHKDnTI https://www.youtube.com/watch?v=yQqWzHKDnTI
- codinghorror 10y agoWait so when someone typoes cnn.com as con.com, that is ipso facto a memory error? I guess I could see that if the characters are radically far apart on the keyboard? But doesn't a simpler explanation like "one person out of billions with Internet access typed the wrong thing" seem a lot more likely?
- duskwuff 10y agoDomains that are only used as CDNs, like cloudfront.com, are almost never typed into an address bar. Errors in the domain name are more frequently the result of a bit-flip error.
- Prego 10y agoFace it: for the type of work most of you people do--swiping left or right on Grinder, or sitting in a coffee shop pretending to write a screenplay--you don't need ECC memory. I, however, won't do any production work without it.
- lucb1e 10y agoThis article is gold in so many ways. It contains interesting bits of information on ECC, company history that I didn't know (Sun's and Google's namely), filesystem reliability (I never knew!), the physics of RAM (50 electrons per capacitor)... It's a must read, even if only to get you thinking about some of these things.
- omash 10y agoWhat are the odds of memory errors causing hard disk corruption / boot failure?
- ReligiousFlames 10y agoAn old article from DJB worth perusal: http://cr.yp.to/hardware/ecc.html http://cr.yp.to/hardware/ecc.html It's also worth noting that not all ECC (SECDED) is created equal: ChipKill™ and similar might not survive physical damage because of likely shorts of the data bus but a single malfunctioning chip producing/experiencing higher hard error rate is possible from which to recover. Also, it'd be really cool if some shop a-la BackBlaze blogged about large-scale monitoring for soft and hard RAM errors across chip/module modules (+ motherboards & CPUs). Without collecting and revealing years data from real use, conversation devolves into opinion and conjecture. Finally, not all use-cases can benefit from ECC (ie Angry Birds) however there are some obvious/nonobvious ones that can (ie router non-ECC DNS bitsquatting or processing bank transactions).
- ReligiousFlames 10y agoPS: Random-crazy thought.. it's curious with reduction of costs via Moore's law improvements that there aren't yet formally-verified, zero-knowlege systems which can end-to-end prove they performed computation/real-world side-effects and/or continue to safely store data. Why blindly trust anyone or any company with data that can be seized, lost or misused when distributed computation, communication and storage can be A2E with only limited participants knowing operations / plaintext? Perhaps: homomorphic encryption, blockchain-similar ledger or proof-of-work and periodic, authenticated hash challenge queries. Mix in relaying and other idle phony traffic to make triangulation more difficult. I think in order to assure sufficient distributed system resources are made available, μpayments a-la AWS but just covering costs would make it possible to have a persistent, anonymous computation and storage collective that would survive outages, FBI raids, single nodes going offline, etc.
- indolering 10y agoStorage, [yes](https://storj.io/ https://storj.io/). Computation ... sure if you don't mind the server viewing the contents of your computation and can verify the results. Sadly, fully homomorphic systems incur waaaay too much overhead so you are constrained in what you can do (i.e. specialized DBs, zkSNARKs, etc). Then, of course, there is the problem of network latency and bandwidth costs vs just keeping it all on one datacenter.
- danielfaust 10y agoI've spent the last two weeks looking at Memtest86+ trying to figure out if either one of my memory modules is damaged, or if it is the motherboard. These tests take a long time, and yield different results from day to day. I've decided to never ever again buy non-ECC memory, at least not on 24/7 servers as well as on workstations. In a gaming machine / visual typewriter? Sure, non-ECC memory is ok.
- notacoward 10y agoSame topic, same conclusion, even more hard facts. http://perspectives.mvdirona.com/2009/10/you-really-do-need-ecc-memory/ http://perspectives.mvdirona.com/2009/10/you-really-do-need-...
- veidr 10y agoYes. Everybody reading this should use ECC RAM, and non-ECC RAM should be called "error-propagating RAM". Random bit flips aren't cool, and they happen regularly. Most computers that have ECC RAM can report whether errors happen. I see them at least once a year or so. For instance, here are 2 ECC-correctable memory errors that occurred just last month. Cosmic rays? Fukushima phantom? Who knows. You'll never know why they happen (unless it's like a bad RAM module and they happen a lot), but if you don't rock ECC you will never know they happened at all. You'll be left guessing when, years later, some encrypted file can no longer decrypt, and all the backups show the same corruption... [1]: https://www.dropbox.com/s/zndvy3nkv1jipri/2017-03-20%20FUCK%20memory%20errors.png?dl=0 https://www.dropbox.com/s/zndvy3nkv1jipri/2017-03-20%20FUCK%... [2]: https://www.dropbox.com/s/6yeoedc7ajzq4u9/2017-03-20%20FUCK%20memory%20errors%20detail.png?dl=0 https://www.dropbox.com/s/6yeoedc7ajzq4u9/2017-03-20%20FUCK%...
- blackflame7000 10y agoNice file names, gonna have to start naming my bug reports similarly
- jandrese 10y agoI remember the one time I bought ECC memory, for a PII-400. It was only 512MB or so I think, but in the 12 years that server ran I saw a grand total of 1 corrected error in the logs. Given how much of a premium that ECC memory was it felt like a waste.
- sitkack 10y agoI want to thank Jeff for assisting Dan in writing this article.
- epx 10y agoI think that, given the personal importance of computing devices and storage, no filesystem should exist w/o checksum of metadata+data, and no RAM should be without ECC. The slight increase in cost does not justify the risk.
- mixmastamyk 10y agoI searched for a good Linux laptop recently with ecc but didn't find much so settled on a kaby lake i5. Does anyone make them?
- jpalomaki 10y agoFor example Lenovo P51 seems to support ECC (if equipped with Xeon processor). About the Linux support I don't know, but I've understood at least some other Lenovo models work ok with Linux. http://psref.lenovo.com/Product/ThinkPad_P51 http://psref.lenovo.com/Product/ThinkPad_P51
- myrandomcomment 10y agoYes. Are we done :)
- intrasight 10y agoI am typing this (finally!) on my new desktop build. I did mull over the decision for a while but finally went with Xeon and ECC. So the memory cost more - perhaps even twice as much - so what? I use my computer pretty heavily for my work - with several VMs running at a time. If ECC saves me a headache once a year, it will have paid for itself. If it never provides ANY benefit I will still not regret the peace of mind.
- usrusr 10y agoThe parameters of the desktop ECC decision have changed massively with today's glacial replacement cycles. Today you make a one time payment for many years of avoided headaches and peace of mind, whereas back then any sign of unreliability would have been a welcome excuse for a cheap upgrade.
- alkonaut 10y agoFor servers this is more or less a no brainer: it's not a huge extra cost and a failure will cost you more than the extra cost. For a regular desktop system for personal use it's not so easy. The data volumes are much smaller, the temperature environments are usually better, they aren't running (other than maybe idling) 24/7, most of the stuff that is in ram isn't going to be mission critical (i.e. you don't have 32Gb of RAM filled with customer database records, you have it filled with read only FPS textures, compiler caches etc). Unlike a business that has tons of data that is mutated, my data is mostly immutable such as photos etc. It's not a continuously changing dataset where a bit flip in memory is likely to find its way into my data and then into my backups which would be the case e.g. for databases or big creative work (movie editing etc).
- Tomte 10y agoRehashing an old comment: IEC 61508 documents an estimate of 700 to 1200 fit/MBit (fit = "failure in time"; per 10e-9 hours of operation) and gives the following sources: a) Altitude SEE Test European Platform (ASTEP) and First Results in CMOS 130 nm SRAM. J-L. Autran, P. Roche, C. Sudre et al. Nuclear Science, IEEE Transactions on Volume 54, Issue 4, Aug. 2007 Page(s):1002 - 1009 b) Radiation-Induced Soft Errors in Advanced Semiconductor Technologies, Robert C. Baumann, Fellow, IEEE, IEEE TRANSACTIONS ON DEVICE AND MATERIALS RELIABILITY, VOL. 5, NO. 3, SEPTEMBER 2005 c) Soft errors' impact on system reliability, Ritesh Mastipuram and Edwin C Wee, Cypress Semiconductor, 2004 d) Trends And Challenges In VLSI Circuit Reliability, C. Costantinescu, Intel, 2003, IEEE Computer Society e) Basic mechanisms and modeling of single-event upset in digital microelectronics, P. E. Dodd and L. W. Massengill, IEEE Trans. Nucl. Sci., vol. 50, no. 3, pp. 583–602, Jun. 2003. f) Destructive single-event effects in semiconductor devices and ICs, F. W. Sexton, IEEE Trans. Nucl. Sci., vol. 50, no. 3, pp. 603–621, Jun. 2003. g) Coming Challenges in Microarchitecture and Architecture, Ronen, Mendelson, Proceedings of the IEEE, Volume 89, Issue 3, Mar 2001 Page(s):325 – 340 h) Scaling and Technology Issues for Soft Error Rates, A Johnston, 4th Annual Research Conference on Reliability Stanford University, October 2000 i) International Technology Roadmap for Semiconductors (ITRS), several papers. If that's correct, the math is simple: you have bit flips in your PC about once a day. It's just that (a) you often won't notice those transient errors (one pixel in your multi-megapixel photo is one bit off) and (b) a lot of your RAM is probably unused.
- tsukikage 10y agoNo-one's mentioned it yet, but we're in a post-Rowhammer world and ISTM this is relevant to the discussion: while not all non-ECC DIMMs are susceptible, the cheaper ranges generally are, and if your purchasing decisions are driven by hardware cost, that's probably what you'll end up with. Corruption due to malice is a rather different beast to corruption due to random cosmic rays...
- justin66 10y agoIn the late nineties, the Intel desktop chipsets such as 440LX and 440BX offered ECC functionality, all you had to do was spend ten or fifteen bucks extra on the memory. Great hardware. I'm unhappy that Intel made things more expensive and complicated with their market differentiation, but from their POV it was logical. PC users were screwing up the reliability of their systems in so many ways via overclocking, and were habituated to accept crappy reliability via pre-NT Windows. PC users could have demanded ECC and they didn't. I'm sure that even when the chipsets made it easy, only a tiny fraction bothered to use ECC.
- JakiesKonto 10y agosome1 already mentioned row hammer so ecc yes :)