11 ms·
Why use ECC? (2015)
- genpfault 2y ago(2015)
- wmf 2y agoThere is a 2024 update in the middle saying that "Gabriele Svelto estimated that approximately 10% to 20% of all Firefox crashes were due to memory corruption."
- summerlight 2y agohttps://discourse.codinghorror.com/t/to-ecc-or-not-to-ecc/3771/62 https://discourse.codinghorror.com/t/to-ecc-or-not-to-ecc/37... Interestingly, Jeff Atwood has changed his mind on ECC memory.
- pixelpoet 2y agoEven more interestingly in the comment below, apparently you can just flick on ECC for RTX 4090 cards! Extra weird is Nvidia singling out ray tracing as a use case which shouldn't use ECC... I suppose it's no biggie if a single ray goes the wrong way down the BVH, out of trillions.
- kragen 2y agoyeah, makes sense to me. it'll just result in unnecessarily antialiasing that pixel usually. unless it corrupts the definition of an object and then that object might disappear or become enormous, and probably most raytracing memory is objects, right? not rays being traced
- sufehmi 2y agoAs soon as I worked with high-load servers in the 90s, it's already clear that ECC should be the default. Intel's marketing ploy on ECC is very desctructive and have costed many parties a lot of wasted time money & resources to handle problems caused by non-ECC memory ; and Linux Torvalds is absolutely right in roasting them for this.
- TheAmazingRace 2y agoSo I have to say, ECC memory is definitely something we should not have gotten away from for consumer hardware. My current PC, which is rocking a Core i9 14900k (pray for me) and an ASUS W680M ACE SE motherboard, allowed me to install some 5600MHz speed DDR5 ECC memory, and it works flawlessly. The only downside in my view is the cost. Unbuffered ECC and the cost of using a workstation class chipset really pushes this into luxury territory. Plus, I'm never too sure what Intel's future plans are for successor processors and chipsets, which is why I settled on W680. I don't really want to go full blown Xeon.
- pixelpoet 2y ago14900k might be one of the least reliable CPUs in recent times, pairing it with ECC seems almost ironic!
- vardump 2y agoThe recent trend with Intel CPU (at least 13900K and 14900K) reliability is indeed rather worrisome.
- walteweiss 2y agoDo you mind sharing more details for those who weren’t paying attention? I’m still on 10 years old generations with my home PCs and my laptop. And don’t feel the urge to upgrade anything yet. Hence, I’m not following the trends. The only thing I’m considering is the upgrade of my iPad Pro and probably MacBook at some point (not sure if I need it at all). My home computer is doing just fine with 4th generation Intel CPU and I don’t see any need for upgrade in upcoming years, if not a decade.
- vardump 2y agoThere are significantly more errors occurring on those CPUs. I don't have much time now, but you can check details: https://www.google.com/search?q=14900K+failures https://www.google.com/search?q=14900K+failures
- dvt 2y agoI tried building an old rig (maybe ~7 years ago or so) using ECC RAM (since I was running two Xeons). It was such a pain in the butt to get it to boot and find sticks that were compatible with each other, don't really want to go down that path again.
- danparsonson 2y agoI built my most recent desktop using ECC and it was a breeze, so maybe you were just unlucky?
- bjoli 2y agoI just chugged 128gb of ddr4 LRDIMMs into my old xeon server. It worked flawlessly.
- Fnoord 2y agoFor me it was a one shot with a Xeon, no issues whatsoever. Any decent price comparison is able to filter on ECC memory.
- _factor 2y agoThe cheap sticks work if you don’t mind buying a bunch and sending back the ones that don’t work together in your config. Binning is likely the problem.
- Animats 2y agoECC memory should have a price premium of only 1 - 9/8, or 12.5%. It costs more than that, because it's "enterprise".
- ido 2y agoTrue, as is explicitly mentioned in the article.
- thfuran 2y agoProbably slightly more than just the increase in memory modules, since there's also the extra complexity of actually checking/reporting, but roughly.
- wmf 2y agoIt's now 25% for DDR5.
- Animats 2y agoThat's not bad. A few years ago, it was more like 100%, and hard to get.
- wmf 2y agoNo, I mean a DDR5 ECC DIMM has 25% more chips (10 vs. 8). I don't know what the price difference is.
- adrian_b 2y agoThat is no longer true. The DDR5 standard allows either 40-bit channels or 36-bit channels in ECC DIMMs. (A DDR5 UDIMM has 2 channels, while the desktop CPUs use 4 channels provided by 2 sockets.) The former choice corresponds with a 25% overhead, while the latter with a 12.5% overhead. In the beginning there were only 80-bit ECC DDR5 UDIMMs, because the DRAM vendors preferred to make only x8 chips. More than a year ago, 72-bit ECC DDR5 UDIMMs have also apppeared, which use a mixture of x8 and x4 chips. Nowadays no ECC memory vendor can justify a 25% overprice, because if they happen to use 25% more memory that is because they believe this reduces their production costs (by making a single kind of chip). Only an overprice slightly more than 12.5% is right. Unfortunately, due to limited offer, the ECC DDR5 UDIMMs can still be up to 50% more expensive than non-ECC modules.
- Nerada 2y agoDDR5 comes with on-die ECC. My understanding is this only checks errors occuring within the RAM itself, not errors that occur during transmission to and from RAM. My question is, how common are transmission errors over errors happening within RAM?
- pclmulqdq 2y agoOn-die ECC is so they can give you a memory array with a few faults. It's a yield enhancement not an introduction of ECC as you think of it. Adding protocol-level ECC on top only helps, although it is somewhat inefficient.
- thfuran 2y agoSimilar to SSDs, which are constantly switching to less and less reliable cells for density and now need fault correction built in to function at all.
- gjjydfhgd 2y agoAnother problem with on-die ECC is the lack of reporting. You have no idea if you have tons of errors and how many were corrected.
- geerlingguy 2y agoLPDDR4/4X has also had on-die ECC for a while (at least the chips I'm used to, like in the Raspberry P); with such small lithography it's basically required to get the ram to work reliably.
- hi-v-rocknroll 2y agoSort of. It's not the same as extended ECC like ChipKill. https://en.wikipedia.org/wiki/Chipkill https://en.wikipedia.org/wiki/Chipkill DRAM Errors in the Wild: A Large-Scale Field Study (2009) https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf https://static.googleusercontent.com/media/research.google.c...
- nextaccountic 2y agoWhy not ECC CPUs and GPUs? They can be hit by cosmic rays too.
- adrian_b 2y agoAlmost all CPUs (except perhaps some very cheap microcontrollers) already use ECC for their internal cache memories, because those are even more prone to errors than the external DRAM. Consumer GPUs, which do not have ECC, are notorious for frequent errors when they are used for general-purpose computation tasks instead of graphics or AI applications (where errrors happen, but they usually do not matter), so for any important computation it is recommended to repeat it and compare the results.
- caf 2y agoIf the CPU exposes statistics from the internal cache ECC, I wonder if you could use that as a rough radiation sensor?
- adrian_b 2y agoMost CPUs do not document the details of the cache controllers, so typically you cannot know when correctable errors happened. Only when non-correctable errors happen, which should be very seldom (e.g. less than one per year), the CPU generates a machine error exception, for which it can be identified that the cause was a (multiple-bit) error in one of the internal cache memories. Therefore you could sense only extremely-high radiation levels, which would also require a customized OS kernel to recover from exceptions without crashes. Such great radiation levels could be much easier detected with some discrete reverse-biased diodes, which are also easier to place wherever the radiation exists, keeping the microcontroller or whatever device records the radiation levels either away or shielded from the radiation.
- caf 2y agoYes, it was correctable errors I was thinking of. Not too surprising that they don't expose those statistics, I guess - it probably doesn't even count them.
- jrockway 2y agoNot having ECC is the biggest scam in computing. Ever hear of "bitrot"? That's memory errors that have been saved to disk. We have made millions of people lose their data so that servers can be artificially more expensive. Intel was responsible for most of this. It is hard to be sad when seeing how they've lost the market lead.
- jan_Sate 2y agoHuh? I thought that "bitrot" was like content saved into the disk and the disk left unpowered for an extended period of time causing data loss. And I thought that the content stored on the disk has ECC on its own?
- lukaslalinsky 2y agoYou get a corrupted bit in memory, you take and save that corrupted content to disk and no error correction on the disk level will help you. I made the mistake of running a big PostgresSQL database on non-ECC memory once and I must say, it taught me some hard lessons.
- cornholio 2y agoI wonder if it's practical to do ECC on 64 bit words instead of bytes. A 13% price increase (or capacity drop for the same DRAM price) is substantial and might justify the penny pinching, 1.5% is negligible if it leads to a similar stability increase as standard ECC DRAM. If you are often getting more then one bit flip per 64 bit word, than that RAM is garbage anyway.
- baq 2y agothis a loaded question. anyone interested in the topic should absolutely start from reading up on https://en.wikipedia.org/wiki/Error_correction_code https://en.wikipedia.org/wiki/Error_correction_code and only then start looking into the engineering side, starting with https://en.wikipedia.org/wiki/ECC_memory https://en.wikipedia.org/wiki/ECC_memory, notably some benchmarks reported 25% performance hits (not sure if it's real or sales propaganda.)
- kimixa 2y agoI wish there was a hard requirement for ECC, as a developer working on GPU drivers, there's a huge amount of reported issues that just... don't make sense? One offs with slightly different symptoms, memory dumps of nonsense, just nowhere to start rooting out the cause for an issue. Even on "widely reported" issues that make it to reddit and similar. Probably not surprising, there's a naturally antagonistic relationship between Performance and Reliability here, and it's clear which way many of those "enthusiast" forums lean. I haven't got actual numbers, but I feel that most [0] of the issues I start looking at just can never be reproduced, or even make sense from the backtrace or similar. I can't say it's 100% hardware issues for this, as many games are a little... loose... with reliability if it works "well enough", and is heavily interacting with code and data we work on so might also be a source for "impossible" issues. But even on straightforward code paths, no weird OS interaction, no allocation, nothing async etc. "Impossible" states happen pretty regularly. I would love there to be enough ECC-using gamers out there to statistically see if it makes a difference. [0] Most in terms of number of different issues, not total reports of the same issue. That's dominated by one or two things, normally around the latest game or update doing something dumb :P
- a_t48 2y agoBack when I heavily used GCP's GPUs I'd actually get failed tests with ECC errors. It was kind of neat.
- vlovich123 2y agoStatistically, in my experience, plain old memory corruption bugs within your code or within the GPU itself are a more likely explanation than issues ECC would fix. Games have pretty terrible quality because games can remain playable in the face of bugs. GPUs have similar issues because numerical accuracy is an after thought but numerical accuracy issues can cause issues in code that assume stability of results. Things have gotten better particular on the GPU vendor side of things, but still, memory safety issues and UB issues in c/C++ code far dominate as the root cause any issue you’re likely to see vs memory bit flips (unless people are running very overclocked and unstable machines).
- 2y ago
- dang 2y agoRelated: Why Use ECC? (2015) - https://news.ycombinator.com/item?id=25167288 https://news.ycombinator.com/item?id=25167288 - Nov 2020 (98 comments) Why Use ECC Memory? - https://news.ycombinator.com/item?id=23361577 https://news.ycombinator.com/item?id=23361577 - May 2020 (2 comments) Should I buy ECC memory? (2015) - https://news.ycombinator.com/item?id=14206635 https://news.ycombinator.com/item?id=14206635 - April 2017 (224 comments) Why use ECC? - https://news.ycombinator.com/item?id=10638324 https://news.ycombinator.com/item?id=10638324 - Nov 2015 (95 comments)
- transpute 2y agoPC Engines $150 APU2 (RIP) shipped with 4GB ECC RAM and AMD Embedded CPU. Since it was a headless device used mostly for 1GbE networking, the RAM was throttled and relatively impervious to Rowhammer. QNAP has a $600 1U short-depth (11") 4x3.5 2xM.2 2x10GbE 2x2.5GbE 4-32GB DDR4 SODIMM Arm NAS that would benefit from OSS community attention. Based on a Marvell/Armada CN9130 SoC which supports ECC, has mainline Linux support, and public-but-non-upstream code for uboot [2]. With local serial console and a bit of effort, the QNAP OS can be replaced by Arm Debian/Devuan with ZFS. Rare combo of low power, small size, fast network, ECC memory and upstream-friendly Linux. QNAP also sell a 10GbE router based on the same SoC. Ryzen Pro (OEM) can support ECC [3]. [1] https://www.qnap.com/en-us/product/ts-435xeu https://www.qnap.com/en-us/product/ts-435xeu [2] https://solidrun.atlassian.net/wiki/spaces/developer/pages/309919745/CN913x+Build https://solidrun.atlassian.net/wiki/spaces/developer/pages/3... [3] https://www.tomshardware.com/pc-components/cpus/amd-confirms-ryzen-8000g-apus-dont-support-ecc-ram-despite-initial-claims https://www.tomshardware.com/pc-components/cpus/amd-confirms...
- karma_pharmer 2y agoMarvell/Armada CN9130 SoC ... public-but-non-upstream code for uboot Dude, not even close. That repo is for an opaque blob: https://github.com/SolidRun/cn913x_build/blob/master/binaries/atf/mrvl_scp_bl2.img https://github.com/SolidRun/cn913x_build/blob/master/binarie... ... which is mashed together-bits of other blobs, including the notorious will-never-be-open-source highly-radioactive "snps blob": https://github.com/MarvellEmbeddedProcessors/mv-ddr-marvell/blob/master/drivers/snps/fw/snps_static.o https://github.com/MarvellEmbeddedProcessors/mv-ddr-marvell/... and its vile EULA which you have agreed to: https://github.com/MarvellEmbeddedProcessors/mv-ddr-marvell/blob/master/drivers/snps/fw/snps_license.txt https://github.com/MarvellEmbeddedProcessors/mv-ddr-marvell/... That shit contains an ARC core. Never heard of ARC? That's okay, most people haven't. It's an obscure niche architecture used for almost nothing except for ... drum roll ... the Intel Management Engine (up until 2019). Gee I wonder why that's in there. Don't touch Marvell shit with a ten foot pole, it's blobs all the way down. They're just good at hiding it.
- transpute 2y ago
- magicalhippo 2y agoMemory corruptions can impact very differently. A sample of decoded music getting corrupted leads to a small glitch, maybe even inaudible. An instruction in executable code getting corrupted can leads to all sorts of havoc. Since ECC is seemingly not getting mandatory, I've been wishing CPUs would support "soft-ECC". That is, the OS could mark certain pages as needing "soft-ECC", and the CPU would then store (at least) three copies of that page in RAM. When reading such pages back from RAM the CPU would read all physical copies and compare. If the majority agrees it can use that, otherwise raise an error. This could then be used for executable pages and important configuration data which occupies relatively few pages, and where integrity matters a lot more than speed. There's probably some good reasons why this is non-trivial to implement, I've forgotten most of what I learned about the virtual memory implementation in CPUs. But a man can dream...
- HideousKojima 2y agoThe triple reading/writing to memory along with the comparing would probably be a significant performance hit. You could just use a bit of extra memory for parity bits etc. instead
- magicalhippo 2y agoSure it would be a significant hit. But only for relatively few pages, and I imagine most of them would either stay cached or are very cold. Of course you could trade implementation complexity for speed as always. My main point was to have effective ECC without any additional support from motherboard and memory modules.
- menaerus 2y agoThe important detail to make your theory plausible, and without a huge performance impact and/or jittering, is how the kernel memory management system is going to decide what exactly is those "relatively few pages" on a server that runs a plentiful of processes, many of them including critical/heavy ones such as databases? One such example comes to my mind - transparent huge pages. And it didn't turn out to be quite successful. I imagine soft-ECC would be more plausible if it could be applied/enabled: 1. Per-process (e.g. whole application) 2. Or fine-grained per-allocation within a process/application I think both could be implemented through a (system) memory allocator by taking the advantage of page alignment LSBs and/or (non) addressable bits of the virtual memory. Those spare bits in memory can be used to store the encoding of ECC algorithm (Hamming, Reed-Solomon, or something more primitive but less robust). And then, depending on the ECC algo, one could substantially minimize the performance impact of encode/decode by using SIMD.
- Sweepi 2y agoI would love to put ECC in my Desktop computers, however its more expensive (ok), is not officially supported on most Desktop Motherboards (and in reality does not work in "ECC-Mode" on the majority of them) and finally: ECC Ram available to purchase is painfully slow, in both bandwidth (:/) and latency (://)
- ceving 2y agoThe only problem with ECC is that it is only an improvement, not a solution. Every error correction has a limit to the number of errors it can detect or correct. There is no such thing as absolute security.
- cbolton 2y agoHaving a small remaining probability of error is not a problem at all, if it's small enough. There are always potential sources of failures (such as a meteorite crashing on your server). If the probability of one source is dwarfed by others then it doesn't matter.
- sph 2y agoIt's not a "problem" if, as you say, perfection is an unreachable goal. The only question is whether it's an improvement over the status quo, and it clearly is.
- luckystarr 2y agoThat's the reason I chose AMD for my own laptop. AFAIR Intel doesn't support ECC.
- BlueTemplar 2y ago> From talking to folks at a lot of large tech companies, it seems that most of them have had a climate control issue resulting in clouds or fog in their datacenters. You might call this a clever plan by Google to reproduce Seattle weather so they can poach MS employees. Alternately, it might be a plan to create literal cloud computing. Or maybe not.
- stefantalpalaru 2y ago[dead]
- forty 2y agoI read somewhere that DDR5 has some kind of internal ECC mechanism even for non ECC stick, is that right? Does it make ECC less relevant?
- hi-v-rocknroll 2y agoOther references: https://cr.yp.to/hardware/ecc.html https://cr.yp.to/hardware/ecc.html (2001) DEF CON 19 - Artem Dinaburg - Bit-squatting DNS Hijacking Without Exploitation (2011) https://youtu.be/aT7mnSstKGs https://youtu.be/aT7mnSstKGs https://media.defcon.org/DEF%20CON%2019/DEF%20CON%2019%20video%20and%20slides/DEF%20CON%2019%20-%20Artem%20Dinaburg%20-%20Bit-squatting%20DNS%20Hijacking%20Without%20Exploitation%20-%20Video%20and%20Slides.mp4 https://media.defcon.org/DEF%20CON%2019/DEF%20CON%2019%20vid...
- anonymousiam 2y agoNice to see Artem's name show up here. I had the pleasure of working with him about ten years ago.
- snvzz 2y agoECC should be a requirement. The FCC could just not allow computers to ship it without. CPU makers like Intel and AMD could simply have their CPUs not work with non-ECC RAM. Microsoft could e.g. require ECC RAM for Windows 12. It is insanity that most computers shipping today do not use ECC and are thus unreliable. With luck they'll crash, but most likely they will fail silently, while corrupting data.
- adhoc32 2y agoI've been there, and it was a pain. All my backups were corrupted due to a faulty RAM module. Initially, I blamed the hard drives because they seemed to be failing right before my eyes. I was copying a large file; sometimes it copied okay, but occasionally it would become corrupted. Since then, I've been paying a premium for ECC.
- TheCondor 2y agoSame experience. We were doing all the things, regular backups, rotating them, verifying them. During a weekly verification test, it failed. Tested some older backups and they failed too! If the data matters, it’s hard to express the stress and disconcert you feel in this moment. Memory is different from all other resources in the system. We are conditioned as engineers, we know drives fail more frequently than other resources. When memory fails it is indistinguishable from a drive failure. There are some system behaviors that matter too, we tend to think that page allocation is random and on heavily loaded systems it appears to be, but on specialized systems it can be rather consistent so the verification can fail in nearly the same place, repeatedly. Riddle me this: what is more likely? A memory failure, a drive failure, or a postgresql bug that results in a corrupted row? Badblocks checks out on the server’s disks… if the data matters, it is extremely unpleasant going through that whole thing, it’s crystal clear after the fact but it’s a bloody nightmare in the heat of it all.
- 1oooqooq 2y agoso true. anecdotal, I've gone from 4 blue screens a month down to zero after going ECC, on the few desktops we require windows. everyone must still be quoting numbers when we had 4mb of premium chips. now that all pcs have 8-128gb of the crumiest, cheapest silicon... i bet the failure rates are way more noticeable. sadly, i got laptops for my company that have a PRO amd cpu and sodimm sockets... only to find out ecc sodimm ram is sold by one manufacturer gouging the NAS market with crazy insane prices.
- oskarkk 2y ago> For example, at 20nm, a DRAM capacitor might hold something like 50 electrons, and that number will get smaller for next generation DRAM and things continue to shrink. Nice. That got me curious, how many electrons are in today's DRAM capacitor? I tried searching but haven't found any recent info.
- sph 2y agoPlease, I'd love someone to tell me how to find and buy computers that support ECC. I am looking to buy a NUC/mini-server, and they basically all sell with non-ECC RAM. Last time I asked on this forum, I was told that on Intel, only Xeon processors support ECC, while all modern (?) AMD CPU support them. Elsewhere I read that was matter is the mobo needs to support it. I have no idea how to go about it. So, let me ask again. I was to buy a NUC new or off Ebay, how can I be 100% sure it works with ECC RAM without having to spend half a hour researching CPU, mobo and BIOS specs for each single product I come across? If I had a budget in the thousands, I would go with a Xeon server that comes with ECC pre-installed. I don't and have modest needs. I only want to splurge on ECC RAM to replace the original sticks. (No "you don't need ECC for a NUC" reply please. That is not the point of my question, yet it is a far too common response)
- petronio 2y agoI was on the NUC search a while ago and I'm not sure you can. Although the AMD motherboards may not support ECC, I haven't heard of any that actually don't. Best bet is probably to buy a recent, barebones AMD NUC, and buy the ECC RAM yourself. Sometimes they'll advertise ECC support as well.
- adrian_b 2y agoI have never seen any true NUC-like computer that supports ECC SODIMMs. Intel has also used the NUC brand for a much larger computer that supported laptop Xeon CPUs, but that line has been abandoned. There have been some NUC-like computers from ASRock industrial, Supermicro and others, with either Tiger Lake or Elkhart Lake CPUs, where you could enable in BIOS the so-called in-band ECC. All these models are obsolete. Moreover, in-band ECC is an ugly and inefficient workaround. It can be used with soldered LPDDR memories, which do not have ECC variants, but it has worse performances than standard ECC. It is not cheaper, because it diminishes the memory capacity in the same ratio as any ECC and it requires a greater die area inside the CPU for its implementation (including a dedicated cache memory for the ECC bits). There are many mini-ITX motherboards that support ECC (but you must check carefully the specifications, even if they are for AMD CPUs). For a smaller size than mini-ITX, there are 2 choices, either expensive industrial single-board computers, which usually have the 3.5" form factor of the PCB, or one of the so-called mobile workstation laptops from Dell, HP or Lenovo, e.g. a Dell Precision mobile workstation, which are also much more expensive than an equivalent NUC-like computer. So, if a low price is desired and an up-to-date fast CPU, you cannot have ECC in form factors smaller than mini-ITX. If paying double or triple is not a problem, there are solutions. If you want a preassembled small computer with ECC and a mini-ITX motherboard, there are some at companies like ASRock Rack or Supermicro, but they are much more expensive than if you get the best components and you assemble them yourself.
- eadmund 2y agoWhat’s the best price/performance for a home lab server running Linux with ECC these days? Bonus points if it is rackable. Sadly, my go-to Linux hardware manufacturers either don’t offer ECC RAM, or only offer it as an option on their absolute top-end machines. Yes, yes, the extra two thousand dollars for a machine with a six-year lifespan probably is worth it on a monthly basis, but man it still hurts.
- Palomides 2y agoput something together with used supermicro parts, maybe with a H11SSL-i or H12SSL motherboard and epyc cpu or whatever dell 730 or something fits your budget
- adql 2y ago> What’s the best price/performance for a home lab server running Linux with ECC these days? Bonus points if it is rackable. Old used enterprise server. None of them will be great at power/performance in typical (i.e. mostly idle) home use tho. Intel ones usually far better here
- NorwegianDude 2y agoI recently(ish) built a new home server using an cheap AM5 motherboard from ASUS that supports ECC. Good performance and power usage is around 45 W idle with a couple of SSDs and a couple of HDDs spinning. Not the cheapest, but I wanted to keep power consumption low for noise and reduced heating while still having good performance if needed. I also considered a motherboard with IPMI on AM5(Asrock rack), but that was much more expensive. Worked out quite nicely.
- nottorp 2y agoI believe the first part could make for the start of a great 'if Google does it it doesn't mean it's good for you' article...
- deleted 2y ago[deleted]