9 ms·
Why Use ECC? (2015)
- account42 6y agoSo I have 8 sticks of ECC RAM in my desktop. I overclock the RAM , and ECC is nice there since it gives more confidence that the overclock is stable. But more importantly, ECC was really useful once one of the sticks started failing. Not only did the ECC errors immediately point the blame at the RAM (no need to triage random crashes or corruption), it made it clear that all errors were coming from only one of the sticks. Figuring out which stick it was involved some guesswork of the physical -> logical -> Linux utilities' layout mapping, but just replacing that one stick fixed things and the system has been stable since. Definitely glad that I went with ECC for this system.
- xienze 6y agoIt’s not like there’s an outrageous price difference between the two anymore. If your server can take ECC, might as well use it.
- ed25519FUUU 6y agoI don’t use ECC in my home server because ECC-supported motherboards are still more expensive than their regular counterparts, and ECC ram is also more expensive. And of course this is only if your CPU supported it, which is basically not any prosumer Intel CPU. Now that AMD CPUs are taking a big market share, hopefully we see a better market for ECC hardware.
- snuxoll 6y agoIf you’re building new, then, yeah, getting a board and CPU with ECC support will cost you. This is why most homelabbers buy server gear that’s been decommissioned, my R3/4/520’s with Ivy Bridge-EN chips may not be top of the line, but they’re affordable.
- theevilsharpie 6y agoUnbuffered ECC memory is definitely more expensive, but ECC support on AMD motherboards is common. I'm currently running ECC on an ASRock B450M Pro4, which was one of the least expensive AMD Ryzen motherboards available at the time I purchased it (without going to the very down-market A320 series).
- Sniffnoy 6y ago(2015)
- h2odragon 6y ago(forgive errors in my memory here, details may be wrong) The first generation of UltraSPARC CPUs with 8mb cache didn't use ECC on that cache. This caused some issues; so the "fix" cut that to 4mb mirrored and checked. These were the 250Mhz parts and everybody went to 400Mhz new process stuff instead anyway, but it was a bit of a thing at the time. However, the fuss meant that generation of CPU hit ebay cheap and in volume, which I was quite grateful for.
- p1mrx 6y agoI have an Intel C226 chipset with an i3 CPU, ECC RAM, and Linux, but as far as I can tell, there is no way to determine whether ECC is actually working. I've heard of people collecting known-bad DIMMs for this purpose, or trying to blast their RAM with heat or radiation. Why does this have to be so difficult?
- bl4ckneon 6y agoAs far as I know Intel doesn't support ecc on their consumer platforms. You need a xeon for that.
- dshep 6y agoi3-8100 cpus support ECC
- p1mrx 6y agoIntel supports ECC on i3, but not i5-i7. At least that was the case for the Haswell generation; I haven't checked recently: https://ark.intel.com/content/www/us/en/ark/products/77495/intel-core-i3-4370-processor-4m-cache-3-80-ghz.html https://ark.intel.com/content/www/us/en/ark/products/77495/i...
- ChuckMcM 6y agoSo this is the datasheet for the C226 (https://www.intel.com/content/www/us/en/products/docs/chipsets/8-series-chipset-pch-datasheet.html https://www.intel.com/content/www/us/en/products/docs/chipse...) and you have an i3-8100 or equivalent that does support ECC, then the BIOS will configure how it wants ECC errors reported. On my Ryzen board they are reported via SMI events and there is a chip that you can talk to on the SMBUS that will tell you how many events it has seen since the log was reset. In my experience getting the information isn't "difficult" so much as it isn't "common". On my FreeNAS device (which has the Xeon) it also has an IPMI controller which lets me ask it over the network how many ecc errors have been seen. (and it can act like a remote console as well but we'll leave that for another time :-)) So find out how the motherboard is tracking ECC, then find the tool that talks to that system to ask it for the logs of ECC errors.
- dang 6y agoIf curious see also 2017 https://news.ycombinator.com/item?id=14206635 https://news.ycombinator.com/item?id=14206635 Discussed at the time: https://news.ycombinator.com/item?id=10638324 https://news.ycombinator.com/item?id=10638324
- mark-r 6y agoOne of the things I really like about AMD Ryzen is that they don't reserve ECC just for server chips. It was a major factor in my desktop PC builds.
- adrian_b 6y agoYes, the Intel policy to intentionally cripple their cheaper SKUs, hoping to extort money from customers by pushing them to buy more expensive SKUs, is extremely annoying. For example, right now all the new Tiger Lake CPUs contain a so-called "In-Band ECC" device, which allows error detection and correction even when using LPDDR4x memories, which do not have an ECC variant, like the DIMMs or SODIMMs. This works by storing the ECC codes in a reserved part of the memory, so increasing the reliability is paid by a slight reduction in the memory capacity and in the memory bandwidth. Nonetheless, those who are risk-averse, like me, would prefer this trade-off and would enable the "In-Band ECC". However, you cannot do that because Intel disables the "In-Band ECC" feature on all Tiger Lake SKUs, except on 3 SKUs intended for Embedded and Industrial Temperature Range, which are presumably more expensive. "In-Band ECC" is even disabled on the other 4 Tiger Lake SKUs for the Embedded market, which leaves them without any visible advantage over the normal Tiger Lake SKUs. When Intel competed with Intel, this kind of business decisions probably made money for Intel, but now, knowledgeable customers should better buy from competitors, e.g. the new Ryzen V2000 for embedded applications, which support standard ECC memories without problems.
- mschuster91 6y ago> Yes, the Intel policy to intentionally cripple their cheaper SKUs, hoping to extort money from customers by pushing them to buy more expensive SKUs, is extremely annoying. I agree that this practice sucks, but to play devils advocate - is it possible that this is due to binning / yield maximization?
- temac 6y agoI would think that maybe they can save a very low proportion of chips with ECC not-working, but the large majority would be completely arbitrary market segmentation: handling of ECC is going to be limited to a small relative surface area of the die. Now Intel 10nm still seems to be so bad that maybe it is interesting enough to bin on that, and maybe that's the cause Tiger Lake is even more limited on ECC capabilities than before in the various SKU. Although we already saw a restrictive move on Comet Lake. Instead of continuing to multiply their SKU (I think they had enough even 5 or 10 years ago), Intel should go back to the drawing board and ship interesting microarch on better nodes...
- hyperpl 6y agoIsn't ECC also slower than non-ECC DRAM? Also, I'm not aware of any overclockable Intel CPU/motherboard combos that even support ECC RAM.
- gruez 6y agoI think that's a side effect of product segmentation. ECC RAM tends to be marketed for server applications, so they aim to be JEDEC compliant and call it a day. JEDEC speeds tend to be slow by today's standards. Meanwhile non-ECC RAM is marketed to consumers, specifically gamers who want the highest performance part.
- theevilsharpie 6y agoThis is a limitation of Intel's platform, not a limitation inherent in ECC DRAM. You can overclock them just fine on AMD platforms. That being said, unbuffered ECC DRAM, all other things being equal, probably won't overclock as high since the extra memory chip places a slightly higher load on the memory controller than equivalent non-ECC memory. In that sense, you could consider it "slower."
- glandium 6y agoRegistered ECC memory is slower, but unbuffered ECC memory is not.
- ChuckMcM 6y agoCheck, check, check. I only use ECC DRAM in my desktop these days and I monitor bit flips which on my 64GB desktop happen about once a month[1]. A similar rate on my FreeNAS device which also has ECC memory. Like others, I was super happy when I could by a Ryzen chip at retail prices and get ECC rather than buying a Xeon sku which was an extra $300. [1] When I first built the system I was getting about 2 a week which I tracked down to a 'weak' ram stick. Replaced it (Corsair, and they replaced it for free) and now it's about one a month.
- hyperpl 6y agoHow did you track these bit flips and how did you ultimately narrow it down to "weak" RAM?
- ChuckMcM 6y agoThe memory log has both the bit number and the DIMM "row". They were all coming from the same row, so that was the DIMM I replaced.
- tzs 6y agoIn the early 2000s for a year or two I had a process running on my Linux desktop without ECC that just filled a 128 MB buffer with a pattern, and then checked every minute to see if the buffer had changed. I ran this for a couple of years and never saw a change. I believe that based on published memory error rates for memory at that time, I should have seen a few errors. I am not sure why I never did. Maybe consumer desktop memory was less dense than server or workstation memory, and the published error rates were for the latter? From 2008 to 2017 I had a 2008 Mac Pro at work with ECC memory, and from 2009 to 2017 I had a 2009 Mac Pro at home with ECC memory. I'd occasional look in the memory section of the System Information report and never saw any ECC corrected errors reported. With the Mac Pros, it is possible that I just never happened to check the report between the time of a corrected error and the time the counter was next reset (assuming it resets every boot...if it is cumulative than I have no idea how I never got an error).
- 6y ago
- viraptor 6y agoDoes the RAM publish the fault/correction count on any accessible register/counter? It would be great if we could just query that on existing systems and have the simple answer - does ECC do anything for us. Quick search later - yes, you can: https://linux.die.net/man/1/edac-util https://linux.die.net/man/1/edac-util For all the discussion I've seen about this topic, I wish more people actually published their numbers.
- formerly_proven 6y agoThe RAM doesn't publish anything because it isn't doing any ECC. ECC memory is just wider memory (72 bits instead of 64 bits per row). The ECC itself is performed by the memory controller, which makes sense, because doing ECC in the memory would leave the memory bus itself unprotected.
- nroets 6y agoECC implies error correcting. That implies more extra bits to store Hamming codes. But you can have far fewer redundant bits if you only want to detect error s. (Parity). In the worst case the computer reboots. But you must foresee this because hardware can die for many reasons (power failures, component failures etc)
- gruez 6y agoA sibling commenter says he gets one bit flip per month on his computer. I don't know about you, but I don't like the prospect of my computer randomly rebooting once a month.
- viraptor 6y agoThere's a long way from a single bit flip to a reboot. You're not likely to even notice a single event. For example it could be any bit of a single pixel on the screen. Or one letter in all the text displayed over a month. Even if it happened in the code, you'd have to hit some function in the kernel which has any impact at all, instead of for example sending a bad packet.
- nix23 6y ago>For example it could be any bit of a single pixel on the screen. Yeah or one bit in your checksum...imagine the fun :)
- kingosticks 6y agoIn my field, what's interesting is how ECC memory corrections are going from being considered abnormal - an indicator you have faulty hardware and should replace it - to totally normal and an absolute must-have part of your implementation.
- mixmastamyk 6y agoIt's a damn shame Intel has kept ECC out of the hands of regular folks. AMD lagged for a long time as well, but apparently newer Ryzens support it. Is there one for laptops? What about Apple? Will they support it now they're unshackled from Intel?
- vardump 6y agoAMD has supported ECC for a pretty long time. My 11 years old AMD consumer desktop sure supports ECC. It's just Intel that doesn't.
- mixmastamyk 6y agoI've read that before, but I remember folks in similar threads say they were never able to confirm it actually working.
- theevilsharpie 6y agoI can confirm that ECC DRAM worked as expected on my current AMD Ryzen 9 3900X, an AMD Phenom II x6 1090T that I had in a previous build, and an AMD Opteron 185 that I had in a build before that. The operating system detects the error-correction capability in the memory controller, and errors are properly detected (tested by reducing the memory timings slightly below stable values). I don't think there's a mass-market laptop with ECC support, but AMD's desktop CPUs have supported ECC for as long as I can remember. That being said, it's been an optional feature that is up to the motherboard maker to implement, and not all do.
- adrian_b 6y agoMost so called "mobile workstation" laptops with 45-W H CPUs, from Dell, Lenovo or HP, support ECC (if you choose a H Xeon CPU instead of the equivalent Intel Core CPU). I have a Dell Precision laptop from 2016, with a Skylake Xeon and ECC memory.
- 6y ago
- anonymousiam 6y agoI am surprised that ECC is not more popular. RAM errors (SEUs) are common in every platform. One hacker that I worked with almost a decade ago did this BlackHat presentation with some analysis that correlates errors with increased temperature (by geo-locating clients of "bit squatting"). I put together a new SuperMicro server a few months ago and went with 256GB of ECC. Yeah it's very expensive, but it's absolutely worth it if you care about the integrity of your data. See: https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinaburg_Bitsquatting_WP.pdf https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabu... Where are you now Artem?
- jeffbee 6y agoIt's not even that expensive. An 16GB "server" DIMM costs what, $65 at retail? Apple will charge you $200 for 8GB of non-ECC. The real cost on ECC memory is getting a CPU and platform that support it properly.
- anonymousiam 6y agoFour of these weren't cheap. ($1,441.36) https://www.amazon.com/gp/product/B07YF249FX https://www.amazon.com/gp/product/B07YF249FX
- Dylan16807 6y ago> An 16GB "server" DIMM costs what, $65 at retail? Clocked at 2400MHz, sure. I can't find a single non-sketchy listing for even 3200MHz unbuffered ECC memory, let alone the 3600MHz you want for current Ryzen chips. And those are just the speeds at reasonable prices, ignoring the 4000-5000 range. At least DDR5 should improve things.
- theevilsharpie 6y ago> I can't find a single non-sketchy listing for even 3200MHz unbuffered ECC memory https://www.crucial.com/memory/server-ddr4/mta18adf2g72az-3g2e1 https://www.crucial.com/memory/server-ddr4/mta18adf2g72az-3g... > let alone the 3600MHz you want for current Ryzen chips Current DDR4 JEDEC-approved speeds top out at DDR4-3200. Anything beyond that is technically overclocking, even if it's overclocked from the factory and marketed to run at those speeds. Assuming that your platform supports overclocking DRAM, you can also run DDR4-3200 ECC memory at 3600 MT/s, if you feel that the performance uplift is worth the risk of stability issues.
- rkagerer 6y agoBeen using a home-built workstation for over a decade now with 64GB ECC RAM (high performance sticks for the time). It's one of three such PC's in service here. Over the years a couple modules developed problems and were replaced. Glad for the ECC. Those errors might have manifest as silent corruption issues instead.
- adrian_b 6y agoYes, I also use only ECC memories in all my computers for the same reason. When they are new, memory errors are very seldom, if at all. Nevertheless, after several years of use, I had many cases when some DIMM modules began suddenly to have frequent errors and being notified by ECC allowed me to replace the modules (or the motherboards if they were very old) before causing unrecoverable data corruption.
- jacobwilliamroy 6y agoECC RAM is also important if you want to run OpenZFS as a lot of its speed comes from caching previously read data in RAM. If there is a RAM error, that error will be propogated back to disk and OpenZFS currently has no way to prevent this from completely destroying all of your data.
- gommm 6y agoI'm actually hesitating, my freenas has ECC ram (it's a nas and it'd be stupid not to) but I'm building a desktop around a ryzen chip that will run pfsense as well as multiple vms. I wanted to buy ECC but I have a hard time finding 2x32GB of DDR 3600, no XMP and I don't care about the data on that desktop as much as I care about the data on my nas.
- theevilsharpie 6y agoJEDEC specifications for DDR4 top out at DDR4-3200, and anything beyond that is vendor overclocking with whatever settings that particular vendor deems is suitable. While there's nothing technically preventing vendors from factory-overclocking ECC memory the same way, ECC's (perceived) main audience is reliability-critical systems, and so it doesn't make sense for vendors to market a feature that their target audience wouldn't appreciate. What you can do is get two 32 GB DDR4-3200 memory modules (example: https://www.crucial.com/memory/server-ddr4/mta18asf4g72az-3g2b1 https://www.crucial.com/memory/server-ddr4/mta18asf4g72az-3g...), and then overclock them yourself to 3600 MT/s. As with any overclock, your results may vary, but that's also the case with factory overclocking as well.
- wscott 6y agoBack when I used to work for BitKeeper, having support problems ultimately be traced back to bad memory (or disk, or nfs, ...) was a common occurrence. BK would checksum everything it writes to disk and verify those checksums when it read it back. And since it uses 'simple' arithmetic checksums rather than 'secure' ones it was possible to pre-compute the checksum we would get when writing new files so even memory errors would be noticed if it changes what we will be writing to disk. The hard part when doing support is convincing the customer that the errors in your software are from their systems. I had to learn the failure patterns of a couple large filers so I could look for them. Now we use git which is fine, and git uses much stronger checksums. My main concern is that since those SHA{1,256} checksums are expensive they are typically generated only once and then only used for lookups. If you intentionally corrupt git files, it is hard to have git notice without explicitly running a check. These checks should be done it cron nightly as part of backups.
- cm2187 6y agoI guess it depends what you use the server for. If you are a bank dealing with bank account balances you don't really want a bit to be flipped even once a year. If one character in a web page index is flipped in google's indexation service, who cares.
- nix23 6y ago>If one character in a web page index is flipped in google's indexation service, who cares. >>When Google used servers without ECC back in 1999, they found a number of symptoms that were ultimately due to memory corruption, including a search index that returned effectively random results to queries.
- cm2187 6y agoYeah. Who cares. Out of the billions of search queries, a handful will return irrelevant results.
- nix23 6y agoGoogle obviously cared, since google has everything on one platform who cares if your ad-sense shows wrong numbers? Your Gmail, Fotos, Drive queries...who cares right? Oh and not to mention debugging, can you trust your program or your hardware, who makes the wrong query, how to find out which node needs to be restarted?
- cm2187 6y agoIf it's a very rare event you might not care. Look at how google applies security rules. They don't mind closing a few accounts unfairly if it means they can automate the whole process. I can't imagine they would care about flipping a few pixels in a handful of the hundred of millions of photos they host.
- nix23 6y agoYeah i stop here, you obviously have no clue since you compare apples to...nothing at all.
- nix23 6y ago>I'm pro-ECC, but if you don't have regular backups set up, doing backups probably has a better ROI than ECC. But since you don't know that you backup a corrupted file for years i'm pro ECC and pro FS-Checksumming. I learned my lesson with some corrupted Sound-Files, the bad thing, i had to check every single file near that death block cluster, the good thing, all of them where ripped from my CD's.
- willis936 6y agoIf every system had ECC then we would lower the probability of a virus spontaneously appearing in memory. Is our data security worth lowering the probability of computer genesis?
- ekns 6y agoPlugging in a past question I had related to this [0]: Given that ECC RAM is not abundant, why don't operating systems have self-checks for RAM? Instead we only tend to have near-post mortem checks with memtest86 and the like. [0] https://news.ycombinator.com/item?id=24642062 https://news.ycombinator.com/item?id=24642062
- theevilsharpie 6y agoWithout hardware-level ECC support, how would a memory self-check (that isn't along the lines of a diagnostic like Memtest86) work? You would have to have some way of detecting that memory is corrupted, which non-ECC platforms do by writing a known pattern to memory and accessing it to verify that the pattern matches. You obviously can't do that during runtime because that memory is being used by the kernel and applications.
- fulafel 6y agoThe idea in ECCs [1] is that you store some ancillary data that can be used to rectify a bounded amount of error introduced in the stored data. A straightforward virtual memory system implementation would be to have this information stored alongisde page tables and the OS would go through the pages and update their ECCs periodically (probably at a limited rate so as not to cause load spikes). The devil in the details would be the bookkeeping about when a page contents has been written to and the ECC is invalid (soft page faults?). [1] https://en.wikipedia.org/wiki/Error_correction_code https://en.wikipedia.org/wiki/Error_correction_code
- cute_boi 6y agoHere on Norway ECC ram is absolutely necessary due to cosmic activity?. I remember there used to have lot of problem in software where checksums would fail. Ultimately it was due to ecc issue.
- swiley 6y agoRam definitely does flip bits more often than might be intuitively obvious. Most people who have worked at places with a large physical server deployment have stories, and there's row hammer, but I think my favorite is the paper on DNS bitsquatting. You can see the statistics on flipped bits on normal PCs.
- CalChris 6y agoApple should spec an ECC configuration for their M1 MacBook Pros. They currently want $200 for an extra 8GB on their M1 Air and 13" Pro. I'd gladly upgrade to the Pro and pay that $200 for an extra 8GB if the 16GB was ECC. That'd be $1499 for a 13" Pro. ECC would be the tipping point. Otherwise, $999 for an Air will do.