7 ms·
ECC and DDR5
- AlexandrB 3mo agoInteresting article, and I agree that ECC needs to be more widely available. But I think this is a major overreach: > We need ECC RAM to be more widely used. Ideally we would have some government action to force this given the ongoing cost to society in corrupted data and lost time due to RAM hardware errors. I think that at minimum we need sufficient taxes on non-ECC RAM (and EC4 RAM for DDR5) to make it more expensive when bought new than ECC RAM. ECC RAM is not needed for every application and mandating it for every gaming rig and machine used to run Chrome will just increase prices unnecessarily. 99% of the time it's also not a "cost to society" but a cost to whoever owns/uses that machine. If they want to eat that cost, it should be up to them.
- graemep 3mo ago> ECC RAM is not needed for every application and mandating it for every gaming rig and machine used to run Chrome will just increase prices unnecessarily. The price increase will be very small if manufacturers are no longer able to segment the market to make a higher margin on those who need it. No one wants even games or web browsers to crash randomly. > If they want to eat that cost, it should be up to them. They need to be given the choice. Most people do not know they even have the choice, and they are not offered it at a reasonable price. A cost to people in general is a cost to society.
- procaryote 3mo agoWithout quantifying the cost to society that someone's web-browser crashes sometimes, it's hard to make a rational case that this issue is so dire that it needs specific regulation. I'm not sure making it illegal to sell consumer grade hardware is the boon for the people you imagine
- markonen 3mo agoCrash is the best-case outcome for a browser bit flip. Here's bitsquatting: https://ripe92.ripe.net/programme/meeting-plan/sessions/112/MFPHZ7/ https://ripe92.ripe.net/programme/meeting-plan/sessions/112/...
- etbe 3mo agoNo-one ever proposed making non-ECC RAM illegal, that's a straw-man. Restricting risky items is standard practice for all governments. All the "use by" dates on food etc are from government regulation. The current experiments with "raw milk" shows that you can get away with flouting food safety regulations and not have immediate problems.
- gruez 3mo ago>The price increase will be very small if manufacturers are no longer able to segment the market to make a higher margin on those who need it. AMD doesn't have this segmentation. Where's the hoard of people building systems with ECC ram? At best you have a few people on HN saying they're doing it, but in practice nobody actually does it.
- wmf 3mo agoECC UDIMMs are crazy expensive, presumably because they don't have economy of scale. Mandating ECC would solve that.
- sdsdssweew213 3mo agoEven before current RAM price craziness ECC UDIMM's were hard to find, and often more expensive than non-ECC. I was lucky to grab 128GB for similar price to non-ECC a few years back. More people would use them, if they were available. RDIMM's do not work on normal consumer boards, so obviously people won't be using them.
- wahern 3mo agoAMD does segment. Only their EPYC, Pro, and some embedded models officially support ECC. Many other AMD chips have unofficial support, but if you care about data integrity, it feels like a gamble. You have to rely on suspect motherboard support and generally rely on gossip about what works well and what doesn't.
- projektfu 3mo agoImportant data should probably be worked on and stored with error-correcting codes, in addition to the hardware-provided ones. The Btrfs mention in TFA suggests that maintaining integrity of the Btrfs metadata is not considered part of the job description of Btrfs, but perhaps it should be.
- deleted 3mo ago[deleted]
- etbe 3mo agoBTRFS has checksums on all metadata. It isn't designed to cope with bad RAM but has been shown to be somewhat resistant to it to the degree that BTRFS developers can recognise symptoms in problem reports. Whether a filesystem could be designed to operate in the face of random memory changes would be an interesting computer science research topic. But for getting working systems having RAM just not fail is a better option.
- nahuel0x 3mo agoSomething important not discussed here is that you need not only error correction (when possible) but also error reporting, so you can know you RAM is failing. Sadly, is not possible to get DDR5 on-die ECC error counts reported to the OS. Alternative solution, use IBECC with DDR5, where the CPU reserves a portion of the RAM for parity bits and does the check / reporting.
- inigyou 3mo agoMy Linux installation came with an AMD MCE driver that reports them to dmesg. It logs a few corrected errors a day.
- rubatuga 3mo agoIsn't that a lot? I see them only when heavily over clocking .
- adrian_b 3mo agoA few errors per day seems much too frequent, unless you live at a high altitude. Normal good DIMMs, at least when new, should not have errors more frequently than one error per many months. Frequent errors may appear with memories that are not seated well in their sockets, or which are old, at least several years old. Frequent errors may also be caused by more general computer problems, like a bad power supply unit.
- bikelang 3mo agoCurious to learn more here - why would altitude play a role here? And when you say high altitude - are we talking La Paz, Bolivia (~12k feet) or Denver, CO (~5k feet)?
- dsr_ 3mo agoThe primary source of uncorrelated bit-flips is, effectively, cosmic radiation. (Correlated bit-flips are likely from a manufacturing error... or, at least in one case, from excessively radioactive ceramic packaging.) The more atmosphere you have to randomly absorb high-energy photons, the better. Higher density memory drops fewer electrons in each well, which means a lower-energy photon can change the state. Corolary: submarine datacenters are much better than orbital datacenters. Cheaper to cool, construct, shield and access.
- cyanmagenta 3mo agoThe problem I have with ECC discussions is that it’s always anecdotal. There is one Google study finding memory flips with some probability in their datacenters, but it was awhile ago. Everyone repeats the statement DDR5 on-die ECC isn’t a replacement for regular ECC, but nobody ever shows data on DDR5 bitflips. That’s not a knock on this article per se, but it’d be nice to have actual data here.
- phkahler 3mo ago>> Everyone repeats the statement DDR5 on-die ECC isn’t a replacement for regular ECC, but nobody ever shows data on DDR5 bitflips. That's because the on-die ECC cannot detect errors in the transmission of data from the DIMMs to the CPU. It seems like on-die ECC is meant to "hide" some level of errors to make less reliable RAM chips appear good. I have no idea if that's a reasonable way to do it, but it seems like it could be misused. Like TFA suggests, we need to know how the two levels of ECC interact.
- cyanmagenta 3mo agoRight. Everyone agrees that real ECC is more robust, and production servers of course run it. But it seems to me that if most RAM bitflips are in the module itself (rather than in transmission to CPU), then on-die ECC could in theory make the probability of a bitflip in practice so low that for things like home servers it isn’t worth the price premium for real ECC. But without data on this, everyone is just speculating.
- zargon 3mo agoThis is the attitude that needs to change. If RAM is 25% of the system cost (say for a home server in your example), then the cost for ECC is 3% of the system cost. There is no world where accepting data corruption in order to save 3% cost is a reasonable tradeoff. ECC should be standard, period.
- Wowfunhappy 3mo ago
- Havoc 3mo agoGot ECC udimm for my ddr4 server and was surprised to see it picking up errors occasionally (once every couple months). Likely from a weakness in one of the sticks. This far it’s always corrected it though so opted to keep them anyway (nobody wants to be minus 32gb in these trying memory times) With normal sticks I’d not have know that there is a potential issue.
- zargon 3mo agoYou should be able to get the stick replaced under warranty. Every memory manufacturer I have ever seen has a lifetime warranty. As long as you RMA it before replacements become unavailable.
- Aurornis 3mo agoThere should not be errors during normal use at rated speeds. With non-ECC RAM running a memory test and getting any errors means it’s not stable. The memory tests run at maximum bandwidth constantly so a few days of full bandwidth memory testing might produce as many transmissions as your months of regular server use. Seeing bit errors (even if corrected) means something is not right. The ECC can correct it for a while but I would want to RMA that stick if it’s in warranty.
- Havoc 3mo agoAlas this is white label mem off eBay and fair bit of time has elapsed so not confident exchange would work. Minor miracle the ECC works at all given that At some point I should try running it at slightly lower speed i guess. It’s very hard to troubleshoot something that only happens every month or two though Never seen it fail a memtest but I can see it in the ECC stats since those cover weeks/months.
- zargon 3mo agoIf you bought it from one of those ebay outfits that specialize in memory modules, they also often offer a lifetime warranty. Usually those sellers put their own sticker on the module. Worth a check.
- ck2 3mo ago16GB ECC DDR5 was $50 early 2025 it is now $500 when the AI/datacenter bubble finally bursts we are going to have some amazing hardware deals
- bartleeanderson 3mo agoI had a conversation with chatgpt about that very thing, whether it was a bubble or not and how the big players could absorb the cost and where it would all land and the interesting thing was if China gets its foothold and grabs enough share because the prices are too high even being subsidized, that could cause real trouble. But the point was its not a bubble in the conventional sense. Have your own conversation about that with an AI.
- HDBaseT 3mo agoI'd rather just have the conversation here. I don't need some hallucinated random seed response to tell me what China can and cannot do. Of course an AI is going to tell you AI isn't a bubble.
- etbe 3mo agoI'm planning some RAM purchases for the end of this year, I expect the crash to happen in a few months.
- warmwaffles 3mo agoWhen that bubble does finally burst, you won't be able to afford those amazing hardware deals due to the economy being in the tank for a while.
- lightedman 3mo ago"Ideally we would have some government action to force this given the ongoing cost to society in corrupted data" Why is it so often Australia with the "We must force unnecessary things upon people" attitude?
- Xirdus 3mo agoBecause despite geographical location, Australia is a western country, and all western countries are like that.
- gleenn 3mo agoThat particular quote did stand out to me. I think the counter claim is that if everyone uses software everywhere for everything, there is a very real societal cost for damaged data and if it's hard to detect then there is no way to prevent or measure harm. I think the "invisible hand" free market is showing some serious strain in modern times. People stopped using leaded-gas because it was causing measurable decreases in IQ for generations. I don't care how much you want to save a few dollars if it harms a lot of people and is difficult to measure. Everything else seems like a race to the bottom price so pushing the market in this manner doesn't bother me in the least. If you are running a server with others' data at stake, it seems like an obvious benefit for society on the whole to address the problem rather than serve some cheap datacenter operators.
- some_random 3mo agoI think it's a pattern that reinforces itself, as a nanny state develops it's opinion of it's people worsens and it's more willing to use force to compel them.
- _factor 3mo agoRight, the opinion shifting to the people being unworthy of uncorrupted data unless they’re wealthy enough to pay the artificial extortion price for integrity.
- some_random 3mo ago
- jcalvinowens 3mo ago> One thing that concerns me is the possibility of on-die ECC interacting with ECC on the motherboard and reducing it’s effectiveness DDR5 on-die ECC detects and corrects one-bit errors. It cannot detect two-bit errors, so it will miscorrect some of them into three-bit errors. My understanding is that the on-die error correction scheme is specifically specially designed such that the resulting three-bit errors are mathematically guaranteed to be detected as uncorrectable two-bit errors by a standard full system-level ECC running on top of the on-die ECC. But I've never found a real authoritative reference that directly says that.
- cobbzilla 3mo agoVery good point. It’s nontrivial but not impossible to determine this, I am very curious! One nightmare scenario is a 3-bit error being detected as a correctable 1-bit error upstream, resulting in data corruption. I agree it’d be really nice to know that could never happen!
- BobbyTables2 3mo agoWhat about rowhammer attacks ? Wonder if they could be used to introduce “correctable” errors…
- etbe 3mo agoThe people designing RAM and motherboards and the people working at JEDEC will of course know more about these things than almost everyone who reads hacker news. But I'd still prefer not to just blindly trust that they do the best things for us and have some documentation available. Is anyone here good at navigating patent records? I would be surprised if there isn't something to be revealed in a patent but I don't know how to find it.
- exabrial 3mo ago> We need ECC RAM to be more widely used. Ideally we would have some government action to force this given the ongoing cost.... The rest of this article is great. This, however, is absolute garbage and nonsense. The last thing you want is more regulations and "government certified systems". Take a look at the bullshit with age verification bills and operating systems.
- wmf 3mo ago"This, however, is absolute garbage and nonsense. The last thing you want is more regulations and government certified systems." — the guy who designed the Ford Pinto
- fleroviumna 3mo ago[dead]
- mihular 3mo agoWe need any RAM at this point.
- jmb99 3mo agoHow is “all RAM sold in consumer PCs must be registered ECC” equivalent to age verification laws? I’m not saying I agree with the premise but that is a bit of a leap
- snvzz 3mo agoI run a 9800x3d with 96GB ECC (2x 48GB DDR5-5800). It was a pain to select a compatible board and source the RAM but glad I did, in June 2024... the RAM prices went up immediately after. non-ECC should be a kernel taint in Linux. Newer versions of Windows (12?) should also refuse to boot without ECC. It should be mandatory in DDR6. The insanity must end.
- etbe 3mo agoYes a kernel taint is a really good idea!
- nu11ptr 3mo agoI've always thought it odd that we don't use ECC as a standard on all computers. I think people really downplay the impact of memory issues. They can be devastating, especially over time. They will slowly corrupt your file system, documents, binary files, code, everything. You notice when things like your compressed files start giving CRC/checksum errors or your downloads don't match SHA-512. You are then left with this "icky" feeling. What parts of your work are compromised? Obviously this is far worse with things like binary files and structures than those built on readable text. I have normally just rebuilt my system from scratch in these instances (after fixing the issue of course), but not all our files are replaceable, and you forever and always might have corrupted files you are unaware of that got written during this time. You might not find them until years later.
- auspiv 3mo agoThis is/was classic market segmentation. Want ECC? Pay up for Xeon (server/workstation CPU)
- scheme271 3mo agoAMD has historically supported ECC on all it's chips. That's changed in the last gen or two where some of the lower end APUs don't support ECC but the Pro APUs do and all the ryzen stuff (5, 7, 9) support ECC if you get a motherboard that supports it.
- mixmastamyk 3mo agoTheoretically, though I’ve not seen any certified builds. At least advertised. Do tell who offers an ECC laptop PC at a reasonable price.
- scheme271 3mo agoLaptops are pretty much unavailable with ECC especially with soldered memory becoming common. On the desktop side, there's multiple motherboards that explicitly note that they support ECC memory. AsRock and Asus both include ECC memory on their QVL lists. I'm not sure how many ITX boards support ECC though so getting it on something that's not a tower might be tough. But it's definitely possible, I have an ECC setup on my desktop and it's caught a memory error or two over the last few years.
- monster_truck 3mo agoThis is not a very good post. Claiming that every issue you've experienced is from seating errors? Come on man, at least deliberately reproduce that scenario once. I've tortured motherboards for fun and never managed that on purpose. We did pry every cap off of an old K6 board while it was on and... it just kept running. It didn't come back after we turned it off The amount of time it takes to reseat memory, even if the case is already open and ready, is also more than enough time for it to cool back down. If you have decent memory running at what it's rated for and you're seeing more than a couple per year, good odds are it's getting too hot or the load line calibration is not aggressive enough. E: I have never, and I mean literally never, seen age or abuse play a factor.
- etbe 3mo agoI can only report what I have seen. Feel free to write your own blog post reporting what you have seen.
- monster_truck 3mo agoI do, often, and respect the time of my readers enough to verify my claims instead of presenting anecdoes as fact.
- etbe 3mo agoI respect the time of my readers by accurately presenting my claims. It seems that everyone other than you was able to understand "My experience of memory errors".
- sandreas 3mo agoOne thing I also did not know until recently was that there actually is ECC unified memory. I noticed the difference because in a Hardware forum someone wrote that the framework desktop does not have ECC while an HP G2 Workstation also featuring the Ryzen 395 Max but in the ECC supporting variant paying a premium of +600 bucks.
- hamilyon2 3mo agoDoes encrypted memory give ECC-like detection benefit, along with rowhammer protection, for cheaper cost (only additional computation), and 100% reliable?
- howard941 3mo agoECC is essentially mandatory for ZFS. Assuming that is you prefer your data uncorrupted.
- scheme271 3mo agoIt's preferred but not having it still works if you're willing to leave some potential avenues for errors to get in. However, it's not like not having ECC is a recipe for a corrupted FS.
- throwaway270925 3mo ago> ECC is essentially mandatory for ZFS Please stop repeating this old myth, it is - and always has been - BS and needs to die. See for example: https://kldload.com/zfs-wiki/myths https://kldload.com/zfs-wiki/myths
- deleted 3mo ago[deleted]
- hangonhn 3mo ago"I have used more than a few systems with ECC UDIMMs and I recommend avoiding them if convenient as ECC UDIMMs are expensive on the second hand market while ECC RDIMMs can get very cheap." Boy did I learn that the hard way. I found an ASRock server motherboard that fit my needs for a NAS perfectly. I just assumed all ECC memory were registered so I didn't even bother to check. I had already put everything together minus the memory. I had actually purchased the memory and had to return them because they were RDIMMs and get the ECC UDIMMs for quite a bit more. Luckily this was just before the AI induced price spike.
- etbe 3mo agoA very large portion of people who go from working on consumer grade systems to getting enterprise grade stuff for their home lab make a mistake like that, I did some similar things, my biggest mistake was spending $300 on non-refundable ECC RDIMMs without realising that my server (the cheapest Dell Txxx server) took ECC UDIMMs. I hope that my blog posts on these topics will save some people some money.
- burnt-resistor 3mo agoECC end-to-end means carrying sufficient parity along the whole way including at-rest (memory and storage hierarchy), links, buffering, and computation. Skipping ECC in transposer/buffering/bus circuitry creates a source of silent errors. Silent errors are way, way too prevalent across billions of users and the vast majority never understand the source of such malfunctions or know they happen.