7 ms·
One thing I'd like to understand better about DDR5 is how well the built-in ECC is going to work to improve reliability. DDR5 comes with "chip level ECC" [1] of
by DCKing 6y ago
One thing I'd like to understand better about DDR5 is how well the built-in ECC is going to work to improve reliability. DDR5 comes with "chip level ECC" [1] of which the main purpose is to be able to better sell highly complicated memory chips with minor defects.
But as a consequence as I understand, it will allow for the correction of single bit memory flips. With regular DDR4 or previous generations, you don't get any error correction. Any bit error in your DDR4 modules has the potential to corrupt data. If you want to be protected from that, you will need to get ECC memory.
Unfortunately, anything with "ECC" in hardware for unfortunate reasons gets labeled with an "enterprise" sticker. And that means a certain price level, and a certain power consumption. (Yes I know you can get Ryzen boxes that work with ECC, but that's still PC sized hardware for hundreds of dollars).
If DDR5 can bring error correction to the masses - like in single board computers, 10W NAS boxes, smartphones - that would be pretty cool. But I'm not sure whether my reading of that is correct.
[1]: https://www.anandtech.com/comments/15912/ddr5-specification-released-setting-the-stage-for-ddr56400-and-beyond/708566 https://www.anandtech.com/comments/15912/ddr5-specification-...
- klodolph 6y agoI wouldn’t be surprised if at some level that physics has forced the manufacturer’s hand—that previously low error rates are now unacceptable when you multiply them by 64GB.
- kasabali 6y agoWhat I read around is that's not the enterprise ECC, it's more akin to ECC bits used in flash memory. It'll allow manufacturers to play fast and loose with memory.
- mc32 6y agoSo in other words, error prone memory modules don’t have to be tossed and can be sold because they will be self correcting...?
- kasabali 6y agoThat's what I understand. Frequencies will be higher and manufacturing process denser later in DDR5 lifecycle so they must be expecting more errors.
- DCKing 6y agoI guess what I don't understand then is what big advantage "enterprise ECC" has left over this DDR5 "non-enterprise ECC". (Seriously, why is ECC "enterprise"? Everybody wins with memory error correction.) If regular DDR5 can correct single bitflips, it is on par in correction capabilities with "enterprise ECC" DDR4. Maybe this won't allow for the detection of multiple flips, and maybe won't even report single bit flips to the OS (it'll just fix them silently). I suppose there's no big need to support detection and reporting for the vast majority of use cases. Ryan Smith at Anandtech in the link above says as much: "Between the number of bits per chip getting quite high, and newer nodes getting successively harder to develop, the odds of a single-bit error is getting uncomfortably high. So on-die ECC is meant to counter that, by transparently dealing with single-bit errors." But for my purposes, if just the correction capabilities are on par with DDR4 ECC I'd be absolutely fine with that. And I guess that goes for many people. Even while using ECC memory now at home, I'm not monitoring the correction statistics and I'm guessing few people do in general. It might as well be silent today if you ask me.
- wmf 6y agoMost servers have chipkill ECC that can survive an entire 4-bit chip going bad so that's more powerful than classic SECDED. I don't know how often chipkill kicks in though.
- ip26 6y agoI don't know what "enterprise ECC" means, but there are certainly grades of protection, from single bit error detection (parity) thru triple error correct quadruple error detect, and inline vs non-inline correction schemes. (for the latter, the machine has to stop, go back, fix the error, & resume, potentially at significant performance cost)
- Freaky 6y ago> I guess what I don't understand then is what big advantage "enterprise ECC" has left over this DDR5 "non-enterprise ECC" ECC should be end-to-end, so it detects and (hopefully) corrects errors anywhere along the path, not just within a chip. Step 1 of handling a lot of ECC correction events is to reseat the DIMM, because often it's just an issue with the connection, not actually a memory defect. And you may not care too much about reports of correction events, but you definitely want to see correction failures reported - the point is, after all, to avoid corruption.
- wmf 6y agoI expect Intel to still cripple some aspect of DDR5 ECC on consumer chips; maybe it will correct errors but the memory controller won't report them. Or maybe it's possible to disable ECC even though it's already implemented. I also expect servers to use two levels of ECC to provide chipkill and also to keep server RAM more expensive than consumer.
- DCKing 6y agoRyan Smith in the link above seems to suggest error correction will be done transparently anyway, so it seems like it won't reported to the OS. So it doesn't look like Intel could cripple it even if they wanted to.
- throw0101a 6y ago> Ryan Smith in the link above seems to suggest error correction will be done transparently anyway, and it won't reported to the OS. I once heard a rant from someone on how not reporting this to the OS is really bad for diagnosing issues, even soft errors that are auto-healed. (It could have been from Bryan Cantrill, but couldn't say for sure.)
- DCKing 6y agoI do think that there will still be some people interested in ECC detection and reporting, but that's not really why I'm interested in ECC memory. The dividing line "enterprise = error correction with monitoring / non-enterprise = silent error correction" is much more sensible than "non-enterprise = no error correction at all, good luck" IMO. Isn't the primary reason why people are looking into diagnostics is because it's very hard to determine whether ECC is working in the first place? Because it depends on the particular hardware setup? If the spec states that all DDR5 is supposed to have internal error correction anyway, then I'm happy to take for granted error correction is working until I read about the scandals of non-spec cheap DDR5 :)
- throw0101a 6y ago
- nine_k 6y agoA number of Celerons 3XXX and 4XXX support ECC. They are intended for storage systems, like a NAS, or industrial electronics.
- DCKing 6y agoIf you could point me to a <20W machine with ECC memory that I can buy online like a mere mortal human person, then I'd very interested! The only thing that comes to mind in that category for me is the PC Engines APU2. The APU2 is a very neat piece of kit, don't get me wrong, but it being the only option is not great either.
- osamagirl69 6y agoIntel released the Xeon D series to fill this niche https://ark.intel.com/content/www/us/en/ark/products/series/87041/intel-xeon-d-processor.html https://ark.intel.com/content/www/us/en/ark/products/series/... The D-1529 can be TDP limited to 20W (at a whopping 1.3GHz), but if you goal is idle power consumption under 20w you are better off with one of the 35w TDP D-1602 since it is only a dual core and has the lowest standby consumption. The D-1518 is also a popular choice, and basically the same as the D-1529 but with a 35W tdp limit so it can turbo to 2.2GHz. You can get a barebones motherboard (even mini-ITX is available for making small machines) for about $500 or a complete computer for a bit under $1000 ex https://www.newegg.com/supermicro-sys-e300-8d-intel-xeon-processor-d-1518-2-2-ghz-cpu-tdp-support-35w-fcbga-1667/p/371-001M-00001 https://www.newegg.com/supermicro-sys-e300-8d-intel-xeon-pro... note - the xeon-d series is not socketed, so the motherboard price includes the CPU. Make sure to get board that takes full size DIMMS since there is not a lot of ECC laptop memory on the market.
- DCKing 6y agoI'm aware of the Xeon D line, but sadly that was also squarely aimed at the enterprise albeit maybe a smaller scale enterprise. It's not something you can easily go out and buy any more, and when you do it's both very expensive, still PC sized, and not any more power efficient at idle or configured TDP when compared to a new Intel workstation or new AMD setup.
- lucb1e 6y agoI actually tried really hard to get RAM to corrupt a bit for a school project and didn't manage a single bit flip. How often have you actually heard of data corruption due to non-ECC memory? Either yourself, any degree of 'friend of a friend', or perhaps a study that looked into the matter with more success than I had. I don't mean a newspaper story because exceptional cases are reported because they're rare exceptions rather than common enough that we'd be likely to come across it in our lifetimes.
- deegles 6y agoBunch of links to papers here: https://stackoverflow.com/questions/2580933/cosmic-rays-what-is-the-probability-they-will-affect-a-program https://stackoverflow.com/questions/2580933/cosmic-rays-what... e.g. "2009 Google's paper "DRAM Errors in the Wild: A Large-Scale Field Study" says that there can be up to 25000-75000 one-bit FIT per Mbit (failures in time per billion hours), which is equal to 1 - 5 bit errors per hour for 8GB of RAM after my calculations. Paper says the same: "mean correctable error rates of 2000–6000 per GB per year". "
- Yeroc 6y agoAnd there are ways to manipulate RAM access patterns to induce errors as described in the initial Rowhammer attack paper plus later RAMBleed papers. Hopefully this newer DDR version is designed to be resistant to this type of attack.
- lucb1e 6y ago> 1 - 5 bit errors per hour for 8GB of RAM after my calculations That is way off from what I'm seeing. When launching Factorio I use 90% of my 8GB RAM and never once have I noticed data corruption, and I could tell you how many hours I've played but that would be embarrassing. The test I did in school with heated-up RAM (the internet said that's when flips should occur more often) also wrote many many gigabytes without a single failure. Not sure what hardware or temperatures that source is running but it's not DDR3/DDR4 at heats below hairdryer melting temperature because that's where I had to stop the experiment with zero failures.
- NelsonMinar 6y agoOne of the great ironies of modern computers is we dispensed with ECC just as we started ballooning out the size of RAM and shrinking the transistors so that single bit errors were more likely. I'd be very grateful for system ECC.