11 ms·

ECC RAM should be a human right

211 points by zdw 4y ago

deleted 4y ago
[deleted]
PaulKeeble 4y ago
ECC has been used as an artificial market segmentation mechanism for a long time and it needs to come to an end. RAM just like SSDs and HDDs ought to have some amount of self protection against basic errors, all places where data is stored even for short periods needs this.
- deadbunny 4y ago
  > ECC has been used by Intel as an artificial market segmentation mechanism FTFY
  - leguminous 4y ago
    What's really annoying is that the current generation of Intel CPUs support ECC, they just don't implement it in consumer chipsets. You can get working ECC with a W680 motherboard, but those are very expensive and availability is slim.
  - nimish 4y ago
    And AMD, and Nvidia.
    - hamandcheese 4y ago
      AMD ryzen CPUs have ECC support, but it varies by motherboard. My home NAS runs a Ryzen in an ASrock Rack motherboard with ECC ram. I never did figure out a way to verify that the ECC is working/if it is able to report errors (to the kernel?). It was also a bit hard finding the right ram, but there’s a brand called Nemix that I found. I was a bit sketched out by the brand, but the chips themselves were Micron.
      - iguessthislldo 4y ago
        I was looking into getting a motherboard for a consumer CPU with ECC support several months ago. The fact that AMD and the motherboard manufactures don't explicitly say they support ECC is the reason I went with a Intel Supermicro motherboard that did explicitly say it supports ECC. I agree with the article though that it's crazy I have to resort to a workstation/server motherboard to be certain I'm getting ECC.
      - toast0 4y ago
        > AMD ryzen CPUs have ECC support Not all of them, afaik the am4 processors with graphics don't unless you have a pro branded version. > I never did figure out a way to verify that the ECC is working/if it is able to report errors (to the kernel? The easiest way is to induce an error. Some of the memtest tools try? But also, you can tweak memory voltage and timing to make things less reliable, and you should be able to get errors and error reporting as a result.
        rom-antics 4y ago
        I think the question is where are the errors reported? Would you see logs in dmesg for example? Or is there some smartctl-like tool that exposes the raw correction counters?
        ar-jan 4y ago
        You can use https://linux.die.net/man/1/edac-util https://linux.die.net/man/1/edac-util
        toast0 4y ago
        I'd expect them in dmesg or other kernel logs. That's where they show up on FreeBSD.
      - karmakaze 4y ago
        AMD used to not segment ECC for non-consuner lines. It wasn't very popular with consumers. So I can't really blame them for not supporting it well. One way that is a good balance is to make it work but not officially supported on consumer lines. Then it's up to mb vendors and buyers to take advantage if worthwhile to them. That's enough to not cut into non-consuner lines or add support costs to consumer pricing. One thing that could change this quick is if Apple used ECC. Others would follow to not seem inferior. I don't hear Apple users complain much at all about lack of ECC options.
  - deleted 4y ago
    [deleted]
- solomatov 4y ago
  I suspect that other methods of CPU segmentation, i.e. RAM channels, and PCIe lane are artificial in a similar way, i.e. cost of producing a die for a CPU is more or less the same, but pro CPUs have more of them.
phkahler 4y ago
Unfortunately DDR5 is going to complicate rather than fix the story.
- zdw 4y ago
  It's very strange that DDR5 mandates internal ECC within each physical package, but not on the longer and possibly more EMI sensitive connections between the memory chips and controller. I would have thought that the additional cost would be minimal (additional wiring on the logic board in some cases), but maybe this is just more artificial market segmentation?
  - ilyt 4y ago
    They have internal ECC coz that allows them to have higher yields, what could be considered faulty chip in DDR4 can be now sold in DDR5. So it is effectively cost-reducing measure for them. Exposing that to the user not only would cost extra pennies, but potentially have uses go "hey, this stick is shit, look at how many correctable errors it is producing, please replace it"
  - Dylan16807 4y ago
    And LPDDR5 gets to have link-level ECC. Thinking about it, when a DIMM doesn't have extra RAM chips for ECC, you could reuse the ECC pins to run a parity bit to each chip. It would cost nothing. And with DDR5 having 2x8 ECC pins, it would work with any chip layout: x16, x8, or x4.
- arp242 4y ago
  Why is that?
  - loeg 4y ago
    DDR5 will have some minimal ECC on the stick but critically does not mandate full runs to the CPU. Or in Wikipedia's words: > Unlike DDR4, all DDR5 chips have on-die ECC, where errors are detected and corrected before sending data to the CPU. This, however, is not the same as true ECC memory with an extra data correction chip on the memory module. DDR5's on-die error correction is to improve reliability and to allow denser RAM chips which lowers the per-chip defect rate. There still exist non-ECC and ECC DDR5 DIMM variants; the ECC variants have extra data lines to the CPU to send error-detection data, letting the CPU detect and correct errors that occurred in transit. So in some ways it is better than previous generations, but it gives vendors another excuse not to implement full-coverage ECC. That's my guess of why GP said it complicates things. https://en.wikipedia.org/wiki/DDR5_SDRAM https://en.wikipedia.org/wiki/DDR5_SDRAM
NelsonMinar 4y ago
Still wild to me we lost ECC RAM. It used to be standard in PCs. Does Apple hardware come with ECC RAM? If anyone could make it make sense as a business, it's them.
- pram 4y ago
  The Xeon based Macs had ECC of course. None of the ARM ones do (yet)
  - deleted 4y ago
    [deleted]
- MBCook 4y ago
  When was it standard? It’s been the high-end extra thing for as long as I can remember.
  - Felger 4y ago
    I can remember 486 Motherboard in Packard Bell (quite the entry brand...) systems frequently used 36 bits ECC FP SIMMs. Printers and plotters from this era used ECC modules most of the time. But by the end of the century, they were replaced by unbuffered, unregistered 16/32/64 bits modules. Every mid range server still use ECC. Entry HPE Servers use ECC UREG (unregistered, 9 chips) modules, while mid range and more use ECC REG modules (9 chip + interface controller onboard). Ironically, UREG module are more expensive than ECC REG. Also, most workstations used ECC modules. Less frequently since 4-5 years.
  - MisterTea 4y ago
    I know the Pentium Pro/2/3 chipsets and motherboards all(?) supported it. Unsure of the Pentium 1 as the 430TX on my Tyan Tomcat IV doesn't, and that is a dual processor board. 486 and earlier likely depended on the chipset as there were many. At work I have two working slot 1 PIII 800's each with 1GB ECC (4x 256MB DIMMS) on a regular Asus board (doing nothing but waiting to go home with me one day). The board reports the RAM is in fact ECC and that it is enabled.
dale_glass 4y ago
ECC RAM would actually be a boon to everyone, including gamers. ECC means not only that you know precisely when you've gone too far with overclocking, but potentially allows overclocking a bit further, relying on that some amount of trouble can now be tolerated. It also means you're not going to break your OS by playing with this stuff. Memory corruption carries a huge risk of disk corruption, which can mean things like corrupt data, random crashes or an unbootable system that persists even after reverting everything to defaults.
- p1necone 4y ago
  The sweet spot for overclocking ECC ram is still before it starts malfunctioning. If it's clocked higher but is correcting for errors it will still be slower.
  - ilyt 4y ago
    Entirely depends on error rate
    - deleted 4y ago
      [deleted]
- RealityVoid 4y ago
  I doubt that would actually be useful with overclocking. I don't know the arch of the modern PC well enough to say with 100% confidence, but on embedded arches, the RAM has the parity bits checked when they get placed on the bus. If the error happens on data retrieval(or was already present) , then the ECC saves you, but if it happen anywhere else... not really? I don't know if.. ALU's for example automatically include the parity bits in the computation.
  - p1mrx 4y ago
    You're talking about overclocking the CPU. ECC is more relevant when overclocking the RAM itself, which also affects gaming performance.
whitepoplar 4y ago
At least make it user-configurable! I'd trade off a bit of memory capacity for ECC protection in a heartbeat.
- deckard1 4y ago
  It is on Ryzen. Stick with AsRock or Asus motherboards, though.
thinking001001 4y ago
Digital privacy should be a human right. ECC RAM is just another iteration.
- theandrewbailey 4y ago
  These things are not comparable.
  - indolering 4y ago
    The miserable industry response to Row Hammer makes ECC RAM a necessity for anyone doing dangerous human rights work.
    - theandrewbailey 4y ago
      Virtually every piece of consumer hardware and software has security vulnerabilities, all the way down to CPUs and RAM. Perhaps computing, in general, is too hazardous for human rights work?
greenbit 4y ago
Commodity PCs back in the 80s and 90s didn't have error correction but they did have parity (iirc). Correction requires three extra bits per byte compared to parity only carrying one extra bit per byte. I recall around 1990 you could get your 30pin SIMs as 9-bit (parity) or 8-bit (no-parity), and virtually all of the PCs at the time wanted the 9 bit modules. Parity can't correct errors, but at least it can cause an exception when you read something that's had a bit flip.
- Felger 4y ago
  Yes I can indeed recall systems being shipped with 4x 9-bits sticks. And god the horrific price of thoses sticks...
- Dylan16807 4y ago
  On current memory, parity and correction get applied to the entire bus width. So with parity taking one bit, and correction taking log2(width) bits, it only requires 7-8 bits to apply both of them to an entire memory channel. (DDR4 has 64/72 bit channels, and DDR5 has 32/40 bit channels)
adhoc32 4y ago
Latest Intel desktop CPUs (i.e. i9-13900KF) supports ECC with the W680 chipset.
- fortran77 4y ago
  One of the main reasons I buy Xeon desktops is the ECC. With 128 GB of memory, and 1 bitflip/GB/year average error rate, it seems too risky to not use ECC for production work.
  - Retric 4y ago
    Real world numbers are closer to 1 bitflip/GB/hour than year because bit flips are highly correlated. “A large-scale study based on Google's very large number of servers was presented at the SIGMETRICS/Performance '09 conference.[6] The actual error rate found was several orders of magnitude higher than the previous small-scale or laboratory studies, with between 25,000 (2.5 × 10−11 error/bit·h) and 70,000 (7.0 × 10−11 error/bit·h, or 1 bit error per gigabyte of RAM per 1.8 hours) errors per billion device hours per megabit. More than 8% of DIMM memory modules were affected by errors per year.” https://en.wikipedia.org/wiki/ECC_memory https://en.wikipedia.org/wiki/ECC_memory A random stick of non ECC memory might be far above average or have several errors per minute, but you just don’t know.
    - gautamcgoel 4y ago
      That study is very old and is based on long-outdated DRAM tech. I suspect that DDR5 has much lower error rates.
      - Retric 4y ago
        I would welcome more recent data, but I doubt we are talking about a 4 orders of magnitude change to get to /year vs /hour error rates.
        gautamcgoel 4y ago
LanternLight83 4y ago
I'm with you, but it's worth noting that errant bit-flips are also the most convincing argument for vertically integrated file-systems like ZFS and BTRFS.
- diarrhea 4y ago
  I was under the impression running such file systems without ECC was kind of reckless and not a solution either? I recall ZFS being “useless” (not technically entirely bulletproof) without ECC.
  - wizeman 4y ago
    > I was under the impression running such file systems without ECC was kind of reckless and not a solution either? It's about as reckless as running any other filesystem without ECC [1]. In fact, you're much more likely to discover memory errors earlier if you use a checksumming filesystem than if you use a non-checksumming one. [1] https://news.ycombinator.com/item?id=34477035 https://news.ycombinator.com/item?id=34477035
    - diarrhea 4y ago
      Thanks! We already had that discussion, your link is your reply to another comment of mine :^) I was wrong twice huh!
laweijfmvo 4y ago
I've never had ECC memory to my knowledge. What have I been missing out on, in practice? Can we confidently say that x% of my past BSOD were the direct consequence?
- TacticalCoder 4y ago
  > I've never had ECC memory to my knowledge. What have I been missing out on, in practice? Can we confidently say that x% of my past BSOD were the direct consequence? I very highly doubt so. I remember BSOD and Guru meditations (on the Amiga) before that. Once I switched to Linux suddenly no more BSOD and very little kernel panics. I've had at times my desktop reach six months of uptime. I think bit-flips are a great excuse for unreliable software. BTW I'm not saying ECC is unnecessary: I'm saying it's unlikely the lack of ECC was the reason for most of the BSOD you saw throughout the ages.
  - indolering 4y ago
    The bitflip would also have to cause an error in logic that would result in a system lockup. That's unlikely, given that most information flowing through ram is probably data, not logic.
- derkades 4y ago
  You may have been lucky enough to never experienced a bad module. But when it does happen, it can have major consequences (losing data, having to reinstall an OS). All while being totally preventable.
  - kiririn 4y ago
    Not enough people realise that the driving reason for ECC in home systems is this, not random once-in-a-blue-moon bit flips I’ll always buy ECC after multiple data loss events from ram suddenly going bad, worth every penny to have ECC ring alarm bells rather than put 2+2 together after random BSODs and silent corruption
deleted 4y ago
[deleted]
chatmasta 4y ago
> It was the worst-case scenario for RAM failure: bit flip errors that get written back to the disk. I discovered that several video files that I had been editing had corrupted bits, and were no longer usable. What a horrible failure mode! Kudos to the OP for even thinking to investigate this. The mentioned memtest86 tool sounds quite useful - so useful, in fact, that it seems it should be part of the operating system. If my OS is perfectly willing to write corrupted memory back to disk, and it's designed to run on thousands of different OEM hardware configurations that may or may not have ECC RAM, then I would expect it to proactively monitor for faulty RAM and bitflips. Is there a good reason why this isn't the default behavior of operating systems (maybe it is - I use Mac and don't know much about Windows)? It seems like a trivial diagnostic tool that could prevent a lot of headache. But perhaps the problem is there's no reliable test without some number of false positives, so popping a warning on the screen that your RAM appears unhealthy seems like a good way to confuse the average user. But then again, so does silently corrupting a video file.
- bsder 4y ago
  > Is there a good reason why this isn't the default behavior of operating systems Because any equivalent of memtestx86 will flag a LOT of hardware for the buggy piles of shit that they really are. If Windows started flagging people's crappy cheap hardware, Microsoft would get a lot of grief and have to spend a chunk of money on customer support and PR.
  - formerly_proven 4y ago
    I dunno, I've had overclocked memory where memtest86 wouldn't report errors in multiple passes, yet TestMem5 (weird win32 tool from some .ru site) tells you it's bad in about ten seconds. memtest86 doesn't seem to be that stressful for memory.
- mike_hock 4y ago
  memtest86 needs to run alone so it can test the entire physical RAM, and the actual, meaningful tests that actually test RAM and not L1 take forever to run. What would you have the OS do?
deckard1 4y ago
> a multitude of bit flip errors ECC can only report >1 bit flip per word. It can't correct it. If your memory is going bad, the chance of multiple bit failure is going to be much higher than the stray bit flip from other causes (mind you, I have 32GB ECC running right now and have seen zero bit flips over 1.5 years). I'm not arguing against ECC. I have it and run it myself. But this blog post is an argument for ZFS and regular backups. Failures can occur at many other places than RAM. I had a drive that had corruption and I thought went bad. Turns out, bad SATA cable. A week later I upgraded from cable internet to 1Gbit fiber. My speeds were not at all what I expected. Turns out... bad ethernet cable. Lightning can strike twice I guess. > By the way, you’d better believe that your disk(s) have all kinds of error correction schemes built into them, which work automatically and transparently. Well... https://twitter.com/xenadu02/status/1495693475584557056 https://twitter.com/xenadu02/status/1495693475584557056 https://www.tomshardware.com/news/sk-hynix-sabrent-rocket-ssds-data-loss https://www.tomshardware.com/news/sk-hynix-sabrent-rocket-ss... https://twitter.com/bsdphk/status/1495899958960136202 https://twitter.com/bsdphk/status/1495899958960136202
- Dylan16807 4y ago
  > If your memory is going bad, the chance of multiple bit failure is going to be much higher When a chip starts going bad, the thousands of error reports will more than make up for the errors that slip through. Just make sure reporting is working. > Well... Well what? Those drives were lying about finishing writes, which is barely related to losing data that was written, and especially data that was written more than a few seconds ago. And if drives didn't have ECC, data loss would be a million times more common.
  - indolering 4y ago
    It also allows for runtime detection of bad actors attempting to flip memory.
- Hizonner 4y ago
  > If your memory is going bad, the chance of multiple bit failure is going to be much higher than the stray bit flip from other causes It's completely possible for a single DRAM cell to go bad permanently. > But this blog post is an argument for ZFS ZFS won't save you if what you tell it to put into a file is wrong. And I think that corrupting ZFS' own in-RAM data structures voids your ZFS warranty. > and regular backups. Of course you should have regular backups. But do you really feel like spending hours restoring from a backup (and then maybe weeks finding things you didn't restore)? Do you feel like trying to guess how far back you have to go to get to a good backup of a corrupted file? Do you want to lose whatever new work got corrupted between the time your memory failed and the time you finally noticed it? OP's machine was corrupting data for weeks. You can't just roll all your work back to several weeks ago. Most of us can't, anyway. You need backups and ECC. > I had a drive that had corruption and I thought went bad. Turns out, bad SATA cable. If a bad cable did that, something wasn't doing SATA CRC checking the way it was supposed to (or wasn't reacting to detected CRC errors the way it should have). Similar in spirit to ECC checking for memory. > My speeds were not at all what I expected. Turns out... bad ethernet cable. It probably got slow because of retransmissions. If there hadn't been CRCs on both the Ethernet layer and higher layers, it could possibly have just silently corrupted your data. You need to cover as much of a computer system as you can with error detection and correction, or you always lose. Even one big failure in a lifetime can pay for a lot of ECC RAM.
marius_k 4y ago
If there would be such right, it should be expresed in terms of error probabilities rather than a specific features like ECC.
phendrenad2 4y ago
This is just unscientific nonsense. > Memory manufacturers assure us that desktop RAM is so reliable that it doesn’t need ECC, that the probability of bit flip events is so low that it’s not worth the extra “cost” of ECC Yea, and they are right.
- Night_Thastus 4y ago
  I'm not very familiar with hardware architecture. Could you explain why you disagree?
  - phendrenad2 4y ago
    ECC requires extra circuitry, and that results in more cost, also more testing cost, and also more failed chips (due to more complexity). Most people have very little need for ECC. The author didn't even know that they wanted ECC until they were unlucky enough to get a stick of RAM that failed, and failed in such a way that the OS booted, but a file silently corrupted (not that common, because if a chip fails it usually doesn't silently fail like that). The author is basically asking for ECC for free.
    - Night_Thastus 4y ago
      How much more is that cost, and how does it scale? Are we talking 5%, 10%, 30%?
      - phendrenad2 4y ago
        Well, ECC memory is more expensive than non-ECC memory by about 10-20% (according to a quick google search). Part of that may be markup due to people who want ECC being willing to pay for a highly reputable brand sold by a highly-reputable seller (they won't be buying ECC in bulk from amazon marketplace).
bsder 4y ago
My big complaint is actually digging through the mess of MB/RAM/BIOS documentation to actually verify that I have the right combination of processor/motherboard/memory for ECC AND TEST IT'S WORKING. AMD seems to support ECC on some consumer-level chipsets, but it's a nightmare to sort it all out and verify that all the bits are actually functioning properly.
- indolering 4y ago
  Were you able to find a consumer motherboard that supports fault injection (or whatever it's called)? I was able to verify that the motherboard reported ECC was working and performed a 24/hour burn in, but I have been unable to perform a functional test showing that ECC had corrected an error.
  - bsder 4y ago
    Sadly, no. Verifying that ECC is actually functioning is ridiculously hard on consumer grade motherboards.
mixedbit 4y ago
Linus recently also had troubles with his development machine caused by the lack of ECC: 'I absolutely detest the crazy industry politics and bad vendors that have made ECC memory so "special".' https://lkml.iu.edu/hypermail/linux/kernel/2210.1/00691.html https://lkml.iu.edu/hypermail/linux/kernel/2210.1/00691.html
intrasight 4y ago
I believe that is is a right. Every machine I've built in the last ~12 years had had ECC. Nobody is trying to take that ability away from me. I treat my machine as a "workstation". I run several VMs. I keep my machines for a few years. I clean the filters periodically. Do game machines need ECC? I'd say no. They are optimized for cost-performance. The worse that can happen with a memory error is a lost game.
- indolering 4y ago
  You have the good fortune of being able to afford lots of expensive hardware. Mandating ECC is basically like mandating seat-belts in all cars: it makes safety cheaper for everyone through the economies of scale. Gamers and libertarians could theoretically disable it if they so chose and the total cost to them would only be a few pennies for the wasted motherboard circuitry.
  - intrasight 4y ago
    But he said "right" not "mandate". My point was only that I currently do have an implicit right. ECC memory isn't that much more expensive. I was fortunate to build my current machine a few months before memory prices doubled. Previous generation Xeon CPUs aren't very expensive. Same with motherboards. And another option is to just buy a used server. These are super cheap now as so many companies are moving to cloud computing.
jsz0 4y ago
Personally used about an even split of ECC vs. non-ECC RAM over the last ~30 years and can't say I've ever encountered any problems with non-ECC RAM. If it happened it was no significance for me to notice. Professionally yeah I've run across a server here and there report bit errors but it was a four leaf clover the of thing. Not unheard of but not common. The more important thing in my experience is using high quality hardware with stable well tested drivers. That is the first practical precaution to consider. If you've got a box full of the cheapest parts you can find on Amazon you got bigger problems to consider before worrying about ECC.
- Dalewyn 4y ago
  This kind of comment is the most insightful kind concerning ECC RAM. If you're relying on your computer to be 100% error-free (eg: doing professional work), paying some extra for ECC RAM support isn't even a drop in the bucket. You either do or don't, money isn't an object. If you're Joe Average browsing Hacker News or playing some games at home, who cares if you win the bit flip lottery? Would it be nice for ECC RAM to be more mainstream? Sure. But the fact it is not isn't breaking anyones' backs either.
  - rapjr9 4y ago
    I wonder if lack of ECC is a form of planned obsolescence. It makes the computer cost less, and it insures the computer starts failing after a while as the OS and files get corrupted, making the owner want a new computer. More profit due to both of those. Kind of like using rubber for seals in a car rather than silicone, even though the cost difference is tiny, so owners eventually have to buy a new car.
- adhoc32 4y ago
  Silent data corruption is silent. ECC should be mandatory just for the error reporting. The system should inform the user that a DIMM has gone bad and needs replacement.
giardia 4y ago
Oh god this gives me flashbacks. I never knew how bad corruption was on home computers until I started using git annex on a multi-terabyte file collection. No matter which computer or disk I used, it would inevitably happen. Then I started to wonder how much corruption had crept in before I was using git annex and never noticed it. I've basically given up on maintaining large digital media collections for long term purposes. It puts my OCD in overdrive.
- ls65536 4y ago
  How about storing the data on a filesystem that performs data checksumming, also configured in a RAID-1-like mirroring profile to enable any corruption to also be corrected? Filesystems like ZFS or BTRFS could be a good choice here.
  - diarrhea 4y ago
    One of the prime prerequisites of e.g. zfs actually is using ECC RAM. I’m running btrfs with integrity checks, in RAID 1 so it can automatically heal. Yet it’s non-ECC and therefore still has this gaping Achilles heel.
    - wizeman 4y ago
      > One of the prime prerequisites of e.g. zfs actually is using ECC RAM. ECC RAM is not a prerequisite for using ZFS. Matt Ahrens, co-creator of ZFS and still one of the main developers, said this [1]: "There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. Actually, ZFS can mitigate this risk to some degree if you enable the unsupported ZFS_DEBUG_MODIFY flag (zfs_flags=0x10). This will checksum the data while at rest in memory, and verify it before writing to disk, thus reducing the window of vulnerability from a memory error. I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS." [1] https://arstechnica.com/civis/threads/ars-walkthrough-using-the-zfs-next-gen-filesystem-on-linux.1235679/page-4#post-26303271 https://arstechnica.com/civis/threads/ars-walkthrough-using-...
deleted 4y ago
[deleted]
bpye 4y ago
I wish Intel and AMD would support in-band ECC on their consumer platforms. That way you can use the same DIMMs and you make the choice - either sacrifice a few percent of memory for memory correction, or don't. Intel _do_ support this on _some_ workstation products, but it's very uncommon and naturally those platforms are expensive.
sitkack 4y ago
Read the paper(s) below. Absolutely frightening, we all basically swimming in quicksand. ECC should be mandatory! Not using it is like huffing leaded gas. Silent Data Corruptions at Scale https://arxiv.org/abs/2102.11245 https://arxiv.org/abs/2102.11245 Harish Dixit has two other papers available https://arxiv.org/search/cs?searchtype=author&query=Dixit%2C+H+D https://arxiv.org/search/cs?searchtype=author&query=Dixit%2C... Revisiting Memory Errors in Large-Scale Production Data Centers: Analysis and Modeling of New Trends from the Field https://users.ece.cmu.edu/~omutlu/pub/memory-errors-at-facebook_dsn15.pdf https://users.ece.cmu.edu/~omutlu/pub/memory-errors-at-faceb... Justin Meza has a similar body of research. https://www.semanticscholar.org/author/Justin-Meza/144606145 https://www.semanticscholar.org/author/Justin-Meza/144606145
- sitkack 4y ago
  Justin Meza's thesis https://kilthub.cmu.edu/articles/thesis/Large_Scale_Studies_of_Memory_Storage_and_Network_Failures_in_a_Modern_Data_Center/7571684/1 https://kilthub.cmu.edu/articles/thesis/Large_Scale_Studies_...
SergeAx 4y ago
I don't know. I am building my own computers for about 30 years, starting with Intel 486, and using them heavily. Couple of times I bought ECC memory, but most of the time - not. For the last 4 years I have 64G of non-ECC RAM. Maybe I had some problem with non-ECC memory during all those years, like sudden BSODs, freezes or some corrupted data, but never anything significant. So I honestly don't see a reason for drama here.
olliecornelia 4y ago
Cosmic rays make things fun.
temptemptemp111 4y ago
[dead]

211 points by zdw 4y ago

deleted 4y ago

[deleted]

ECC has been used as an artificial market segmentation mechanism for a long time and it needs to come to an end. RAM just like SSDs and HDDs ought to have some amount of self protection against basic errors, all places where data is stored even for short periods needs this.

deadbunny 4y ago
> ECC has been used by Intel as an artificial market segmentation mechanism FTFY
- leguminous 4y ago
  What's really annoying is that the current generation of Intel CPUs support ECC, they just don't implement it in consumer chipsets. You can get working ECC with a W680 motherboard, but those are very expensive and availability is slim.
- nimish 4y ago
  And AMD, and Nvidia.
  - hamandcheese 4y ago
    AMD ryzen CPUs have ECC support, but it varies by motherboard. My home NAS runs a Ryzen in an ASrock Rack motherboard with ECC ram. I never did figure out a way to verify that the ECC is working/if it is able to report errors (to the kernel?). It was also a bit hard finding the right ram, but there’s a brand called Nemix that I found. I was a bit sketched out by the brand, but the chips themselves were Micron.
    - iguessthislldo 4y ago
      I was looking into getting a motherboard for a consumer CPU with ECC support several months ago. The fact that AMD and the motherboard manufactures don't explicitly say they support ECC is the reason I went with a Intel Supermicro motherboard that did explicitly say it supports ECC. I agree with the article though that it's crazy I have to resort to a workstation/server motherboard to be certain I'm getting ECC.
    - toast0 4y ago
      > AMD ryzen CPUs have ECC support Not all of them, afaik the am4 processors with graphics don't unless you have a pro branded version. > I never did figure out a way to verify that the ECC is working/if it is able to report errors (to the kernel? The easiest way is to induce an error. Some of the memtest tools try? But also, you can tweak memory voltage and timing to make things less reliable, and you should be able to get errors and error reporting as a result.
      - rom-antics 4y ago
        I think the question is where are the errors reported? Would you see logs in dmesg for example? Or is there some smartctl-like tool that exposes the raw correction counters?
        ar-jan 4y ago
        You can use https://linux.die.net/man/1/edac-util https://linux.die.net/man/1/edac-util
        toast0 4y ago
        I'd expect them in dmesg or other kernel logs. That's where they show up on FreeBSD.
    - karmakaze 4y ago
      AMD used to not segment ECC for non-consuner lines. It wasn't very popular with consumers. So I can't really blame them for not supporting it well. One way that is a good balance is to make it work but not officially supported on consumer lines. Then it's up to mb vendors and buyers to take advantage if worthwhile to them. That's enough to not cut into non-consuner lines or add support costs to consumer pricing. One thing that could change this quick is if Apple used ECC. Others would follow to not seem inferior. I don't hear Apple users complain much at all about lack of ECC options.
- deleted 4y ago
  [deleted]
solomatov 4y ago
I suspect that other methods of CPU segmentation, i.e. RAM channels, and PCIe lane are artificial in a similar way, i.e. cost of producing a die for a CPU is more or less the same, but pro CPUs have more of them.

phkahler 4y ago

Unfortunately DDR5 is going to complicate rather than fix the story.

zdw 4y ago
It's very strange that DDR5 mandates internal ECC within each physical package, but not on the longer and possibly more EMI sensitive connections between the memory chips and controller. I would have thought that the additional cost would be minimal (additional wiring on the logic board in some cases), but maybe this is just more artificial market segmentation?
- ilyt 4y ago
  They have internal ECC coz that allows them to have higher yields, what could be considered faulty chip in DDR4 can be now sold in DDR5. So it is effectively cost-reducing measure for them. Exposing that to the user not only would cost extra pennies, but potentially have uses go "hey, this stick is shit, look at how many correctable errors it is producing, please replace it"
- Dylan16807 4y ago
  And LPDDR5 gets to have link-level ECC. Thinking about it, when a DIMM doesn't have extra RAM chips for ECC, you could reuse the ECC pins to run a parity bit to each chip. It would cost nothing. And with DDR5 having 2x8 ECC pins, it would work with any chip layout: x16, x8, or x4.
arp242 4y ago
Why is that?
- loeg 4y ago
  DDR5 will have some minimal ECC on the stick but critically does not mandate full runs to the CPU. Or in Wikipedia's words: > Unlike DDR4, all DDR5 chips have on-die ECC, where errors are detected and corrected before sending data to the CPU. This, however, is not the same as true ECC memory with an extra data correction chip on the memory module. DDR5's on-die error correction is to improve reliability and to allow denser RAM chips which lowers the per-chip defect rate. There still exist non-ECC and ECC DDR5 DIMM variants; the ECC variants have extra data lines to the CPU to send error-detection data, letting the CPU detect and correct errors that occurred in transit. So in some ways it is better than previous generations, but it gives vendors another excuse not to implement full-coverage ECC. That's my guess of why GP said it complicates things. https://en.wikipedia.org/wiki/DDR5_SDRAM https://en.wikipedia.org/wiki/DDR5_SDRAM

NelsonMinar 4y ago

Still wild to me we lost ECC RAM. It used to be standard in PCs. Does Apple hardware come with ECC RAM? If anyone could make it make sense as a business, it's them.

pram 4y ago
The Xeon based Macs had ECC of course. None of the ARM ones do (yet)
- deleted 4y ago
  [deleted]
MBCook 4y ago
When was it standard? It’s been the high-end extra thing for as long as I can remember.
- Felger 4y ago
  I can remember 486 Motherboard in Packard Bell (quite the entry brand...) systems frequently used 36 bits ECC FP SIMMs. Printers and plotters from this era used ECC modules most of the time. But by the end of the century, they were replaced by unbuffered, unregistered 16/32/64 bits modules. Every mid range server still use ECC. Entry HPE Servers use ECC UREG (unregistered, 9 chips) modules, while mid range and more use ECC REG modules (9 chip + interface controller onboard). Ironically, UREG module are more expensive than ECC REG. Also, most workstations used ECC modules. Less frequently since 4-5 years.
- MisterTea 4y ago
  I know the Pentium Pro/2/3 chipsets and motherboards all(?) supported it. Unsure of the Pentium 1 as the 430TX on my Tyan Tomcat IV doesn't, and that is a dual processor board. 486 and earlier likely depended on the chipset as there were many. At work I have two working slot 1 PIII 800's each with 1GB ECC (4x 256MB DIMMS) on a regular Asus board (doing nothing but waiting to go home with me one day). The board reports the RAM is in fact ECC and that it is enabled.

dale_glass 4y ago

ECC RAM would actually be a boon to everyone, including gamers. ECC means not only that you know precisely when you've gone too far with overclocking, but potentially allows overclocking a bit further, relying on that some amount of trouble can now be tolerated. It also means you're not going to break your OS by playing with this stuff. Memory corruption carries a huge risk of disk corruption, which can mean things like corrupt data, random crashes or an unbootable system that persists even after reverting everything to defaults.

p1necone 4y ago
The sweet spot for overclocking ECC ram is still before it starts malfunctioning. If it's clocked higher but is correcting for errors it will still be slower.
- ilyt 4y ago
  Entirely depends on error rate
  - deleted 4y ago
    [deleted]
RealityVoid 4y ago
I doubt that would actually be useful with overclocking. I don't know the arch of the modern PC well enough to say with 100% confidence, but on embedded arches, the RAM has the parity bits checked when they get placed on the bus. If the error happens on data retrieval(or was already present) , then the ECC saves you, but if it happen anywhere else... not really? I don't know if.. ALU's for example automatically include the parity bits in the computation.
- p1mrx 4y ago
  You're talking about overclocking the CPU. ECC is more relevant when overclocking the RAM itself, which also affects gaming performance.

whitepoplar 4y ago

At least make it user-configurable! I'd trade off a bit of memory capacity for ECC protection in a heartbeat.

deckard1 4y ago
It is on Ryzen. Stick with AsRock or Asus motherboards, though.

thinking001001 4y ago

Digital privacy should be a human right. ECC RAM is just another iteration.

theandrewbailey 4y ago
These things are not comparable.
- indolering 4y ago
  The miserable industry response to Row Hammer makes ECC RAM a necessity for anyone doing dangerous human rights work.
  - theandrewbailey 4y ago
    Virtually every piece of consumer hardware and software has security vulnerabilities, all the way down to CPUs and RAM. Perhaps computing, in general, is too hazardous for human rights work?

greenbit 4y ago

Commodity PCs back in the 80s and 90s didn't have error correction but they did have parity (iirc). Correction requires three extra bits per byte compared to parity only carrying one extra bit per byte. I recall around 1990 you could get your 30pin SIMs as 9-bit (parity) or 8-bit (no-parity), and virtually all of the PCs at the time wanted the 9 bit modules. Parity can't correct errors, but at least it can cause an exception when you read something that's had a bit flip.

Felger 4y ago
Yes I can indeed recall systems being shipped with 4x 9-bits sticks. And god the horrific price of thoses sticks...
Dylan16807 4y ago
On current memory, parity and correction get applied to the entire bus width. So with parity taking one bit, and correction taking log2(width) bits, it only requires 7-8 bits to apply both of them to an entire memory channel. (DDR4 has 64/72 bit channels, and DDR5 has 32/40 bit channels)

adhoc32 4y ago

Latest Intel desktop CPUs (i.e. i9-13900KF) supports ECC with the W680 chipset.

fortran77 4y ago
One of the main reasons I buy Xeon desktops is the ECC. With 128 GB of memory, and 1 bitflip/GB/year average error rate, it seems too risky to not use ECC for production work.
- Retric 4y ago
  Real world numbers are closer to 1 bitflip/GB/hour than year because bit flips are highly correlated. “A large-scale study based on Google's very large number of servers was presented at the SIGMETRICS/Performance '09 conference.[6] The actual error rate found was several orders of magnitude higher than the previous small-scale or laboratory studies, with between 25,000 (2.5 × 10−11 error/bit·h) and 70,000 (7.0 × 10−11 error/bit·h, or 1 bit error per gigabyte of RAM per 1.8 hours) errors per billion device hours per megabit. More than 8% of DIMM memory modules were affected by errors per year.” https://en.wikipedia.org/wiki/ECC_memory https://en.wikipedia.org/wiki/ECC_memory A random stick of non ECC memory might be far above average or have several errors per minute, but you just don’t know.
  - gautamcgoel 4y ago
    That study is very old and is based on long-outdated DRAM tech. I suspect that DDR5 has much lower error rates.
    - Retric 4y ago
      I would welcome more recent data, but I doubt we are talking about a 4 orders of magnitude change to get to /year vs /hour error rates.
      - gautamcgoel 4y ago

LanternLight83 4y ago

I'm with you, but it's worth noting that errant bit-flips are also the most convincing argument for vertically integrated file-systems like ZFS and BTRFS.

diarrhea 4y ago
I was under the impression running such file systems without ECC was kind of reckless and not a solution either? I recall ZFS being “useless” (not technically entirely bulletproof) without ECC.
- wizeman 4y ago
  > I was under the impression running such file systems without ECC was kind of reckless and not a solution either? It's about as reckless as running any other filesystem without ECC [1]. In fact, you're much more likely to discover memory errors earlier if you use a checksumming filesystem than if you use a non-checksumming one. [1] https://news.ycombinator.com/item?id=34477035 https://news.ycombinator.com/item?id=34477035
  - diarrhea 4y ago
    Thanks! We already had that discussion, your link is your reply to another comment of mine :^) I was wrong twice huh!

laweijfmvo 4y ago

I've never had ECC memory to my knowledge. What have I been missing out on, in practice? Can we confidently say that x% of my past BSOD were the direct consequence?

TacticalCoder 4y ago
> I've never had ECC memory to my knowledge. What have I been missing out on, in practice? Can we confidently say that x% of my past BSOD were the direct consequence? I very highly doubt so. I remember BSOD and Guru meditations (on the Amiga) before that. Once I switched to Linux suddenly no more BSOD and very little kernel panics. I've had at times my desktop reach six months of uptime. I think bit-flips are a great excuse for unreliable software. BTW I'm not saying ECC is unnecessary: I'm saying it's unlikely the lack of ECC was the reason for most of the BSOD you saw throughout the ages.
- indolering 4y ago
  The bitflip would also have to cause an error in logic that would result in a system lockup. That's unlikely, given that most information flowing through ram is probably data, not logic.
derkades 4y ago
You may have been lucky enough to never experienced a bad module. But when it does happen, it can have major consequences (losing data, having to reinstall an OS). All while being totally preventable.
- kiririn 4y ago
  Not enough people realise that the driving reason for ECC in home systems is this, not random once-in-a-blue-moon bit flips I’ll always buy ECC after multiple data loss events from ram suddenly going bad, worth every penny to have ECC ring alarm bells rather than put 2+2 together after random BSODs and silent corruption

deleted 4y ago

[deleted]

chatmasta 4y ago

> It was the worst-case scenario for RAM failure: bit flip errors that get written back to the disk. I discovered that several video files that I had been editing had corrupted bits, and were no longer usable. What a horrible failure mode! Kudos to the OP for even thinking to investigate this. The mentioned memtest86 tool sounds quite useful - so useful, in fact, that it seems it should be part of the operating system. If my OS is perfectly willing to write corrupted memory back to disk, and it's designed to run on thousands of different OEM hardware configurations that may or may not have ECC RAM, then I would expect it to proactively monitor for faulty RAM and bitflips. Is there a good reason why this isn't the default behavior of operating systems (maybe it is - I use Mac and don't know much about Windows)? It seems like a trivial diagnostic tool that could prevent a lot of headache. But perhaps the problem is there's no reliable test without some number of false positives, so popping a warning on the screen that your RAM appears unhealthy seems like a good way to confuse the average user. But then again, so does silently corrupting a video file.

bsder 4y ago
> Is there a good reason why this isn't the default behavior of operating systems Because any equivalent of memtestx86 will flag a LOT of hardware for the buggy piles of shit that they really are. If Windows started flagging people's crappy cheap hardware, Microsoft would get a lot of grief and have to spend a chunk of money on customer support and PR.
- formerly_proven 4y ago
  I dunno, I've had overclocked memory where memtest86 wouldn't report errors in multiple passes, yet TestMem5 (weird win32 tool from some .ru site) tells you it's bad in about ten seconds. memtest86 doesn't seem to be that stressful for memory.
mike_hock 4y ago
memtest86 needs to run alone so it can test the entire physical RAM, and the actual, meaningful tests that actually test RAM and not L1 take forever to run. What would you have the OS do?

deckard1 4y ago

> a multitude of bit flip errors ECC can only report >1 bit flip per word. It can't correct it. If your memory is going bad, the chance of multiple bit failure is going to be much higher than the stray bit flip from other causes (mind you, I have 32GB ECC running right now and have seen zero bit flips over 1.5 years). I'm not arguing against ECC. I have it and run it myself. But this blog post is an argument for ZFS and regular backups. Failures can occur at many other places than RAM. I had a drive that had corruption and I thought went bad. Turns out, bad SATA cable. A week later I upgraded from cable internet to 1Gbit fiber. My speeds were not at all what I expected. Turns out... bad ethernet cable. Lightning can strike twice I guess. > By the way, you’d better believe that your disk(s) have all kinds of error correction schemes built into them, which work automatically and transparently. Well... https://twitter.com/xenadu02/status/1495693475584557056 https://twitter.com/xenadu02/status/1495693475584557056 https://www.tomshardware.com/news/sk-hynix-sabrent-rocket-ssds-data-loss https://www.tomshardware.com/news/sk-hynix-sabrent-rocket-ss... https://twitter.com/bsdphk/status/1495899958960136202 https://twitter.com/bsdphk/status/1495899958960136202

Dylan16807 4y ago
> If your memory is going bad, the chance of multiple bit failure is going to be much higher When a chip starts going bad, the thousands of error reports will more than make up for the errors that slip through. Just make sure reporting is working. > Well... Well what? Those drives were lying about finishing writes, which is barely related to losing data that was written, and especially data that was written more than a few seconds ago. And if drives didn't have ECC, data loss would be a million times more common.
- indolering 4y ago
  It also allows for runtime detection of bad actors attempting to flip memory.
Hizonner 4y ago
> If your memory is going bad, the chance of multiple bit failure is going to be much higher than the stray bit flip from other causes It's completely possible for a single DRAM cell to go bad permanently. > But this blog post is an argument for ZFS ZFS won't save you if what you tell it to put into a file is wrong. And I think that corrupting ZFS' own in-RAM data structures voids your ZFS warranty. > and regular backups. Of course you should have regular backups. But do you really feel like spending hours restoring from a backup (and then maybe weeks finding things you didn't restore)? Do you feel like trying to guess how far back you have to go to get to a good backup of a corrupted file? Do you want to lose whatever new work got corrupted between the time your memory failed and the time you finally noticed it? OP's machine was corrupting data for weeks. You can't just roll all your work back to several weeks ago. Most of us can't, anyway. You need backups and ECC. > I had a drive that had corruption and I thought went bad. Turns out, bad SATA cable. If a bad cable did that, something wasn't doing SATA CRC checking the way it was supposed to (or wasn't reacting to detected CRC errors the way it should have). Similar in spirit to ECC checking for memory. > My speeds were not at all what I expected. Turns out... bad ethernet cable. It probably got slow because of retransmissions. If there hadn't been CRCs on both the Ethernet layer and higher layers, it could possibly have just silently corrupted your data. You need to cover as much of a computer system as you can with error detection and correction, or you always lose. Even one big failure in a lifetime can pay for a lot of ECC RAM.

marius_k 4y ago

If there would be such right, it should be expresed in terms of error probabilities rather than a specific features like ECC.

phendrenad2 4y ago

This is just unscientific nonsense. > Memory manufacturers assure us that desktop RAM is so reliable that it doesn’t need ECC, that the probability of bit flip events is so low that it’s not worth the extra “cost” of ECC Yea, and they are right.

Night_Thastus 4y ago
I'm not very familiar with hardware architecture. Could you explain why you disagree?
- phendrenad2 4y ago
  ECC requires extra circuitry, and that results in more cost, also more testing cost, and also more failed chips (due to more complexity). Most people have very little need for ECC. The author didn't even know that they wanted ECC until they were unlucky enough to get a stick of RAM that failed, and failed in such a way that the OS booted, but a file silently corrupted (not that common, because if a chip fails it usually doesn't silently fail like that). The author is basically asking for ECC for free.
  - Night_Thastus 4y ago
    How much more is that cost, and how does it scale? Are we talking 5%, 10%, 30%?
    - phendrenad2 4y ago
      Well, ECC memory is more expensive than non-ECC memory by about 10-20% (according to a quick google search). Part of that may be markup due to people who want ECC being willing to pay for a highly reputable brand sold by a highly-reputable seller (they won't be buying ECC in bulk from amazon marketplace).

bsder 4y ago

My big complaint is actually digging through the mess of MB/RAM/BIOS documentation to actually verify that I have the right combination of processor/motherboard/memory for ECC AND TEST IT'S WORKING. AMD seems to support ECC on some consumer-level chipsets, but it's a nightmare to sort it all out and verify that all the bits are actually functioning properly.

indolering 4y ago
Were you able to find a consumer motherboard that supports fault injection (or whatever it's called)? I was able to verify that the motherboard reported ECC was working and performed a 24/hour burn in, but I have been unable to perform a functional test showing that ECC had corrected an error.
- bsder 4y ago
  Sadly, no. Verifying that ECC is actually functioning is ridiculously hard on consumer grade motherboards.

mixedbit 4y ago

Linus recently also had troubles with his development machine caused by the lack of ECC: 'I absolutely detest the crazy industry politics and bad vendors that have made ECC memory so "special".' https://lkml.iu.edu/hypermail/linux/kernel/2210.1/00691.html https://lkml.iu.edu/hypermail/linux/kernel/2210.1/00691.html

intrasight 4y ago

I believe that is is a right. Every machine I've built in the last ~12 years had had ECC. Nobody is trying to take that ability away from me. I treat my machine as a "workstation". I run several VMs. I keep my machines for a few years. I clean the filters periodically. Do game machines need ECC? I'd say no. They are optimized for cost-performance. The worse that can happen with a memory error is a lost game.

indolering 4y ago
You have the good fortune of being able to afford lots of expensive hardware. Mandating ECC is basically like mandating seat-belts in all cars: it makes safety cheaper for everyone through the economies of scale. Gamers and libertarians could theoretically disable it if they so chose and the total cost to them would only be a few pennies for the wasted motherboard circuitry.
- intrasight 4y ago
  But he said "right" not "mandate". My point was only that I currently do have an implicit right. ECC memory isn't that much more expensive. I was fortunate to build my current machine a few months before memory prices doubled. Previous generation Xeon CPUs aren't very expensive. Same with motherboards. And another option is to just buy a used server. These are super cheap now as so many companies are moving to cloud computing.

jsz0 4y ago

Personally used about an even split of ECC vs. non-ECC RAM over the last ~30 years and can't say I've ever encountered any problems with non-ECC RAM. If it happened it was no significance for me to notice. Professionally yeah I've run across a server here and there report bit errors but it was a four leaf clover the of thing. Not unheard of but not common. The more important thing in my experience is using high quality hardware with stable well tested drivers. That is the first practical precaution to consider. If you've got a box full of the cheapest parts you can find on Amazon you got bigger problems to consider before worrying about ECC.

Dalewyn 4y ago
This kind of comment is the most insightful kind concerning ECC RAM. If you're relying on your computer to be 100% error-free (eg: doing professional work), paying some extra for ECC RAM support isn't even a drop in the bucket. You either do or don't, money isn't an object. If you're Joe Average browsing Hacker News or playing some games at home, who cares if you win the bit flip lottery? Would it be nice for ECC RAM to be more mainstream? Sure. But the fact it is not isn't breaking anyones' backs either.
- rapjr9 4y ago
  I wonder if lack of ECC is a form of planned obsolescence. It makes the computer cost less, and it insures the computer starts failing after a while as the OS and files get corrupted, making the owner want a new computer. More profit due to both of those. Kind of like using rubber for seals in a car rather than silicone, even though the cost difference is tiny, so owners eventually have to buy a new car.
adhoc32 4y ago
Silent data corruption is silent. ECC should be mandatory just for the error reporting. The system should inform the user that a DIMM has gone bad and needs replacement.

giardia 4y ago

Oh god this gives me flashbacks. I never knew how bad corruption was on home computers until I started using git annex on a multi-terabyte file collection. No matter which computer or disk I used, it would inevitably happen. Then I started to wonder how much corruption had crept in before I was using git annex and never noticed it. I've basically given up on maintaining large digital media collections for long term purposes. It puts my OCD in overdrive.

ls65536 4y ago
How about storing the data on a filesystem that performs data checksumming, also configured in a RAID-1-like mirroring profile to enable any corruption to also be corrected? Filesystems like ZFS or BTRFS could be a good choice here.
- diarrhea 4y ago
  One of the prime prerequisites of e.g. zfs actually is using ECC RAM. I’m running btrfs with integrity checks, in RAID 1 so it can automatically heal. Yet it’s non-ECC and therefore still has this gaping Achilles heel.
  - wizeman 4y ago
    > One of the prime prerequisites of e.g. zfs actually is using ECC RAM. ECC RAM is not a prerequisite for using ZFS. Matt Ahrens, co-creator of ZFS and still one of the main developers, said this [1]: "There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. Actually, ZFS can mitigate this risk to some degree if you enable the unsupported ZFS_DEBUG_MODIFY flag (zfs_flags=0x10). This will checksum the data while at rest in memory, and verify it before writing to disk, thus reducing the window of vulnerability from a memory error. I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS." [1] https://arstechnica.com/civis/threads/ars-walkthrough-using-the-zfs-next-gen-filesystem-on-linux.1235679/page-4#post-26303271 https://arstechnica.com/civis/threads/ars-walkthrough-using-...

deleted 4y ago

[deleted]

bpye 4y ago

I wish Intel and AMD would support in-band ECC on their consumer platforms. That way you can use the same DIMMs and you make the choice - either sacrifice a few percent of memory for memory correction, or don't. Intel _do_ support this on _some_ workstation products, but it's very uncommon and naturally those platforms are expensive.

sitkack 4y ago

Read the paper(s) below. Absolutely frightening, we all basically swimming in quicksand. ECC should be mandatory! Not using it is like huffing leaded gas. Silent Data Corruptions at Scale https://arxiv.org/abs/2102.11245 https://arxiv.org/abs/2102.11245 Harish Dixit has two other papers available https://arxiv.org/search/cs?searchtype=author&query=Dixit%2C+H+D https://arxiv.org/search/cs?searchtype=author&query=Dixit%2C... Revisiting Memory Errors in Large-Scale Production Data Centers: Analysis and Modeling of New Trends from the Field https://users.ece.cmu.edu/~omutlu/pub/memory-errors-at-facebook_dsn15.pdf https://users.ece.cmu.edu/~omutlu/pub/memory-errors-at-faceb... Justin Meza has a similar body of research. https://www.semanticscholar.org/author/Justin-Meza/144606145 https://www.semanticscholar.org/author/Justin-Meza/144606145

sitkack 4y ago
Justin Meza's thesis https://kilthub.cmu.edu/articles/thesis/Large_Scale_Studies_of_Memory_Storage_and_Network_Failures_in_a_Modern_Data_Center/7571684/1 https://kilthub.cmu.edu/articles/thesis/Large_Scale_Studies_...

SergeAx 4y ago

I don't know. I am building my own computers for about 30 years, starting with Intel 486, and using them heavily. Couple of times I bought ECC memory, but most of the time - not. For the last 4 years I have 64G of non-ECC RAM. Maybe I had some problem with non-ECC memory during all those years, like sudden BSODs, freezes or some corrupted data, but never anything significant. So I honestly don't see a reason for drama here.

olliecornelia 4y ago

Cosmic rays make things fun.

temptemptemp111 4y ago

[dead]