11 ms·

Non-ECC memory corrupted my hard drive image [video]

129 points by zeristor 4y ago

TacticalCoder 4y ago
> Why ECC Memory Is So Important Except that it's not for many use cases. It's great for servers but for people on their personal and/or work computer, it's simply not that useful. Seriously: which percentage of developers have ECC on their development machine(s)? As developers we live in a world of SSH, cryptographic hashes, checksums everywhere, Git repositories (that is a big one), Merkle trees, digital signatures, reproducible builds (which are gaining traction), etc. Heck, I'm torrenting the latest Debian or Devuan .iso image. My torrent client is using every known trick under the sun to make sure that should anything go wrong, the broken data shall be discarded and re-downloaded. Download is done, I dd the image to some installation medium. I can then verify its checksum matches the official one. A bit flip didn't slip by unnoticed. All the music I carefully ripped from my audio CDs? They're all cross-checked with an online DB of known bit-perfect rips. There's an accompanying file containing each song's hash and I can verify at anytime that all my files are 100% correct. But really most of all I live in a world of Git repositories. My entire Emacs config is versioned under Git (I know YMMV but I like it that way). Some people version under Git their entire user dir. Tell me how my lack of ECC is going to really make life miserable here? I have nothing against ECC... But if I want to upgrade my AMD 3700X to a 7700X, apparently I cannot get ECC. And that's totally fine: I certainly won't discard the 7700X because I cannot get ECC for it. And if anything looks suspicious, running Memtest is the first thing you should do. I've had bad RAM at times. I'm still there.
- rwmj 4y ago
  But if I open up an editor and write a document then that could be corrupted in RAM and then the corrupted data saved to disk. The document is likely to be more important than some ripped CDs or a git repo that I can download again. The CPU, RAM and mobo manufacturers need to get together and make ECC RAM mandatory. It's absurd that we have machines with gigabytes of storage using microscopic (nanoscopic??) capacitors that doesn't have this basic protection. And honestly this should have happened years ago. (Edit: And before anyone says DDR5 is ECC by default, that's not quite true although the difference is a bit subtle: https://en.wikipedia.org/wiki/DDR5_SDRAM#DIMMs_versus_memory_chips https://en.wikipedia.org/wiki/DDR5_SDRAM#DIMMs_versus_memory...)
  - ilyt 4y ago
    >But if I open up an editor and write a document then that could be corrupted in RAM and then the corrupted data saved to disk. The document is likely to be more important than some ripped CDs or a git repo that I can download again. Devils advocate: if it is just some bits in character flipped it's entirely recoverable while flipping some bits in compressed stream for video would corrupt more.
    - rwmj 4y ago
      Agreed. One problem with RAM errors (which I've actually experienced) is they are insidious. You probably won't notice them immediately so the error can be propagated, and they're very very difficult to diagnose if you do notice them. Back in the 90s we had a database server which had a stuck bit in memory normally mapped to the page cache. This caused sectors to be written to the backing software RAID which couldn't be read back in (because I think some checksum was corrupted when written and then failed when read back). It took an absolute age to diagnose this. I think I only worked it out by eliminating everything else.
- craftkiller 4y ago
  > Seriously: which percentage of developers have ECC on their development machine(s)? Every single one of my non-mini computers uses ECC ram except for my laptop. If someone would release a framework laptop motherboard that supports ECC ram (and preferably risc-v) I'd finally be able to close to reliability gap. It blows my mind that we say "Well sure, we COULD make infallible ram, but that would cost a tiny bit extra so instead lets just hope nothing bad happens." That's right up there with not wearing a seat-belt because I haven't needed one yet.
  - Brian_K_White 4y ago
    "If someone would release a framework laptop motherboard that supports ECC ram..." This. If I could, I would. The only reason my laptop doesn't have ecc is because the manufacturer doesn't offer the option in any macines I otherwise want. That comment was very misguided in trying to suggest that there is any valid excuse to tolerate unreliable execution hardware. git and ssh and md5sums do not mean that it's ok if your very brain can't be trusted to deliver data from one part to another within itself, or spit back the same data that was put in a cell. Everything else is built upon that!
  - digitallyfree 4y ago
    I develop on VMs on my server (with ECC and RAIDZ) which i access using SSH or VDI protocols. I would love to have ECC on all my machines but that isn't feasible until the status quo changes, so I stick with the remote approach. To me that's an acceptable tradeoff as the non-ECC desktops/laptops are just used as dumb terminals while the real work happens on the reliable server. I can't speak for ECC at the moment but ZFS has definitely saved me from data corruption that would have been left to manifset otherwise.
- rhn_mk1 4y ago
  > which percentage of developers have ECC on their development machine(s)? That question won't help you evaluate the demand for ECC simply because the supply is strangled. Those who want it have to make compromises to get ECC: get a Xeon, or get one of few AMD motherboards with matching CPUs and overpriced RAM without a guarantee that it will end up working.
  - simoncion 4y ago
    > ...get one of few AMD motherboards with matching CPUs and overpriced RAM... I mean, if you want a _guarantee_, then sure, get one of those certified-for-ECC motherboards. But, like, as far as I know, going _at least_ far back as the Phenom II (released in 2008) AMD desktop processors have always supported ECC RAM. And -as far as I know- ASUS motherboards for said processors have always supported dropping in ECC RAM (and Linux and memtest and friends have always agreed that ECC was enabled and functioning in such a system). Source: Personal experience with Phenom II, Threadripper, and Ryzen 5 CPUs and ASUS motherboards, and looking-from-a-distance at the rest of the AMD CPUs between the Phenom and the Ryzen 5.
- tasubotadas 4y ago
  When people complain about these random OS crashes and freezes it's usually RAM corruption at fault. Per 1gb of RAM, you can expect to see 266 bit errors per month[1-2] if you are using your PC 16h per day. Multiply that by 64GB or 128GB of RAM and it's crazy to think that you won't run into any of the stability issues. [1-2] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf https://static.googleusercontent.com/media/research.google.c... [1-2] https://en.wikipedia.org/wiki/ECC_memory#Research https://en.wikipedia.org/wiki/ECC_memory#Research
  - ubercow13 4y ago
    >266 bit errors per month That seems like an overestimate. How can memtest ever pass on non-ECC RAM if errors are that frequent?
    - moloch-hai 4y ago
      Because memtest only looks at values it wrote out very recently, before they have had a chance to flip. Memtest is looking for reliable failures, not evanescent one-off events.
      - deleted 4y ago
        [deleted]
      - willis936 4y ago
        SDRAM is continuously refreshing all cells. How long ago data was written doesn't make a big difference (aside from the case where you're reading data immediately after writing or reading that data).
        moloch-hai 4y ago
        This does not, of course, make any sense. Any given memory cell will be read once per, say, millisecond, and the value written back. Once it has flipped once, the wrong value will then be written back, and after that the wrong value is read back out and rewritten again, indefinitely. Errors are sticky, and accumulate. (Flipping back again is negligible unlikely.) With ECC in the refresh path, such an error could be corrected and the right value would be overwritten over top of the bad one. Then errors would not accumulate, but would instead be "scrubbed". Mainframe machines scrub their RAM. Disks too.
- justsomehnguy 4y ago
  And condoms are only 99.8% effective, so let's not use it? You can always pull out in time or verify checksu^W^W Plan B? > But really most of all I live in a world of Git repositories That's great for you, but 99% don't even know what Git is, just like checksums, cryptographic hashes, Merkle trees, digital signatures and reproducible builds. You just miss the one imortant thing: ECC isn't that helpful where you have the means to check and verify the data. ECC is the only way to at least know what something is happening with data when there is no way to check. To give you a slight idea I would tell you an anecdote from my L1 life almost two decades ago: I visited a client who claimed what the PC was working erratically and constantly threw weird error messages. Welp, the usual deal, just some ugly software or a virus. In the first 3 minutes I got like 5 errors about failing to load a .dll from C:\WINDOWS\SYSTEM33\USER32.DLL, nothing unusual, just need to.. WAIT. Why system33? Stupid virus masking for a well-known folder? Doubt. So I go to C:\Windows and I see: System32 System33 System34 If you are a smart fellow you probably already understood what that was a bit-flip error which somehow managed to be in the in-memory copy of MFT of the system drive. And this is the only reason the user noticed it - because sometimes programs wouldn't start and sometimes there was weird error messages. If that error was in the data area of some program - it wouldn't be discovered at all.
  - deleted 4y ago
    [deleted]
  - dmitrybrant 4y ago
    Hang on... to go from "System32" to "System33" is one bit flip, but to go from "System33" to "System34" is three bit flips at once. Doesn't that seem astronomically more likely to be a malicious program than bad RAM?
    - justsomehnguy 4y ago
      It's astronomically more likely to be a manufacturing defect what affected multiple cells than a malicious program what ... wrote to System34? And how the replacement of the DRAM module 'cured' that malicious program?
- dale_glass 4y ago
  > But really most of all I live in a world of Git repositories. My entire Emacs config is versioned under Git (I know YMMV but I like it that way). Some people version under Git their entire user dir. > Tell me how my lack of ECC is going to really make life miserable here? Git will break if RAM is bad just like anything else. That it checksums everything won't save you from checking in corrupt data, the filesystem itself being corrupt, or some internal git structure becoming corrupt. Losing your repo because something in it was written wrong is very much a possibility. Having multiple machines involved helps, but it's not a complete fix, because the possibility exists of something damaged being transmitted from a broken machine to a good one, ensuring there's no good copy anywhere. There's really nothing software can do to operate correctly with bad RAM all of the time. Instructions for the software are in RAM. The OS that the software expects to behave right is in RAM. Various buffers used for disk access and networking are in RAM. An application like git assumes all of that is performing correctly, and can't compensate for every possible malfunction that could happen.
  - everybodyknows 4y ago
    How trustworthy is git-fsck for detecting random-bit corruption?
    - dale_glass 4y ago
      Depends on how you look at it. For actual verification, unless there's a bug my understanding is that very trustworthy. But by that point it's already too late. Okay, you know something is broken, but that won't give your good data back. But that only tells you that Git data is intact and that all the hashes match. If git got a corrupted file to start with, then correctly hashed it, everything will verify 100% and still be broken.
- kevingadd 4y ago
  I went out of my way for ECC because losing files can mean losing days of work. With how much a sweng gets paid it's worth the ECC premium if it saves me a few days of work over the life of the machine. Recently I discovered that one of my SSDs was quietly failing without setting off any warnings; doing a chkdsk showed that some files had already gotten corrupted. One of them was my backblaze backup index! Even though I have automated backups (backblaze + macrium backups to a NAS), recovering files from them is non-trivial. If I were to lose work to non-ECC ram who knows how long it would take me to reconstruct a known-good work environment and file set. Imagine if you're working on something huge and hard to validate like neural net weights where corruption can occur silently and be hard to detect after it's happened?
  - newZWhoDis 4y ago
    My favorite is when data silently corrupts, and then is happily propagated to your off-site recovery :/
    - kevingadd 4y ago
      Yeah, I don't know when the drive started failing, so in practice my backups are probably all screwed too. My only option would be to get a spare drive, restore a backup to it, then do a block level diff of the current and backup volumes and try to figure out whether any of the differences are file corruption.
      - Filligree 4y ago
        Probably I'm saying nothing you haven't thought of, but ZFS is great for preventing this. If you also pair it with ECC, then you've eliminated most ways to cause corruption.
- leguminous 4y ago
  Checksums don't save you from memory corruption. If your data gets corrupted in memory, you will just end up checksumming and committing bad data. Or your checksum could get corrupted, and you commit a checksum that doesn't match your data. Checksums are more useful for safeguarding against disk or network corruption (although you shouldn't have network corruption issues over TLS or SSH). Apparently Ryzen 7000 cpus can use ECC. I've heard reports that AMD needs to release an AGESA update, though, and ECC DDR5 memory availability is terrible. I'm hopeful that the situation will improve, because I also want to update my desktop. I've been using ECC memory since losing a filesystem on a desktop when a DIMM went bad.
stefantalpalaru 4y ago
[dead]
moloch-hai 4y ago
We don't have ECC mainly because Intel has long been hostile to "consumer" access to ECC. Apparently this was conceived as a market segmentation scheme: people outfitting servers could get ECC when they pay a huge premium. They would thereby not be tempted to cheap out and buy consumer-grade equipment, otherwise wholly adequate to meet all their needs at a radically cheaper price. That we cannot get laptops or even desk machines with ECC, and so have them crash frequently, is seen as a trivial side effect of the strategy. If you did not hate Intel enough before, you may increase your hatred accordingly. Intel doesn't hate you back; they simply care not even a little how you feel. (Historically, just running Microsoft software was overwhelmingly more likely to be the cause of a crash than a memory bit-flip; and there were orders of magnitude fewer RAM bits at risk. Microsoft succeeded in getting customers to accept and even expect frequent crashes; before MS, a program crashing was grounds for a refund.)
- eternityforest 4y ago
  I used to be a big fan of Intel, up until the latest chips from other companies that seem to have beat them on performance/watt. My next laptop will probably be AMD if the situation hasn't changed.
  - vladvasiliu 4y ago
    I'm happy with my AMD based laptop. But I haven't seen any that support ECC. But I did see a Lenovo model, IIRC, that had some kind of Xeon and ECC. Not sure what the noise and battery life situations on that thing were, though.
    - OJFord 4y ago
      I realise that's blurrier when it comes to laptops, but AIUI it's more a case of whether the motherboard than supports it than about the AMD chip. i.e. given a desktop CPU, as far as I know you can put it in a motherboard that either does or does not support ECC RAM.
      - 4y ago
dale_glass 4y ago
Yup, been there. Way back I had a Pentium 133 doing firewall duty in a closet. It did approximately nothing besides iptables, but of course any machine has logs, updates and so on going on. After running fine for months one day it suddenly died. I rebooted it. A few days later it died again. Another reboot. Then it died for the last time and failed to boot at all. Examination showed the disk was corrupt and couldn't be mounted. Further examination showed that one of the memory modules was loose for some reason, could be that it was never firmly in and I just bumped the box when messing with something else. Then came the wasted weekend of dealing with that my normal internet connection relied on the thing that was now completely broken. And that was the luckiest case I can imagine, when the broken machine contains no data of actual value. Since then I'm very paranoid, always run memtest on any new RAM I buy overnight, and have ECC where it's possible to have it.
- vladvasiliu 4y ago
  > Since then I'm very paranoid, always run memtest on any new RAM I buy overnight, and have ECC where it's possible to have it. Yeah, I do the same, but I've learned that you have to do it regularly. In one of my desktop machines, the RAM ran fine for like two years. Then, all of sudden, random Firefox segfaults, etc. Whipped up a memtest ISO, and sure enough, one of the sticks was bad.
  - dale_glass 4y ago
    That's the nice thing about ECC, it acts like an always running memory test. You normally have a scrub time that can be configured in the BIOS, which also adds a regular verification of the entire RAM at regular intervals, just in case something goes wrong in some rarely used part of the memory.
    - IYasha 4y ago
      Unfortunately, background scrubbing significantly increases power consumption and impacts performance as well.
H8crilA 4y ago
This problem has no ultimate solution. I've seen all components flip bits, CPUs, networking cards, RAM, most often you just can't know for sure what did it. You can remedy it a bit (like with ECC), but ultimately there will always be corruption if you process hundreds of petabytes of data. Get used to it, your computer executes an instruction with a probability extremely close to 1, but not equal to 1. Deep in the archives of a well known tech company is a very well documented case of a bit flip that caused the wrong function to be executed in a C++ v-table. The big oof was that this function was the equivalent of an SQL "drop table", and just happened to be 32 bytes off of a very benign function that did something like stat(). Really funny stuff once the crisis is over :)
- water8 4y ago
  [dead]
- don-code 4y ago
  Is this something that's documented publicly? I'd love to read more.
- dale_glass 4y ago
  ECC isn't a terribly complicated technology, and can be used in all those cases. In limited cases, a checksum is good enough. If you checksum outgoing data, and verify it on reception, then it being corrupted in transit whether on the network card or the cable can be detected and transparently compensated for. Really, we can do much better than to "get used to it".
  - H8crilA 4y ago
    You are under the impression that CPUs and other chips always perform the same instructions as are written in the code, and only RAM can flip bits because DRAM is DRAM :) It can (and should! whenever possible) be improved, not fixed. There's always that pesky gamma that can hit a specific transistor, even if it is deep underground. Gamma cannot be fully stopped. At certain scales data corruption becomes directly measurable. And yes, corruption levels vary between pieces of hardware.
baeaz 4y ago
[flagged]
jeroenhd 4y ago
It's good that with DDR5 consumer memory will get some super basic ECC on die, so hopefully the next generation of memory will make the problematic sticks more obvious (or prevent damage in the very least). ECC won't save you from memory corruption, but it'll save your data at least. Personally, I would've just checksummed the individual failing files rather than the disk image and only back up the bad files separately. There are all kinds of ways for a disk image to fail and I wouldn't spend a second longer on it than absolutely necessary. The whole memtest permutation setup also would've been too much work for. E, I would just declare the motherboard faulty when two sticks that otherwise pass the test fail in specific configurations. A new motherboard is cheaper than super specific RAM sticks.
ksec 4y ago
This is especially true for NAS. And why you need BTRFS or preferably ZFS. Unfortunately none of the consumer NAS offers ZFS, and BTRFS is still not a default option. Neither Synology or Qnap seems to care.
- aborsy 4y ago
  Synology offers ECC in 2023 consumer modes such as 923. Still the experience that synology btrfs provides is nowhere as good as ZFS (due to a lot of limitations).
  - deleted 4y ago
    [deleted]
  - metadat 4y ago
    I bought a 6 disk Synology a few months ago and it came with ECC by default. I did a cursory web search about this just now and ECC support appears to be the norm for 22 (as in the year 2022) model revisions and newer (thankfully!).
    - layer8 4y ago
      It’s because they use AMD CPUs now (already in the 21 models). The trade-off is that those CPUs have worse hardware codec support than the Intel ones they previously used, if you want to do video transcoding.
      - wil421 4y ago
        Intel sold i3s for years that had ECC unlocked. I run a i3-8100 on an x11 mobo in my TrueNAS box.
- gmokki 4y ago
  I had RAM go bad after running 18T (5 HDDs in raid1) btrfs system in closet for years. Btrfs of course noticed it and fixed most of them automatically when some of blocks were corrupt. But eventually the system failed: the tree that contains the checksums for all the other trees corrupted itself on both copies of one node. Fixed the HW problem and then had to use hex editor to set the checksum manually to correct value (I modified the kernel to print the expected value). Now the system has been again stable for 3 years.
encryptluks2 4y ago
I watched the full video. It was long but very informative. The humor at times made up for the length and the presenter showed a lot of deep knowledge that most people won't have. My biggest gripe is that they just didn't try replacing the RAM sticks in the first place. I get that they wanted to do a root cause analysis, but geez the time and patience they had to do all those memory tests. No wonder they did a video about it, cause otherwise that lost time would have been painful. I was baffled as well that dd and ddrescue work differently in how they utilize the RAM. Caught me offguard. Onto the discussion of ECC RAM. In a perfect world, all memory would be ECC... but try finding some high performance 16GB sticks of ECC DDR4 RAM like what you'll see on gaming computers. I don't even think they make anything comparable in terms of speed and definitely not costs. I guess you don't really know that you needed ECC until it's too late.
- morelikeborelax 4y ago
  > " I guess you don't really know that you needed ECC until it's too late." I spent many years on hardware consultation and was amazed at the all the times I had to explain it was just a what if insurance like any other things their business was mitigating against. Sometimes they'd even decided they needed to save costs in non-ecc ram when it was $4 a gb in difference, or (during the FB-DIMM era) there wasn't even an option to avoid it. Never really understood the resistance towards it. Maybe the lack of evidence before the Google study and people thinking RAM manufacturers were trying to rip them off or something. The "never had a problem so why would I need" it attitude with no way to know if an issue was caused by a bit flip was most baffling.
- simoncion 4y ago
  > ...but try finding some high performance 16GB sticks of ECC DDR4 RAM like what you'll see on gaming computers. Here ya go: https://nemixram.com/16gb-ddr4-3200-pc4-25600-ecc-udimm-2rx8-memory-for-server-workstation-by-nemix-ram/ https://nemixram.com/16gb-ddr4-3200-pc4-25600-ecc-udimm-2rx8... It doesn't have pretty lights on it, but it does seem to be in the same speed class that gets called "gaming RAM" by a _whole_ bunch of retailers.
  - 4y ago
chunk_waffle 4y ago
For anyone curious (as I was once and looked into it) Dell and Lenovo both ship Laptops with Xeon's and ECC memory though they are very expensive.
- thekombustor 4y ago
  Yep, you can get it on the higher end trims of the P-series Thinkpads.
- aortega 4y ago
  I have a lenovo P52, it came with 16 GB ECC. The dimm is not super-expensive, but it's hard to find, and only comes in half the size of regular dimms. I replaced it with non-ECC memory and never had an issue. In fact you can combine ECC and non-ECC memory without any problem.
lizardactivist 4y ago
I'm curious what the actual manufacturing costs for ECC DRAM is compared to regular DRAM. Is it considerably more expensive, or just the usual over-charge because it's better?
- jeffbee 4y ago
  It’s exactly 1/8th more.
- wil421 4y ago
  It has another error correcting chip on DIMM, 9 total instead of 8. Don’t be fooled by on-die ecc marketing that’s come out recently, you need the 9th chip for parity. https://community.fs.com/blog/ecc-vs-non-ecc-memory-which-one-is-better.html https://community.fs.com/blog/ecc-vs-non-ecc-memory-which-on...
deleted 4y ago
[deleted]
IYasha 4y ago
It would be so nice if DD and DDRescue did calculate hashes while copying.
tibbydudeza 4y ago
Got a Gen 10 HPE Microserver for my NAS run some AMD dual core SoC - factory fitted with ECC memory running Unraid. Think it mysteriously crashed once or twice in the 4 years I had it and the HP diagnostic light came on.
tibbydudeza 4y ago
Afaik ECC memory is slower than normal memory, so it does not impress the folks who base their purchase decisions on benchmark scores rather than utility and best bang for the buck.
mrlonglong 4y ago
We've got a server that keeps rebooting due to a bad ECC DIMM chip. I thought the whole point of ECC was to keep the server going until we can replace the DIMM?
- AaronFriel 4y ago
  ECC can typically correct 1 bit errors and usually detect (and fault) on 2 bit errors. Your server is faulting and preventing itself from corrupting data.
- sgtnoodle 4y ago
  It sounds like it's working as intended. Redundancy doesn't help you with reliability once your system is in a degraded state. Running a server with a known bad ram stick is like driving a car on run-flat tires. It will probably get you where you need to go, but you really should be driving to a tire shop.
  - mrlonglong 4y ago
    Ok thanks I understand things better now. I'll pester my boss to get new DIMM asap.
- exikyut 4y ago
  IIUC it's the kernel EDAC subsystem rebooting the server for you. If you turn it off you'll get non-ECC behaviour, but in this case you'll also get really interesting system issues because as the sibling comments have said your RAM is actually faulty :P
- to11mtm 4y ago
  ECC Corrects 1 bit errors and detects 2 bit errors. It does not handle all hardware failures however. An actual bad chip may fall into a different category, I think that's when technologies like Chipkill[0] might come into play. [0] - https://en.wikipedia.org/wiki/Chipkill https://en.wikipedia.org/wiki/Chipkill
aortega 4y ago
I have 128 GB of non-ecc memory in my notebook, never detected a single error, and has been on 24/7 for more than 4 years. Unless you live over 4000 meters over the sea level, like to compile while flying or live close to an unshielded nuclear reactor, you don't need ECC. And most memory problems you can fix by better cooling, and better shielding.
- LeoPanthera 4y ago
  > I have 128 GB of non-ecc memory in my notebook, never detected a single error How would you know?
  - aortega 4y ago
    I know that I never detected them. Also it never crashed nor malfunctioned in any way (thanks in part to Linux), so any undetected error didn't caused any problem.
    - russdill 4y ago
      There's been plenty of studies they measure bit flips in memory. If you really wanted to measure, you'd find them without much difficulty. Yes though, when you have gobs of memory, you're unlikely to see the effects of bit flips. That's not the same at them not occuring.
- mech4bg 4y ago
  My 4 year old Mac Mini had been showing some flakiness and finally I got an error about "hash mismatch detected on volume" - ran memtest and sure enough the RAM had gone bad. I rely on this machine for all my media, backups, etc, goodness knows what data I may have lost while the RAM was bad. It's a moderately common problem, found plenty of people with the same issue.
- PragmaticPulp 4y ago
  I’ve had 128GB of ECC memory with error reporting. Never logged a bit flip as far as I can tell. Memory errors are heavily concentrated to bad modules. They’re not evenly distributed across all RAM. Always test the stability of your memory when doing a new build.
meindnoch 4y ago
Data written but not read back may as well go straight to /dev/null. Always read back and verify integrity.
ritcgab 4y ago
Another important perspective of ECC memory is whether your machine can properly respond to the interrupt. ECC without reporting and handling is not a working ECC. I have been using Ryzen machines with AsRock motherboards and the Linux EDAC driver reports errors correctly (verified by overclocking the memory sticks to the edge). On the other hand, I owned a Dell Precision mobile workstation with Xeon W-11955M processor and ECC SODIMM sticks. However, since its EDAC driver never really lands in the mainline kernel, I won't get any ECC reports, so the corresponding process that has a bit flipped will not get killed.
justinclift 4y ago
Near the end, the video mentions his (old) computer "apparently can't take 'high density' memory". Then the author goes through a process of randomly buying memory and hoping it works. Anyone know what that "high density" memory problem is about? Maybe it's a misunderstanding of memory channels and ranks?
- to11mtm 4y ago
  Not sure if it's what he means, but in my experience 'High density' refers to less chips for a given capacity. When I used to sling PC hardware, as memory densities increased, you would run into things like 'This Motherboard will only take a 256MB module if it has 8 chips on both sides (16 chips total), it will not take a 256MB module with 8 chips on one side (8 chips total)' Depending on the board, it might only register part of the capacity and be stable, might register part of the capacity and be unstable, or just not boot at all.
  - justinclift 4y ago
    Interesting. Have heard mention of similar over the years, but never seen it in person myself. Sounds strange, as - without knowing any better - I'd expect the memory interface to be agnostic to the chips implementing it. eg "address 0x11111111" on 'high density' ram would be seen exactly the same as "address 0x11111111" on other ram The mind boggles. :)

129 points by zeristor 4y ago

> Why ECC Memory Is So Important Except that it's not for many use cases. It's great for servers but for people on their personal and/or work computer, it's simply not that useful. Seriously: which percentage of developers have ECC on their development machine(s)? As developers we live in a world of SSH, cryptographic hashes, checksums everywhere, Git repositories (that is a big one), Merkle trees, digital signatures, reproducible builds (which are gaining traction), etc. Heck, I'm torrenting the latest Debian or Devuan .iso image. My torrent client is using every known trick under the sun to make sure that should anything go wrong, the broken data shall be discarded and re-downloaded. Download is done, I dd the image to some installation medium. I can then verify its checksum matches the official one. A bit flip didn't slip by unnoticed. All the music I carefully ripped from my audio CDs? They're all cross-checked with an online DB of known bit-perfect rips. There's an accompanying file containing each song's hash and I can verify at anytime that all my files are 100% correct. But really most of all I live in a world of Git repositories. My entire Emacs config is versioned under Git (I know YMMV but I like it that way). Some people version under Git their entire user dir. Tell me how my lack of ECC is going to really make life miserable here? I have nothing against ECC... But if I want to upgrade my AMD 3700X to a 7700X, apparently I cannot get ECC. And that's totally fine: I certainly won't discard the 7700X because I cannot get ECC for it. And if anything looks suspicious, running Memtest is the first thing you should do. I've had bad RAM at times. I'm still there.

rwmj 4y ago
But if I open up an editor and write a document then that could be corrupted in RAM and then the corrupted data saved to disk. The document is likely to be more important than some ripped CDs or a git repo that I can download again. The CPU, RAM and mobo manufacturers need to get together and make ECC RAM mandatory. It's absurd that we have machines with gigabytes of storage using microscopic (nanoscopic??) capacitors that doesn't have this basic protection. And honestly this should have happened years ago. (Edit: And before anyone says DDR5 is ECC by default, that's not quite true although the difference is a bit subtle: https://en.wikipedia.org/wiki/DDR5_SDRAM#DIMMs_versus_memory_chips https://en.wikipedia.org/wiki/DDR5_SDRAM#DIMMs_versus_memory...)
- ilyt 4y ago
  >But if I open up an editor and write a document then that could be corrupted in RAM and then the corrupted data saved to disk. The document is likely to be more important than some ripped CDs or a git repo that I can download again. Devils advocate: if it is just some bits in character flipped it's entirely recoverable while flipping some bits in compressed stream for video would corrupt more.
  - rwmj 4y ago
    Agreed. One problem with RAM errors (which I've actually experienced) is they are insidious. You probably won't notice them immediately so the error can be propagated, and they're very very difficult to diagnose if you do notice them. Back in the 90s we had a database server which had a stuck bit in memory normally mapped to the page cache. This caused sectors to be written to the backing software RAID which couldn't be read back in (because I think some checksum was corrupted when written and then failed when read back). It took an absolute age to diagnose this. I think I only worked it out by eliminating everything else.
craftkiller 4y ago
> Seriously: which percentage of developers have ECC on their development machine(s)? Every single one of my non-mini computers uses ECC ram except for my laptop. If someone would release a framework laptop motherboard that supports ECC ram (and preferably risc-v) I'd finally be able to close to reliability gap. It blows my mind that we say "Well sure, we COULD make infallible ram, but that would cost a tiny bit extra so instead lets just hope nothing bad happens." That's right up there with not wearing a seat-belt because I haven't needed one yet.
- Brian_K_White 4y ago
  "If someone would release a framework laptop motherboard that supports ECC ram..." This. If I could, I would. The only reason my laptop doesn't have ecc is because the manufacturer doesn't offer the option in any macines I otherwise want. That comment was very misguided in trying to suggest that there is any valid excuse to tolerate unreliable execution hardware. git and ssh and md5sums do not mean that it's ok if your very brain can't be trusted to deliver data from one part to another within itself, or spit back the same data that was put in a cell. Everything else is built upon that!
- digitallyfree 4y ago
  I develop on VMs on my server (with ECC and RAIDZ) which i access using SSH or VDI protocols. I would love to have ECC on all my machines but that isn't feasible until the status quo changes, so I stick with the remote approach. To me that's an acceptable tradeoff as the non-ECC desktops/laptops are just used as dumb terminals while the real work happens on the reliable server. I can't speak for ECC at the moment but ZFS has definitely saved me from data corruption that would have been left to manifset otherwise.
rhn_mk1 4y ago
> which percentage of developers have ECC on their development machine(s)? That question won't help you evaluate the demand for ECC simply because the supply is strangled. Those who want it have to make compromises to get ECC: get a Xeon, or get one of few AMD motherboards with matching CPUs and overpriced RAM without a guarantee that it will end up working.
- simoncion 4y ago
  > ...get one of few AMD motherboards with matching CPUs and overpriced RAM... I mean, if you want a _guarantee_, then sure, get one of those certified-for-ECC motherboards. But, like, as far as I know, going _at least_ far back as the Phenom II (released in 2008) AMD desktop processors have always supported ECC RAM. And -as far as I know- ASUS motherboards for said processors have always supported dropping in ECC RAM (and Linux and memtest and friends have always agreed that ECC was enabled and functioning in such a system). Source: Personal experience with Phenom II, Threadripper, and Ryzen 5 CPUs and ASUS motherboards, and looking-from-a-distance at the rest of the AMD CPUs between the Phenom and the Ryzen 5.
tasubotadas 4y ago
When people complain about these random OS crashes and freezes it's usually RAM corruption at fault. Per 1gb of RAM, you can expect to see 266 bit errors per month[1-2] if you are using your PC 16h per day. Multiply that by 64GB or 128GB of RAM and it's crazy to think that you won't run into any of the stability issues. [1-2] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf https://static.googleusercontent.com/media/research.google.c... [1-2] https://en.wikipedia.org/wiki/ECC_memory#Research https://en.wikipedia.org/wiki/ECC_memory#Research
- ubercow13 4y ago
  >266 bit errors per month That seems like an overestimate. How can memtest ever pass on non-ECC RAM if errors are that frequent?
  - moloch-hai 4y ago
    Because memtest only looks at values it wrote out very recently, before they have had a chance to flip. Memtest is looking for reliable failures, not evanescent one-off events.
    - deleted 4y ago
      [deleted]
    - willis936 4y ago
      SDRAM is continuously refreshing all cells. How long ago data was written doesn't make a big difference (aside from the case where you're reading data immediately after writing or reading that data).
      - moloch-hai 4y ago
        This does not, of course, make any sense. Any given memory cell will be read once per, say, millisecond, and the value written back. Once it has flipped once, the wrong value will then be written back, and after that the wrong value is read back out and rewritten again, indefinitely. Errors are sticky, and accumulate. (Flipping back again is negligible unlikely.) With ECC in the refresh path, such an error could be corrected and the right value would be overwritten over top of the bad one. Then errors would not accumulate, but would instead be "scrubbed". Mainframe machines scrub their RAM. Disks too.
justsomehnguy 4y ago
And condoms are only 99.8% effective, so let's not use it? You can always pull out in time or verify checksu^W^W Plan B? > But really most of all I live in a world of Git repositories That's great for you, but 99% don't even know what Git is, just like checksums, cryptographic hashes, Merkle trees, digital signatures and reproducible builds. You just miss the one imortant thing: ECC isn't that helpful where you have the means to check and verify the data. ECC is the only way to at least know what something is happening with data when there is no way to check. To give you a slight idea I would tell you an anecdote from my L1 life almost two decades ago: I visited a client who claimed what the PC was working erratically and constantly threw weird error messages. Welp, the usual deal, just some ugly software or a virus. In the first 3 minutes I got like 5 errors about failing to load a .dll from C:\WINDOWS\SYSTEM33\USER32.DLL, nothing unusual, just need to.. WAIT. Why system33? Stupid virus masking for a well-known folder? Doubt. So I go to C:\Windows and I see: System32 System33 System34 If you are a smart fellow you probably already understood what that was a bit-flip error which somehow managed to be in the in-memory copy of MFT of the system drive. And this is the only reason the user noticed it - because sometimes programs wouldn't start and sometimes there was weird error messages. If that error was in the data area of some program - it wouldn't be discovered at all.
- deleted 4y ago
  [deleted]
- dmitrybrant 4y ago
  Hang on... to go from "System32" to "System33" is one bit flip, but to go from "System33" to "System34" is three bit flips at once. Doesn't that seem astronomically more likely to be a malicious program than bad RAM?
  - justsomehnguy 4y ago
    It's astronomically more likely to be a manufacturing defect what affected multiple cells than a malicious program what ... wrote to System34? And how the replacement of the DRAM module 'cured' that malicious program?
dale_glass 4y ago
> But really most of all I live in a world of Git repositories. My entire Emacs config is versioned under Git (I know YMMV but I like it that way). Some people version under Git their entire user dir. > Tell me how my lack of ECC is going to really make life miserable here? Git will break if RAM is bad just like anything else. That it checksums everything won't save you from checking in corrupt data, the filesystem itself being corrupt, or some internal git structure becoming corrupt. Losing your repo because something in it was written wrong is very much a possibility. Having multiple machines involved helps, but it's not a complete fix, because the possibility exists of something damaged being transmitted from a broken machine to a good one, ensuring there's no good copy anywhere. There's really nothing software can do to operate correctly with bad RAM all of the time. Instructions for the software are in RAM. The OS that the software expects to behave right is in RAM. Various buffers used for disk access and networking are in RAM. An application like git assumes all of that is performing correctly, and can't compensate for every possible malfunction that could happen.
- everybodyknows 4y ago
  How trustworthy is git-fsck for detecting random-bit corruption?
  - dale_glass 4y ago
    Depends on how you look at it. For actual verification, unless there's a bug my understanding is that very trustworthy. But by that point it's already too late. Okay, you know something is broken, but that won't give your good data back. But that only tells you that Git data is intact and that all the hashes match. If git got a corrupted file to start with, then correctly hashed it, everything will verify 100% and still be broken.
kevingadd 4y ago
I went out of my way for ECC because losing files can mean losing days of work. With how much a sweng gets paid it's worth the ECC premium if it saves me a few days of work over the life of the machine. Recently I discovered that one of my SSDs was quietly failing without setting off any warnings; doing a chkdsk showed that some files had already gotten corrupted. One of them was my backblaze backup index! Even though I have automated backups (backblaze + macrium backups to a NAS), recovering files from them is non-trivial. If I were to lose work to non-ECC ram who knows how long it would take me to reconstruct a known-good work environment and file set. Imagine if you're working on something huge and hard to validate like neural net weights where corruption can occur silently and be hard to detect after it's happened?
- newZWhoDis 4y ago
  My favorite is when data silently corrupts, and then is happily propagated to your off-site recovery :/
  - kevingadd 4y ago
    Yeah, I don't know when the drive started failing, so in practice my backups are probably all screwed too. My only option would be to get a spare drive, restore a backup to it, then do a block level diff of the current and backup volumes and try to figure out whether any of the differences are file corruption.
    - Filligree 4y ago
      Probably I'm saying nothing you haven't thought of, but ZFS is great for preventing this. If you also pair it with ECC, then you've eliminated most ways to cause corruption.
leguminous 4y ago
Checksums don't save you from memory corruption. If your data gets corrupted in memory, you will just end up checksumming and committing bad data. Or your checksum could get corrupted, and you commit a checksum that doesn't match your data. Checksums are more useful for safeguarding against disk or network corruption (although you shouldn't have network corruption issues over TLS or SSH). Apparently Ryzen 7000 cpus can use ECC. I've heard reports that AMD needs to release an AGESA update, though, and ECC DDR5 memory availability is terrible. I'm hopeful that the situation will improve, because I also want to update my desktop. I've been using ECC memory since losing a filesystem on a desktop when a DIMM went bad.

stefantalpalaru 4y ago

[dead]

moloch-hai 4y ago

We don't have ECC mainly because Intel has long been hostile to "consumer" access to ECC. Apparently this was conceived as a market segmentation scheme: people outfitting servers could get ECC when they pay a huge premium. They would thereby not be tempted to cheap out and buy consumer-grade equipment, otherwise wholly adequate to meet all their needs at a radically cheaper price. That we cannot get laptops or even desk machines with ECC, and so have them crash frequently, is seen as a trivial side effect of the strategy. If you did not hate Intel enough before, you may increase your hatred accordingly. Intel doesn't hate you back; they simply care not even a little how you feel. (Historically, just running Microsoft software was overwhelmingly more likely to be the cause of a crash than a memory bit-flip; and there were orders of magnitude fewer RAM bits at risk. Microsoft succeeded in getting customers to accept and even expect frequent crashes; before MS, a program crashing was grounds for a refund.)

eternityforest 4y ago
I used to be a big fan of Intel, up until the latest chips from other companies that seem to have beat them on performance/watt. My next laptop will probably be AMD if the situation hasn't changed.
- vladvasiliu 4y ago
  I'm happy with my AMD based laptop. But I haven't seen any that support ECC. But I did see a Lenovo model, IIRC, that had some kind of Xeon and ECC. Not sure what the noise and battery life situations on that thing were, though.
  - OJFord 4y ago
    I realise that's blurrier when it comes to laptops, but AIUI it's more a case of whether the motherboard than supports it than about the AMD chip. i.e. given a desktop CPU, as far as I know you can put it in a motherboard that either does or does not support ECC RAM.
    - 4y ago

dale_glass 4y ago

Yup, been there. Way back I had a Pentium 133 doing firewall duty in a closet. It did approximately nothing besides iptables, but of course any machine has logs, updates and so on going on. After running fine for months one day it suddenly died. I rebooted it. A few days later it died again. Another reboot. Then it died for the last time and failed to boot at all. Examination showed the disk was corrupt and couldn't be mounted. Further examination showed that one of the memory modules was loose for some reason, could be that it was never firmly in and I just bumped the box when messing with something else. Then came the wasted weekend of dealing with that my normal internet connection relied on the thing that was now completely broken. And that was the luckiest case I can imagine, when the broken machine contains no data of actual value. Since then I'm very paranoid, always run memtest on any new RAM I buy overnight, and have ECC where it's possible to have it.

vladvasiliu 4y ago
> Since then I'm very paranoid, always run memtest on any new RAM I buy overnight, and have ECC where it's possible to have it. Yeah, I do the same, but I've learned that you have to do it regularly. In one of my desktop machines, the RAM ran fine for like two years. Then, all of sudden, random Firefox segfaults, etc. Whipped up a memtest ISO, and sure enough, one of the sticks was bad.
- dale_glass 4y ago
  That's the nice thing about ECC, it acts like an always running memory test. You normally have a scrub time that can be configured in the BIOS, which also adds a regular verification of the entire RAM at regular intervals, just in case something goes wrong in some rarely used part of the memory.
  - IYasha 4y ago
    Unfortunately, background scrubbing significantly increases power consumption and impacts performance as well.

H8crilA 4y ago

This problem has no ultimate solution. I've seen all components flip bits, CPUs, networking cards, RAM, most often you just can't know for sure what did it. You can remedy it a bit (like with ECC), but ultimately there will always be corruption if you process hundreds of petabytes of data. Get used to it, your computer executes an instruction with a probability extremely close to 1, but not equal to 1. Deep in the archives of a well known tech company is a very well documented case of a bit flip that caused the wrong function to be executed in a C++ v-table. The big oof was that this function was the equivalent of an SQL "drop table", and just happened to be 32 bytes off of a very benign function that did something like stat(). Really funny stuff once the crisis is over :)

water8 4y ago
[dead]
don-code 4y ago
Is this something that's documented publicly? I'd love to read more.
dale_glass 4y ago
ECC isn't a terribly complicated technology, and can be used in all those cases. In limited cases, a checksum is good enough. If you checksum outgoing data, and verify it on reception, then it being corrupted in transit whether on the network card or the cable can be detected and transparently compensated for. Really, we can do much better than to "get used to it".
- H8crilA 4y ago
  You are under the impression that CPUs and other chips always perform the same instructions as are written in the code, and only RAM can flip bits because DRAM is DRAM :) It can (and should! whenever possible) be improved, not fixed. There's always that pesky gamma that can hit a specific transistor, even if it is deep underground. Gamma cannot be fully stopped. At certain scales data corruption becomes directly measurable. And yes, corruption levels vary between pieces of hardware.

baeaz 4y ago

[flagged]

jeroenhd 4y ago

It's good that with DDR5 consumer memory will get some super basic ECC on die, so hopefully the next generation of memory will make the problematic sticks more obvious (or prevent damage in the very least). ECC won't save you from memory corruption, but it'll save your data at least. Personally, I would've just checksummed the individual failing files rather than the disk image and only back up the bad files separately. There are all kinds of ways for a disk image to fail and I wouldn't spend a second longer on it than absolutely necessary. The whole memtest permutation setup also would've been too much work for. E, I would just declare the motherboard faulty when two sticks that otherwise pass the test fail in specific configurations. A new motherboard is cheaper than super specific RAM sticks.

ksec 4y ago

This is especially true for NAS. And why you need BTRFS or preferably ZFS. Unfortunately none of the consumer NAS offers ZFS, and BTRFS is still not a default option. Neither Synology or Qnap seems to care.

aborsy 4y ago
Synology offers ECC in 2023 consumer modes such as 923. Still the experience that synology btrfs provides is nowhere as good as ZFS (due to a lot of limitations).
- deleted 4y ago
  [deleted]
- metadat 4y ago
  I bought a 6 disk Synology a few months ago and it came with ECC by default. I did a cursory web search about this just now and ECC support appears to be the norm for 22 (as in the year 2022) model revisions and newer (thankfully!).
  - layer8 4y ago
    It’s because they use AMD CPUs now (already in the 21 models). The trade-off is that those CPUs have worse hardware codec support than the Intel ones they previously used, if you want to do video transcoding.
    - wil421 4y ago
      Intel sold i3s for years that had ECC unlocked. I run a i3-8100 on an x11 mobo in my TrueNAS box.
gmokki 4y ago
I had RAM go bad after running 18T (5 HDDs in raid1) btrfs system in closet for years. Btrfs of course noticed it and fixed most of them automatically when some of blocks were corrupt. But eventually the system failed: the tree that contains the checksums for all the other trees corrupted itself on both copies of one node. Fixed the HW problem and then had to use hex editor to set the checksum manually to correct value (I modified the kernel to print the expected value). Now the system has been again stable for 3 years.

encryptluks2 4y ago

I watched the full video. It was long but very informative. The humor at times made up for the length and the presenter showed a lot of deep knowledge that most people won't have. My biggest gripe is that they just didn't try replacing the RAM sticks in the first place. I get that they wanted to do a root cause analysis, but geez the time and patience they had to do all those memory tests. No wonder they did a video about it, cause otherwise that lost time would have been painful. I was baffled as well that dd and ddrescue work differently in how they utilize the RAM. Caught me offguard. Onto the discussion of ECC RAM. In a perfect world, all memory would be ECC... but try finding some high performance 16GB sticks of ECC DDR4 RAM like what you'll see on gaming computers. I don't even think they make anything comparable in terms of speed and definitely not costs. I guess you don't really know that you needed ECC until it's too late.

morelikeborelax 4y ago
> " I guess you don't really know that you needed ECC until it's too late." I spent many years on hardware consultation and was amazed at the all the times I had to explain it was just a what if insurance like any other things their business was mitigating against. Sometimes they'd even decided they needed to save costs in non-ecc ram when it was $4 a gb in difference, or (during the FB-DIMM era) there wasn't even an option to avoid it. Never really understood the resistance towards it. Maybe the lack of evidence before the Google study and people thinking RAM manufacturers were trying to rip them off or something. The "never had a problem so why would I need" it attitude with no way to know if an issue was caused by a bit flip was most baffling.
simoncion 4y ago
> ...but try finding some high performance 16GB sticks of ECC DDR4 RAM like what you'll see on gaming computers. Here ya go: https://nemixram.com/16gb-ddr4-3200-pc4-25600-ecc-udimm-2rx8-memory-for-server-workstation-by-nemix-ram/ https://nemixram.com/16gb-ddr4-3200-pc4-25600-ecc-udimm-2rx8... It doesn't have pretty lights on it, but it does seem to be in the same speed class that gets called "gaming RAM" by a _whole_ bunch of retailers.
- 4y ago

chunk_waffle 4y ago

For anyone curious (as I was once and looked into it) Dell and Lenovo both ship Laptops with Xeon's and ECC memory though they are very expensive.

thekombustor 4y ago
Yep, you can get it on the higher end trims of the P-series Thinkpads.
aortega 4y ago
I have a lenovo P52, it came with 16 GB ECC. The dimm is not super-expensive, but it's hard to find, and only comes in half the size of regular dimms. I replaced it with non-ECC memory and never had an issue. In fact you can combine ECC and non-ECC memory without any problem.

lizardactivist 4y ago

I'm curious what the actual manufacturing costs for ECC DRAM is compared to regular DRAM. Is it considerably more expensive, or just the usual over-charge because it's better?

jeffbee 4y ago
It’s exactly 1/8th more.
wil421 4y ago
It has another error correcting chip on DIMM, 9 total instead of 8. Don’t be fooled by on-die ecc marketing that’s come out recently, you need the 9th chip for parity. https://community.fs.com/blog/ecc-vs-non-ecc-memory-which-one-is-better.html https://community.fs.com/blog/ecc-vs-non-ecc-memory-which-on...

deleted 4y ago

[deleted]

IYasha 4y ago

It would be so nice if DD and DDRescue did calculate hashes while copying.

tibbydudeza 4y ago

Got a Gen 10 HPE Microserver for my NAS run some AMD dual core SoC - factory fitted with ECC memory running Unraid. Think it mysteriously crashed once or twice in the 4 years I had it and the HP diagnostic light came on.

tibbydudeza 4y ago

Afaik ECC memory is slower than normal memory, so it does not impress the folks who base their purchase decisions on benchmark scores rather than utility and best bang for the buck.

mrlonglong 4y ago

We've got a server that keeps rebooting due to a bad ECC DIMM chip. I thought the whole point of ECC was to keep the server going until we can replace the DIMM?

AaronFriel 4y ago
ECC can typically correct 1 bit errors and usually detect (and fault) on 2 bit errors. Your server is faulting and preventing itself from corrupting data.
sgtnoodle 4y ago
It sounds like it's working as intended. Redundancy doesn't help you with reliability once your system is in a degraded state. Running a server with a known bad ram stick is like driving a car on run-flat tires. It will probably get you where you need to go, but you really should be driving to a tire shop.
- mrlonglong 4y ago
  Ok thanks I understand things better now. I'll pester my boss to get new DIMM asap.
exikyut 4y ago
IIUC it's the kernel EDAC subsystem rebooting the server for you. If you turn it off you'll get non-ECC behaviour, but in this case you'll also get really interesting system issues because as the sibling comments have said your RAM is actually faulty :P
to11mtm 4y ago
ECC Corrects 1 bit errors and detects 2 bit errors. It does not handle all hardware failures however. An actual bad chip may fall into a different category, I think that's when technologies like Chipkill[0] might come into play. [0] - https://en.wikipedia.org/wiki/Chipkill https://en.wikipedia.org/wiki/Chipkill

aortega 4y ago

I have 128 GB of non-ecc memory in my notebook, never detected a single error, and has been on 24/7 for more than 4 years. Unless you live over 4000 meters over the sea level, like to compile while flying or live close to an unshielded nuclear reactor, you don't need ECC. And most memory problems you can fix by better cooling, and better shielding.

LeoPanthera 4y ago
> I have 128 GB of non-ecc memory in my notebook, never detected a single error How would you know?
- aortega 4y ago
  I know that I never detected them. Also it never crashed nor malfunctioned in any way (thanks in part to Linux), so any undetected error didn't caused any problem.
  - russdill 4y ago
    There's been plenty of studies they measure bit flips in memory. If you really wanted to measure, you'd find them without much difficulty. Yes though, when you have gobs of memory, you're unlikely to see the effects of bit flips. That's not the same at them not occuring.
mech4bg 4y ago
My 4 year old Mac Mini had been showing some flakiness and finally I got an error about "hash mismatch detected on volume" - ran memtest and sure enough the RAM had gone bad. I rely on this machine for all my media, backups, etc, goodness knows what data I may have lost while the RAM was bad. It's a moderately common problem, found plenty of people with the same issue.
PragmaticPulp 4y ago
I’ve had 128GB of ECC memory with error reporting. Never logged a bit flip as far as I can tell. Memory errors are heavily concentrated to bad modules. They’re not evenly distributed across all RAM. Always test the stability of your memory when doing a new build.

meindnoch 4y ago

Data written but not read back may as well go straight to /dev/null. Always read back and verify integrity.

ritcgab 4y ago

Another important perspective of ECC memory is whether your machine can properly respond to the interrupt. ECC without reporting and handling is not a working ECC. I have been using Ryzen machines with AsRock motherboards and the Linux EDAC driver reports errors correctly (verified by overclocking the memory sticks to the edge). On the other hand, I owned a Dell Precision mobile workstation with Xeon W-11955M processor and ECC SODIMM sticks. However, since its EDAC driver never really lands in the mainline kernel, I won't get any ECC reports, so the corresponding process that has a bit flipped will not get killed.

justinclift 4y ago

Near the end, the video mentions his (old) computer "apparently can't take 'high density' memory". Then the author goes through a process of randomly buying memory and hoping it works. Anyone know what that "high density" memory problem is about? Maybe it's a misunderstanding of memory channels and ranks?

to11mtm 4y ago
Not sure if it's what he means, but in my experience 'High density' refers to less chips for a given capacity. When I used to sling PC hardware, as memory densities increased, you would run into things like 'This Motherboard will only take a 256MB module if it has 8 chips on both sides (16 chips total), it will not take a 256MB module with 8 chips on one side (8 chips total)' Depending on the board, it might only register part of the capacity and be stable, might register part of the capacity and be unstable, or just not boot at all.
- justinclift 4y ago
  Interesting. Have heard mention of similar over the years, but never seen it in person myself. Sounds strange, as - without knowing any better - I'd expect the memory interface to be agnostic to the chips implementing it. eg "address 0x11111111" on 'high density' ram would be seen exactly the same as "address 0x11111111" on other ram The mind boggles. :)