4 ms·

> Why ECC Memory Is So Important Except that it's not for many use cases. It's great for servers but for people on their personal and/or work computer, it's s

> Why ECC Memory Is So Important Except that it's not for many use cases. It's great for servers but for people on their personal and/or work computer, it's simply not that useful. Seriously: which percentage of developers have ECC on their development machine(s)? As developers we live in a world of SSH, cryptographic hashes, checksums everywhere, Git repositories (that is a big one), Merkle trees, digital signatures, reproducible builds (which are gaining traction), etc. Heck, I'm torrenting the latest Debian or Devuan .iso image. My torrent client is using every known trick under the sun to make sure that should anything go wrong, the broken data shall be discarded and re-downloaded. Download is done, I dd the image to some installation medium. I can then verify its checksum matches the official one. A bit flip didn't slip by unnoticed. All the music I carefully ripped from my audio CDs? They're all cross-checked with an online DB of known bit-perfect rips. There's an accompanying file containing each song's hash and I can verify at anytime that all my files are 100% correct. But really most of all I live in a world of Git repositories. My entire Emacs config is versioned under Git (I know YMMV but I like it that way). Some people version under Git their entire user dir. Tell me how my lack of ECC is going to really make life miserable here? I have nothing against ECC... But if I want to upgrade my AMD 3700X to a 7700X, apparently I cannot get ECC. And that's totally fine: I certainly won't discard the 7700X because I cannot get ECC for it. And if anything looks suspicious, running Memtest is the first thing you should do. I've had bad RAM at times. I'm still there.

rwmj 4y ago
But if I open up an editor and write a document then that could be corrupted in RAM and then the corrupted data saved to disk. The document is likely to be more important than some ripped CDs or a git repo that I can download again. The CPU, RAM and mobo manufacturers need to get together and make ECC RAM mandatory. It's absurd that we have machines with gigabytes of storage using microscopic (nanoscopic??) capacitors that doesn't have this basic protection. And honestly this should have happened years ago. (Edit: And before anyone says DDR5 is ECC by default, that's not quite true although the difference is a bit subtle: https://en.wikipedia.org/wiki/DDR5_SDRAM#DIMMs_versus_memory_chips https://en.wikipedia.org/wiki/DDR5_SDRAM#DIMMs_versus_memory...)
- ilyt 4y ago
  >But if I open up an editor and write a document then that could be corrupted in RAM and then the corrupted data saved to disk. The document is likely to be more important than some ripped CDs or a git repo that I can download again. Devils advocate: if it is just some bits in character flipped it's entirely recoverable while flipping some bits in compressed stream for video would corrupt more.
  - rwmj 4y ago
    Agreed. One problem with RAM errors (which I've actually experienced) is they are insidious. You probably won't notice them immediately so the error can be propagated, and they're very very difficult to diagnose if you do notice them. Back in the 90s we had a database server which had a stuck bit in memory normally mapped to the page cache. This caused sectors to be written to the backing software RAID which couldn't be read back in (because I think some checksum was corrupted when written and then failed when read back). It took an absolute age to diagnose this. I think I only worked it out by eliminating everything else.
craftkiller 4y ago
> Seriously: which percentage of developers have ECC on their development machine(s)? Every single one of my non-mini computers uses ECC ram except for my laptop. If someone would release a framework laptop motherboard that supports ECC ram (and preferably risc-v) I'd finally be able to close to reliability gap. It blows my mind that we say "Well sure, we COULD make infallible ram, but that would cost a tiny bit extra so instead lets just hope nothing bad happens." That's right up there with not wearing a seat-belt because I haven't needed one yet.
- Brian_K_White 4y ago
  "If someone would release a framework laptop motherboard that supports ECC ram..." This. If I could, I would. The only reason my laptop doesn't have ecc is because the manufacturer doesn't offer the option in any macines I otherwise want. That comment was very misguided in trying to suggest that there is any valid excuse to tolerate unreliable execution hardware. git and ssh and md5sums do not mean that it's ok if your very brain can't be trusted to deliver data from one part to another within itself, or spit back the same data that was put in a cell. Everything else is built upon that!
- digitallyfree 4y ago
  I develop on VMs on my server (with ECC and RAIDZ) which i access using SSH or VDI protocols. I would love to have ECC on all my machines but that isn't feasible until the status quo changes, so I stick with the remote approach. To me that's an acceptable tradeoff as the non-ECC desktops/laptops are just used as dumb terminals while the real work happens on the reliable server. I can't speak for ECC at the moment but ZFS has definitely saved me from data corruption that would have been left to manifset otherwise.
rhn_mk1 4y ago
> which percentage of developers have ECC on their development machine(s)? That question won't help you evaluate the demand for ECC simply because the supply is strangled. Those who want it have to make compromises to get ECC: get a Xeon, or get one of few AMD motherboards with matching CPUs and overpriced RAM without a guarantee that it will end up working.
- simoncion 4y ago
  > ...get one of few AMD motherboards with matching CPUs and overpriced RAM... I mean, if you want a _guarantee_, then sure, get one of those certified-for-ECC motherboards. But, like, as far as I know, going _at least_ far back as the Phenom II (released in 2008) AMD desktop processors have always supported ECC RAM. And -as far as I know- ASUS motherboards for said processors have always supported dropping in ECC RAM (and Linux and memtest and friends have always agreed that ECC was enabled and functioning in such a system). Source: Personal experience with Phenom II, Threadripper, and Ryzen 5 CPUs and ASUS motherboards, and looking-from-a-distance at the rest of the AMD CPUs between the Phenom and the Ryzen 5.
tasubotadas 4y ago
When people complain about these random OS crashes and freezes it's usually RAM corruption at fault. Per 1gb of RAM, you can expect to see 266 bit errors per month[1-2] if you are using your PC 16h per day. Multiply that by 64GB or 128GB of RAM and it's crazy to think that you won't run into any of the stability issues. [1-2] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf https://static.googleusercontent.com/media/research.google.c... [1-2] https://en.wikipedia.org/wiki/ECC_memory#Research https://en.wikipedia.org/wiki/ECC_memory#Research
- ubercow13 4y ago
  >266 bit errors per month That seems like an overestimate. How can memtest ever pass on non-ECC RAM if errors are that frequent?
  - moloch-hai 4y ago
    Because memtest only looks at values it wrote out very recently, before they have had a chance to flip. Memtest is looking for reliable failures, not evanescent one-off events.
    - deleted 4y ago
      [deleted]
    - willis936 4y ago
      SDRAM is continuously refreshing all cells. How long ago data was written doesn't make a big difference (aside from the case where you're reading data immediately after writing or reading that data).
      - moloch-hai 4y ago
        This does not, of course, make any sense. Any given memory cell will be read once per, say, millisecond, and the value written back. Once it has flipped once, the wrong value will then be written back, and after that the wrong value is read back out and rewritten again, indefinitely. Errors are sticky, and accumulate. (Flipping back again is negligible unlikely.) With ECC in the refresh path, such an error could be corrected and the right value would be overwritten over top of the bad one. Then errors would not accumulate, but would instead be "scrubbed". Mainframe machines scrub their RAM. Disks too.
justsomehnguy 4y ago
And condoms are only 99.8% effective, so let's not use it? You can always pull out in time or verify checksu^W^W Plan B? > But really most of all I live in a world of Git repositories That's great for you, but 99% don't even know what Git is, just like checksums, cryptographic hashes, Merkle trees, digital signatures and reproducible builds. You just miss the one imortant thing: ECC isn't that helpful where you have the means to check and verify the data. ECC is the only way to at least know what something is happening with data when there is no way to check. To give you a slight idea I would tell you an anecdote from my L1 life almost two decades ago: I visited a client who claimed what the PC was working erratically and constantly threw weird error messages. Welp, the usual deal, just some ugly software or a virus. In the first 3 minutes I got like 5 errors about failing to load a .dll from C:\WINDOWS\SYSTEM33\USER32.DLL, nothing unusual, just need to.. WAIT. Why system33? Stupid virus masking for a well-known folder? Doubt. So I go to C:\Windows and I see: System32 System33 System34 If you are a smart fellow you probably already understood what that was a bit-flip error which somehow managed to be in the in-memory copy of MFT of the system drive. And this is the only reason the user noticed it - because sometimes programs wouldn't start and sometimes there was weird error messages. If that error was in the data area of some program - it wouldn't be discovered at all.
- deleted 4y ago
  [deleted]
- dmitrybrant 4y ago
  Hang on... to go from "System32" to "System33" is one bit flip, but to go from "System33" to "System34" is three bit flips at once. Doesn't that seem astronomically more likely to be a malicious program than bad RAM?
  - justsomehnguy 4y ago
    It's astronomically more likely to be a manufacturing defect what affected multiple cells than a malicious program what ... wrote to System34? And how the replacement of the DRAM module 'cured' that malicious program?
dale_glass 4y ago
> But really most of all I live in a world of Git repositories. My entire Emacs config is versioned under Git (I know YMMV but I like it that way). Some people version under Git their entire user dir. > Tell me how my lack of ECC is going to really make life miserable here? Git will break if RAM is bad just like anything else. That it checksums everything won't save you from checking in corrupt data, the filesystem itself being corrupt, or some internal git structure becoming corrupt. Losing your repo because something in it was written wrong is very much a possibility. Having multiple machines involved helps, but it's not a complete fix, because the possibility exists of something damaged being transmitted from a broken machine to a good one, ensuring there's no good copy anywhere. There's really nothing software can do to operate correctly with bad RAM all of the time. Instructions for the software are in RAM. The OS that the software expects to behave right is in RAM. Various buffers used for disk access and networking are in RAM. An application like git assumes all of that is performing correctly, and can't compensate for every possible malfunction that could happen.
- everybodyknows 4y ago
  How trustworthy is git-fsck for detecting random-bit corruption?
  - dale_glass 4y ago
    Depends on how you look at it. For actual verification, unless there's a bug my understanding is that very trustworthy. But by that point it's already too late. Okay, you know something is broken, but that won't give your good data back. But that only tells you that Git data is intact and that all the hashes match. If git got a corrupted file to start with, then correctly hashed it, everything will verify 100% and still be broken.
kevingadd 4y ago
I went out of my way for ECC because losing files can mean losing days of work. With how much a sweng gets paid it's worth the ECC premium if it saves me a few days of work over the life of the machine. Recently I discovered that one of my SSDs was quietly failing without setting off any warnings; doing a chkdsk showed that some files had already gotten corrupted. One of them was my backblaze backup index! Even though I have automated backups (backblaze + macrium backups to a NAS), recovering files from them is non-trivial. If I were to lose work to non-ECC ram who knows how long it would take me to reconstruct a known-good work environment and file set. Imagine if you're working on something huge and hard to validate like neural net weights where corruption can occur silently and be hard to detect after it's happened?
- newZWhoDis 4y ago
  My favorite is when data silently corrupts, and then is happily propagated to your off-site recovery :/
  - kevingadd 4y ago
    Yeah, I don't know when the drive started failing, so in practice my backups are probably all screwed too. My only option would be to get a spare drive, restore a backup to it, then do a block level diff of the current and backup volumes and try to figure out whether any of the differences are file corruption.
    - Filligree 4y ago
      Probably I'm saying nothing you haven't thought of, but ZFS is great for preventing this. If you also pair it with ECC, then you've eliminated most ways to cause corruption.
leguminous 4y ago
Checksums don't save you from memory corruption. If your data gets corrupted in memory, you will just end up checksumming and committing bad data. Or your checksum could get corrupted, and you commit a checksum that doesn't match your data. Checksums are more useful for safeguarding against disk or network corruption (although you shouldn't have network corruption issues over TLS or SSH). Apparently Ryzen 7000 cpus can use ECC. I've heard reports that AMD needs to release an AGESA update, though, and ECC DDR5 memory availability is terrible. I'm hopeful that the situation will improve, because I also want to update my desktop. I've been using ECC memory since losing a filesystem on a desktop when a DIMM went bad.

rwmj 4y ago

But if I open up an editor and write a document then that could be corrupted in RAM and then the corrupted data saved to disk. The document is likely to be more important than some ripped CDs or a git repo that I can download again. The CPU, RAM and mobo manufacturers need to get together and make ECC RAM mandatory. It's absurd that we have machines with gigabytes of storage using microscopic (nanoscopic??) capacitors that doesn't have this basic protection. And honestly this should have happened years ago. (Edit: And before anyone says DDR5 is ECC by default, that's not quite true although the difference is a bit subtle: https://en.wikipedia.org/wiki/DDR5_SDRAM#DIMMs_versus_memory_chips https://en.wikipedia.org/wiki/DDR5_SDRAM#DIMMs_versus_memory...)

ilyt 4y ago
>But if I open up an editor and write a document then that could be corrupted in RAM and then the corrupted data saved to disk. The document is likely to be more important than some ripped CDs or a git repo that I can download again. Devils advocate: if it is just some bits in character flipped it's entirely recoverable while flipping some bits in compressed stream for video would corrupt more.
- rwmj 4y ago
  Agreed. One problem with RAM errors (which I've actually experienced) is they are insidious. You probably won't notice them immediately so the error can be propagated, and they're very very difficult to diagnose if you do notice them. Back in the 90s we had a database server which had a stuck bit in memory normally mapped to the page cache. This caused sectors to be written to the backing software RAID which couldn't be read back in (because I think some checksum was corrupted when written and then failed when read back). It took an absolute age to diagnose this. I think I only worked it out by eliminating everything else.

craftkiller 4y ago

> Seriously: which percentage of developers have ECC on their development machine(s)? Every single one of my non-mini computers uses ECC ram except for my laptop. If someone would release a framework laptop motherboard that supports ECC ram (and preferably risc-v) I'd finally be able to close to reliability gap. It blows my mind that we say "Well sure, we COULD make infallible ram, but that would cost a tiny bit extra so instead lets just hope nothing bad happens." That's right up there with not wearing a seat-belt because I haven't needed one yet.

Brian_K_White 4y ago
"If someone would release a framework laptop motherboard that supports ECC ram..." This. If I could, I would. The only reason my laptop doesn't have ecc is because the manufacturer doesn't offer the option in any macines I otherwise want. That comment was very misguided in trying to suggest that there is any valid excuse to tolerate unreliable execution hardware. git and ssh and md5sums do not mean that it's ok if your very brain can't be trusted to deliver data from one part to another within itself, or spit back the same data that was put in a cell. Everything else is built upon that!
digitallyfree 4y ago
I develop on VMs on my server (with ECC and RAIDZ) which i access using SSH or VDI protocols. I would love to have ECC on all my machines but that isn't feasible until the status quo changes, so I stick with the remote approach. To me that's an acceptable tradeoff as the non-ECC desktops/laptops are just used as dumb terminals while the real work happens on the reliable server. I can't speak for ECC at the moment but ZFS has definitely saved me from data corruption that would have been left to manifset otherwise.

rhn_mk1 4y ago

> which percentage of developers have ECC on their development machine(s)? That question won't help you evaluate the demand for ECC simply because the supply is strangled. Those who want it have to make compromises to get ECC: get a Xeon, or get one of few AMD motherboards with matching CPUs and overpriced RAM without a guarantee that it will end up working.

simoncion 4y ago
> ...get one of few AMD motherboards with matching CPUs and overpriced RAM... I mean, if you want a _guarantee_, then sure, get one of those certified-for-ECC motherboards. But, like, as far as I know, going _at least_ far back as the Phenom II (released in 2008) AMD desktop processors have always supported ECC RAM. And -as far as I know- ASUS motherboards for said processors have always supported dropping in ECC RAM (and Linux and memtest and friends have always agreed that ECC was enabled and functioning in such a system). Source: Personal experience with Phenom II, Threadripper, and Ryzen 5 CPUs and ASUS motherboards, and looking-from-a-distance at the rest of the AMD CPUs between the Phenom and the Ryzen 5.

tasubotadas 4y ago

When people complain about these random OS crashes and freezes it's usually RAM corruption at fault. Per 1gb of RAM, you can expect to see 266 bit errors per month[1-2] if you are using your PC 16h per day. Multiply that by 64GB or 128GB of RAM and it's crazy to think that you won't run into any of the stability issues. [1-2] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf https://static.googleusercontent.com/media/research.google.c... [1-2] https://en.wikipedia.org/wiki/ECC_memory#Research https://en.wikipedia.org/wiki/ECC_memory#Research

ubercow13 4y ago
>266 bit errors per month That seems like an overestimate. How can memtest ever pass on non-ECC RAM if errors are that frequent?
- moloch-hai 4y ago
  Because memtest only looks at values it wrote out very recently, before they have had a chance to flip. Memtest is looking for reliable failures, not evanescent one-off events.
  - deleted 4y ago
    [deleted]
  - willis936 4y ago
    SDRAM is continuously refreshing all cells. How long ago data was written doesn't make a big difference (aside from the case where you're reading data immediately after writing or reading that data).
    - moloch-hai 4y ago
      This does not, of course, make any sense. Any given memory cell will be read once per, say, millisecond, and the value written back. Once it has flipped once, the wrong value will then be written back, and after that the wrong value is read back out and rewritten again, indefinitely. Errors are sticky, and accumulate. (Flipping back again is negligible unlikely.) With ECC in the refresh path, such an error could be corrected and the right value would be overwritten over top of the bad one. Then errors would not accumulate, but would instead be "scrubbed". Mainframe machines scrub their RAM. Disks too.

justsomehnguy 4y ago

And condoms are only 99.8% effective, so let's not use it? You can always pull out in time or verify checksu^W^W Plan B? > But really most of all I live in a world of Git repositories That's great for you, but 99% don't even know what Git is, just like checksums, cryptographic hashes, Merkle trees, digital signatures and reproducible builds. You just miss the one imortant thing: ECC isn't that helpful where you have the means to check and verify the data. ECC is the only way to at least know what something is happening with data when there is no way to check. To give you a slight idea I would tell you an anecdote from my L1 life almost two decades ago: I visited a client who claimed what the PC was working erratically and constantly threw weird error messages. Welp, the usual deal, just some ugly software or a virus. In the first 3 minutes I got like 5 errors about failing to load a .dll from C:\WINDOWS\SYSTEM33\USER32.DLL, nothing unusual, just need to.. WAIT. Why system33? Stupid virus masking for a well-known folder? Doubt. So I go to C:\Windows and I see: System32 System33 System34 If you are a smart fellow you probably already understood what that was a bit-flip error which somehow managed to be in the in-memory copy of MFT of the system drive. And this is the only reason the user noticed it - because sometimes programs wouldn't start and sometimes there was weird error messages. If that error was in the data area of some program - it wouldn't be discovered at all.

deleted 4y ago
[deleted]
dmitrybrant 4y ago
Hang on... to go from "System32" to "System33" is one bit flip, but to go from "System33" to "System34" is three bit flips at once. Doesn't that seem astronomically more likely to be a malicious program than bad RAM?
- justsomehnguy 4y ago
  It's astronomically more likely to be a manufacturing defect what affected multiple cells than a malicious program what ... wrote to System34? And how the replacement of the DRAM module 'cured' that malicious program?

dale_glass 4y ago

> But really most of all I live in a world of Git repositories. My entire Emacs config is versioned under Git (I know YMMV but I like it that way). Some people version under Git their entire user dir. > Tell me how my lack of ECC is going to really make life miserable here? Git will break if RAM is bad just like anything else. That it checksums everything won't save you from checking in corrupt data, the filesystem itself being corrupt, or some internal git structure becoming corrupt. Losing your repo because something in it was written wrong is very much a possibility. Having multiple machines involved helps, but it's not a complete fix, because the possibility exists of something damaged being transmitted from a broken machine to a good one, ensuring there's no good copy anywhere. There's really nothing software can do to operate correctly with bad RAM all of the time. Instructions for the software are in RAM. The OS that the software expects to behave right is in RAM. Various buffers used for disk access and networking are in RAM. An application like git assumes all of that is performing correctly, and can't compensate for every possible malfunction that could happen.

everybodyknows 4y ago
How trustworthy is git-fsck for detecting random-bit corruption?
- dale_glass 4y ago
  Depends on how you look at it. For actual verification, unless there's a bug my understanding is that very trustworthy. But by that point it's already too late. Okay, you know something is broken, but that won't give your good data back. But that only tells you that Git data is intact and that all the hashes match. If git got a corrupted file to start with, then correctly hashed it, everything will verify 100% and still be broken.

kevingadd 4y ago

I went out of my way for ECC because losing files can mean losing days of work. With how much a sweng gets paid it's worth the ECC premium if it saves me a few days of work over the life of the machine. Recently I discovered that one of my SSDs was quietly failing without setting off any warnings; doing a chkdsk showed that some files had already gotten corrupted. One of them was my backblaze backup index! Even though I have automated backups (backblaze + macrium backups to a NAS), recovering files from them is non-trivial. If I were to lose work to non-ECC ram who knows how long it would take me to reconstruct a known-good work environment and file set. Imagine if you're working on something huge and hard to validate like neural net weights where corruption can occur silently and be hard to detect after it's happened?

newZWhoDis 4y ago
My favorite is when data silently corrupts, and then is happily propagated to your off-site recovery :/
- kevingadd 4y ago
  Yeah, I don't know when the drive started failing, so in practice my backups are probably all screwed too. My only option would be to get a spare drive, restore a backup to it, then do a block level diff of the current and backup volumes and try to figure out whether any of the differences are file corruption.
  - Filligree 4y ago
    Probably I'm saying nothing you haven't thought of, but ZFS is great for preventing this. If you also pair it with ECC, then you've eliminated most ways to cause corruption.

leguminous 4y ago

Checksums don't save you from memory corruption. If your data gets corrupted in memory, you will just end up checksumming and committing bad data. Or your checksum could get corrupted, and you commit a checksum that doesn't match your data. Checksums are more useful for safeguarding against disk or network corruption (although you shouldn't have network corruption issues over TLS or SSH). Apparently Ryzen 7000 cpus can use ECC. I've heard reports that AMD needs to release an AGESA update, though, and ECC DDR5 memory availability is terrible. I'm hopeful that the situation will improve, because I also want to update my desktop. I've been using ECC memory since losing a filesystem on a desktop when a DIMM went bad.