7 ms·

The numbers from this Google SIGMETRICS09 paper are my usual benchmark for thinking about ECC DIMMs: https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf

The numbers from this Google SIGMETRICS09 paper are my usual benchmark for thinking about ECC DIMMs: https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf Their metric of “25,000 to 70,000 errors per billion device hours per megabit” is a bit hard to grapple with. If you assume each error is a single bit then that’s 20 to 50 bytes per GB DIMM per month, or one bit per GB every two hours.

p-e-w 3y ago
> or one bit per GB every two hours How is that possible? Wouldn't such an error frequency lead to user-observable problems all the time? For example, in the average statically compiled codebase, flipping a single bit in source code being edited (with the code then being saved to disk) will make it fail to compile with high probability, which would be noticed immediately. Yet I've never encountered this situation in practice, nor heard of anyone else encountering it, and like most people I don't even use ECC RAM. That seems incongruent with the figure quoted above.
- viraptor 3y ago
  Yeah, that's not realistic. This is as much code as goes through a CI I'm working with many times a day. I'd constantly see errors on an unknown variable if this was a real rate.
  - teekert 3y ago
    Or you are using ECC RAM?
  - bjornsing 3y ago
    Aren’t you forgetting the time aspect? If you’re keeping that code in RAM for just a few seconds at most then the probability of a random bit flip is negligible, even if the hourly rate is high. But I agree this sounds unrealistic.
    - codeflo 3y ago
      Presumably, most of the code doesn't change often, and consequently, should be kept in the filesystem cache and never actually read from disk. So the numbers could apply, I think.
  - Gordonjcp 3y ago
    It's about one second of video at "editing quality" ;-)
- jhugo 3y ago
  One bit per GB of RAM, not per GB of processed source code (most of your RAM is likely empty most of the time, or used for stuff where a bit flip is less likely to be noticed). However, it still seems like a high estimate to me.
  - pmontra 3y ago
    I have 32 GB of RAM. If I only use 1 GB of it it's still one bit flip in the RAM I'm using.
    - jhugo 3y ago
      Eh? The "cosmic ray" (or similar upset) is roughly equally likely to affect any bit in the 32GB, so the chance of it appearing in the 1GB that you're using right now is 32 times smaller than the chance of it happening anywhere in your RAM.
  - zaphirplane 3y ago
    Operating systems keep files in ram In addition to memory allocation till they need the ram, so I think it should be mostly full all the time
    - mrspuratic 3y ago
      Unused RAM ~= wasted RAM... Recently I had to troubleshoot a linux process that started segfaulting randomly. Having dealt with non-ECC bit flips before I took a copy of the binary (i.e. from the filesystem cache), dropped the caches, and compared and disassembled the binaries: a single bit flip resulted in an illegal instruction. Cost of investigating issues >>> cost of ECC
    - jhugo 3y ago
      But bit flips may not be noticed any time soon in the disk cache, and when they are later noticed, may be wrongly attributed to disk failure. It's a different failure mode than a bit flip in some source code that you are about to compile (which is far less likely to happen, due to that source code being a much "smaller target").
- eklitzke 3y ago
  This paper is ~15 years old. It was published in 2009, but the data was collected 2006-2008. As densities for things like RAM increase, error rates have to decrease commensurately to hide errors from end users. Additionally, Google runs hardware in their datacenters in a much more pathological way than most consumers will (e.g. they are more likely to run hardware much hotter than typical consumers). The point is that the figures in the paper are an interesting starting point for discussion, but they're not necessarily applicable to your laptop.
  - wongarsu 3y ago
    > they are more likely to run hardware much hotter than typical consumers I'd expect laptops and phones with their limited cooling capacity to run their hardware much hotter than datacenters with their expensive HVAC systems
  - p-e-w 3y ago
    I'm not convinced by this argument. In practice, errors on consumer hardware seem to be several orders of magnitude less common than implied by the paper. If they weren't, we'd be hearing anecdotes of one-character diffs appearing out of nowhere all the time, among many other issues. It's hard to see how this could be explained away with the slight increase in RAM densities since 2008, or different temperatures in data centers (plenty of individuals run their systems crazy hot, e.g. when the tower is stuffed with dust after years without cleaning).
    - alpaca128 3y ago
      Run any consumer device for more than 4 weeks and it'll slowly accumulate all kinds of weird bugs. I've seen smartphones unable to receive SMS because they simply ran too long without reboot, a Windows PC where selecting files wouldn't work anymore etc. > we'd be hearing anecdotes of one-character diffs appearing out of nowhere all the time On consumer devices it might simply be a small "attack" surface. A server running 24/7 serving countless requests would probably see more issues than a laptop running idle 95% of the time and loading a website every few minutes. Also a bit flip isn't guaranteed to cause a noticeable bug - and if it does it might not stand out from the usual noise of janky software oddities.
- theginger 3y ago
  They observed the majority of dimms had no errors. They had ecc ram so it's possible for a dimm that is going bad to carry on in service regularly producing correctable errors. If they were in your laptop you would start getting problems quite quickly and end up replacing.
- OskarS 3y ago
  If it's this common, it should be straight-forward to test as well, right? Allocate 1 gig of memory, do whatever mmap call you need to make sure it's not paged out, fill it with deterministically random stuff, then every hour check the contents and refill it. Keep running for a month. Seems odd to me that we have rely on some paper from 2009...
- deleted 3y ago
  [deleted]
- duffyjp 3y ago
  I'll run memtest86 for a full day when I build a system or get new ram. I've never seen an error outside of the situation where a stick has gone bad and it errors like crazy. I don't use ECC other than on my homelab server either. Tons of people do this and have no issues with bit flips or we'd be hearing about it.
  - bcrl 3y ago
    Anecdotes are not real world statistics. Most people wouldn't identify a flipped bit of memory that caused a web page to glitch in their browser as a hardware problem. They'll just write it off as general enshittification of the web.
  - sliken 3y ago
    In my experience, after handling many dozens of systems (out of 3000) with ECC errors, it's quite rare for the problem to be reproducible in memtest86. Running in production often triggers many errors per day. Our alarm threshold is over 100 correctable or 1 uncorrectable error per day. After a producing zero errors with 24 hours in memtest86, I usually see more errors when returned into production. Not sure if it's heat/cooling related, the access pattern, or maybe memtest86 tests the memory value, but not the memory address. I.e. if you fill memory with 0xdeadbeef you'll read 0xdeadbeef successfully even if the address is off. Dimms can "fail", but still have low error rates. Stories are pretty common, LinusT had one in the last year spent something like a week tracking it down, assumed it was a new kernel error. Tracking down memory errors with ECC is a painful process that involve tracking down numerous possible causes crashes. Sure bitflips are easy to ignore, but weird things do happen. Linux crashes, processes crash, suddenly things are weird, the desktop doesn't work like you expect. People assume that "the system is buggy" or it's related to patching, or some other user space error, but sometimes it's just a bitflip. Sadly there are few cases where a bit flip is obvious, but if you keep 10k files around for a decade it's pretty common to find a few corrupted. Various weird behaviors have been tracked down to memory issues. GCC for years had an error that nearly always was an internal compiler error ... triggered by a memory problem. One famous speed run of a game had something very weird happened, which was tracked to a single bit memory error. Sure can you get away with ECC, certainly. But the minimal price increase is in my opinion is quite reasonable. I'd much rather get a "single bit error corrected on dimm #3" dmesg, then occasional (or not so occasional) process, kernel, or filesystem crashes. I'm typing this from an 2015 system with a Xeon e3-1230 CPU that was cheaper than the equivalent i7, but has ECC support. Sure I spent a bit more on the motherboard and dimms, something around $120. Money well spent. I find system crashes quite disruptive, even 1 less per year is valuable to me.
- TacticalCoder 3y ago
  > How is that possible? I'd say that we live in a world of checksums, cryptographic checksums, retries, deterministic builds, distributed VCS (like Git), digital signatures, etc. which helps a lot with the very rare bitflips that do actually happen. > For example, in the average statically compiled codebase, flipping a single bit in source code being edited (with the code then being saved to disk) will make it fail to compile with high probability, which would be noticed immediately. I fully agree. If I set say "unstable" settings in my UEFI/BIOS for the RAM, I cannot compile, say, Emacs, without getting a segfault. > For example, in the average statically compiled codebase, flipping a single bit in source code being edited (with the code then being saved to disk) will make it fail to compile with high probability, which would be noticed immediately. That seems incongruent with the figure quoted above. I don't understand it either.
- FuriouslyAdrift 3y ago
  DDR5 includes on chip ECC in the standard (due to the high density). It's not the same as "full" ECC, though, as it does not have vision once the data leaves the module (on the wire, at the CPU, etc)
proto_lambda 3y ago
I think you're off by a couple orders of magnitude there. 25e3 errors/(1e9 h * 1e6 b) = 25e-12 errors/hour/bit, ~~or 4 errors/hour/terabyte.~~ Edit: nope, I got that last step wrong, it's actually 200 errors/hour/terabyte, or 1 error/gigabyte/5 hours, pretty close to what you said.
PinguTS 3y ago
I never really experienced that in real life. That is the thing we need to cope with in functional safety or some call it machine safety. There are people working with (mainly) electrical machines and by possible all measures an error cause by interference, bit flip, stuck bit, … needs be detected that the worst case of dead and injured people needs to be prevented. That are things we also need to cope with in automated/self driving cars.
- nottorp 3y ago
  > I never really experienced that in real life. Most of the time the flipped bit will be in a cat photo and go unnoticed.
pmontra 3y ago
4 bit flips per GB in a working day, in every GB of my laptop. Let's keep using powers of 2: 256 days when I'm using it it's 1024 bit flips per GB per year. Some will happen in unused memory, some in running programs. It's 1024 * N bit flips in our code editor per year, were N is how many times our editor + language servers are larger than 1 GB. Some in unused and unsaved data, some in code paths I'm not using. However some will probably crash a program because of a wrong or invalid machine code instruction or visibly alter data. Times that per the number of people using computers. Those spelling mistakes or wrong figures were not our fault after all? In a code editor people should see one character git diffs in files that are open in the editor, parked in a long unused tab (if the editor detects the change and saves it or asks to.)
oakwhiz 3y ago
I've observed that error rates are not the same for every machine - there are very bad machines, which usually have some way of triggering enough errors systematically that they end up getting someone's attention, then there are bad machines which have memory errors or are observed to report errors or crash occasionally, and then a bunch of good machines which on the surface appear to work flawlessly. Sometimes what makes a machine bad is the memory itself, other times it's the hardware or the configuration that the hardware settles on. The statistics may be skewed by the uneven distribution of "bad machines" amongst good ones, especially if you take into account hardware in the hands of end users rather than servers. It gives a new meaning to "works on my machine."

p-e-w 3y ago

> or one bit per GB every two hours How is that possible? Wouldn't such an error frequency lead to user-observable problems all the time? For example, in the average statically compiled codebase, flipping a single bit in source code being edited (with the code then being saved to disk) will make it fail to compile with high probability, which would be noticed immediately. Yet I've never encountered this situation in practice, nor heard of anyone else encountering it, and like most people I don't even use ECC RAM. That seems incongruent with the figure quoted above.

viraptor 3y ago
Yeah, that's not realistic. This is as much code as goes through a CI I'm working with many times a day. I'd constantly see errors on an unknown variable if this was a real rate.
- teekert 3y ago
  Or you are using ECC RAM?
- bjornsing 3y ago
  Aren’t you forgetting the time aspect? If you’re keeping that code in RAM for just a few seconds at most then the probability of a random bit flip is negligible, even if the hourly rate is high. But I agree this sounds unrealistic.
  - codeflo 3y ago
    Presumably, most of the code doesn't change often, and consequently, should be kept in the filesystem cache and never actually read from disk. So the numbers could apply, I think.
- Gordonjcp 3y ago
  It's about one second of video at "editing quality" ;-)
jhugo 3y ago
One bit per GB of RAM, not per GB of processed source code (most of your RAM is likely empty most of the time, or used for stuff where a bit flip is less likely to be noticed). However, it still seems like a high estimate to me.
- pmontra 3y ago
  I have 32 GB of RAM. If I only use 1 GB of it it's still one bit flip in the RAM I'm using.
  - jhugo 3y ago
    Eh? The "cosmic ray" (or similar upset) is roughly equally likely to affect any bit in the 32GB, so the chance of it appearing in the 1GB that you're using right now is 32 times smaller than the chance of it happening anywhere in your RAM.
- zaphirplane 3y ago
  Operating systems keep files in ram In addition to memory allocation till they need the ram, so I think it should be mostly full all the time
  - mrspuratic 3y ago
    Unused RAM ~= wasted RAM... Recently I had to troubleshoot a linux process that started segfaulting randomly. Having dealt with non-ECC bit flips before I took a copy of the binary (i.e. from the filesystem cache), dropped the caches, and compared and disassembled the binaries: a single bit flip resulted in an illegal instruction. Cost of investigating issues >>> cost of ECC
  - jhugo 3y ago
    But bit flips may not be noticed any time soon in the disk cache, and when they are later noticed, may be wrongly attributed to disk failure. It's a different failure mode than a bit flip in some source code that you are about to compile (which is far less likely to happen, due to that source code being a much "smaller target").
eklitzke 3y ago
This paper is ~15 years old. It was published in 2009, but the data was collected 2006-2008. As densities for things like RAM increase, error rates have to decrease commensurately to hide errors from end users. Additionally, Google runs hardware in their datacenters in a much more pathological way than most consumers will (e.g. they are more likely to run hardware much hotter than typical consumers). The point is that the figures in the paper are an interesting starting point for discussion, but they're not necessarily applicable to your laptop.
- wongarsu 3y ago
  > they are more likely to run hardware much hotter than typical consumers I'd expect laptops and phones with their limited cooling capacity to run their hardware much hotter than datacenters with their expensive HVAC systems
- p-e-w 3y ago
  I'm not convinced by this argument. In practice, errors on consumer hardware seem to be several orders of magnitude less common than implied by the paper. If they weren't, we'd be hearing anecdotes of one-character diffs appearing out of nowhere all the time, among many other issues. It's hard to see how this could be explained away with the slight increase in RAM densities since 2008, or different temperatures in data centers (plenty of individuals run their systems crazy hot, e.g. when the tower is stuffed with dust after years without cleaning).
  - alpaca128 3y ago
    Run any consumer device for more than 4 weeks and it'll slowly accumulate all kinds of weird bugs. I've seen smartphones unable to receive SMS because they simply ran too long without reboot, a Windows PC where selecting files wouldn't work anymore etc. > we'd be hearing anecdotes of one-character diffs appearing out of nowhere all the time On consumer devices it might simply be a small "attack" surface. A server running 24/7 serving countless requests would probably see more issues than a laptop running idle 95% of the time and loading a website every few minutes. Also a bit flip isn't guaranteed to cause a noticeable bug - and if it does it might not stand out from the usual noise of janky software oddities.
theginger 3y ago
They observed the majority of dimms had no errors. They had ecc ram so it's possible for a dimm that is going bad to carry on in service regularly producing correctable errors. If they were in your laptop you would start getting problems quite quickly and end up replacing.
OskarS 3y ago
If it's this common, it should be straight-forward to test as well, right? Allocate 1 gig of memory, do whatever mmap call you need to make sure it's not paged out, fill it with deterministically random stuff, then every hour check the contents and refill it. Keep running for a month. Seems odd to me that we have rely on some paper from 2009...
deleted 3y ago
[deleted]
duffyjp 3y ago
I'll run memtest86 for a full day when I build a system or get new ram. I've never seen an error outside of the situation where a stick has gone bad and it errors like crazy. I don't use ECC other than on my homelab server either. Tons of people do this and have no issues with bit flips or we'd be hearing about it.
- bcrl 3y ago
  Anecdotes are not real world statistics. Most people wouldn't identify a flipped bit of memory that caused a web page to glitch in their browser as a hardware problem. They'll just write it off as general enshittification of the web.
- sliken 3y ago
  In my experience, after handling many dozens of systems (out of 3000) with ECC errors, it's quite rare for the problem to be reproducible in memtest86. Running in production often triggers many errors per day. Our alarm threshold is over 100 correctable or 1 uncorrectable error per day. After a producing zero errors with 24 hours in memtest86, I usually see more errors when returned into production. Not sure if it's heat/cooling related, the access pattern, or maybe memtest86 tests the memory value, but not the memory address. I.e. if you fill memory with 0xdeadbeef you'll read 0xdeadbeef successfully even if the address is off. Dimms can "fail", but still have low error rates. Stories are pretty common, LinusT had one in the last year spent something like a week tracking it down, assumed it was a new kernel error. Tracking down memory errors with ECC is a painful process that involve tracking down numerous possible causes crashes. Sure bitflips are easy to ignore, but weird things do happen. Linux crashes, processes crash, suddenly things are weird, the desktop doesn't work like you expect. People assume that "the system is buggy" or it's related to patching, or some other user space error, but sometimes it's just a bitflip. Sadly there are few cases where a bit flip is obvious, but if you keep 10k files around for a decade it's pretty common to find a few corrupted. Various weird behaviors have been tracked down to memory issues. GCC for years had an error that nearly always was an internal compiler error ... triggered by a memory problem. One famous speed run of a game had something very weird happened, which was tracked to a single bit memory error. Sure can you get away with ECC, certainly. But the minimal price increase is in my opinion is quite reasonable. I'd much rather get a "single bit error corrected on dimm #3" dmesg, then occasional (or not so occasional) process, kernel, or filesystem crashes. I'm typing this from an 2015 system with a Xeon e3-1230 CPU that was cheaper than the equivalent i7, but has ECC support. Sure I spent a bit more on the motherboard and dimms, something around $120. Money well spent. I find system crashes quite disruptive, even 1 less per year is valuable to me.
TacticalCoder 3y ago
> How is that possible? I'd say that we live in a world of checksums, cryptographic checksums, retries, deterministic builds, distributed VCS (like Git), digital signatures, etc. which helps a lot with the very rare bitflips that do actually happen. > For example, in the average statically compiled codebase, flipping a single bit in source code being edited (with the code then being saved to disk) will make it fail to compile with high probability, which would be noticed immediately. I fully agree. If I set say "unstable" settings in my UEFI/BIOS for the RAM, I cannot compile, say, Emacs, without getting a segfault. > For example, in the average statically compiled codebase, flipping a single bit in source code being edited (with the code then being saved to disk) will make it fail to compile with high probability, which would be noticed immediately. That seems incongruent with the figure quoted above. I don't understand it either.
FuriouslyAdrift 3y ago
DDR5 includes on chip ECC in the standard (due to the high density). It's not the same as "full" ECC, though, as it does not have vision once the data leaves the module (on the wire, at the CPU, etc)

proto_lambda 3y ago

I think you're off by a couple orders of magnitude there. 25e3 errors/(1e9 h * 1e6 b) = 25e-12 errors/hour/bit, ~~or 4 errors/hour/terabyte.~~ Edit: nope, I got that last step wrong, it's actually 200 errors/hour/terabyte, or 1 error/gigabyte/5 hours, pretty close to what you said.

PinguTS 3y ago

I never really experienced that in real life. That is the thing we need to cope with in functional safety or some call it machine safety. There are people working with (mainly) electrical machines and by possible all measures an error cause by interference, bit flip, stuck bit, … needs be detected that the worst case of dead and injured people needs to be prevented. That are things we also need to cope with in automated/self driving cars.

nottorp 3y ago
> I never really experienced that in real life. Most of the time the flipped bit will be in a cat photo and go unnoticed.

pmontra 3y ago

4 bit flips per GB in a working day, in every GB of my laptop. Let's keep using powers of 2: 256 days when I'm using it it's 1024 bit flips per GB per year. Some will happen in unused memory, some in running programs. It's 1024 * N bit flips in our code editor per year, were N is how many times our editor + language servers are larger than 1 GB. Some in unused and unsaved data, some in code paths I'm not using. However some will probably crash a program because of a wrong or invalid machine code instruction or visibly alter data. Times that per the number of people using computers. Those spelling mistakes or wrong figures were not our fault after all? In a code editor people should see one character git diffs in files that are open in the editor, parked in a long unused tab (if the editor detects the change and saves it or asks to.)

oakwhiz 3y ago

I've observed that error rates are not the same for every machine - there are very bad machines, which usually have some way of triggering enough errors systematically that they end up getting someone's attention, then there are bad machines which have memory errors or are observed to report errors or crash occasionally, and then a bunch of good machines which on the surface appear to work flawlessly. Sometimes what makes a machine bad is the memory itself, other times it's the hardware or the configuration that the hardware settles on. The statistics may be skewed by the uneven distribution of "bad machines" amongst good ones, especially if you take into account hardware in the hands of end users rather than servers. It gives a new meaning to "works on my machine."