6 ms·

> or one bit per GB every two hours How is that possible? Wouldn't such an error frequency lead to user-observable problems all the time? For example, in the a

by p-e-w 3y ago

> or one bit per GB every two hours How is that possible? Wouldn't such an error frequency lead to user-observable problems all the time? For example, in the average statically compiled codebase, flipping a single bit in source code being edited (with the code then being saved to disk) will make it fail to compile with high probability, which would be noticed immediately. Yet I've never encountered this situation in practice, nor heard of anyone else encountering it, and like most people I don't even use ECC RAM. That seems incongruent with the figure quoted above.

viraptor 3y ago
Yeah, that's not realistic. This is as much code as goes through a CI I'm working with many times a day. I'd constantly see errors on an unknown variable if this was a real rate.
- teekert 3y ago
  Or you are using ECC RAM?
- bjornsing 3y ago
  Aren’t you forgetting the time aspect? If you’re keeping that code in RAM for just a few seconds at most then the probability of a random bit flip is negligible, even if the hourly rate is high. But I agree this sounds unrealistic.
  - codeflo 3y ago
    Presumably, most of the code doesn't change often, and consequently, should be kept in the filesystem cache and never actually read from disk. So the numbers could apply, I think.
- Gordonjcp 3y ago
  It's about one second of video at "editing quality" ;-)
jhugo 3y ago
One bit per GB of RAM, not per GB of processed source code (most of your RAM is likely empty most of the time, or used for stuff where a bit flip is less likely to be noticed). However, it still seems like a high estimate to me.
- pmontra 3y ago
  I have 32 GB of RAM. If I only use 1 GB of it it's still one bit flip in the RAM I'm using.
  - jhugo 3y ago
    Eh? The "cosmic ray" (or similar upset) is roughly equally likely to affect any bit in the 32GB, so the chance of it appearing in the 1GB that you're using right now is 32 times smaller than the chance of it happening anywhere in your RAM.
- zaphirplane 3y ago
  Operating systems keep files in ram In addition to memory allocation till they need the ram, so I think it should be mostly full all the time
  - mrspuratic 3y ago
    Unused RAM ~= wasted RAM... Recently I had to troubleshoot a linux process that started segfaulting randomly. Having dealt with non-ECC bit flips before I took a copy of the binary (i.e. from the filesystem cache), dropped the caches, and compared and disassembled the binaries: a single bit flip resulted in an illegal instruction. Cost of investigating issues >>> cost of ECC
  - jhugo 3y ago
    But bit flips may not be noticed any time soon in the disk cache, and when they are later noticed, may be wrongly attributed to disk failure. It's a different failure mode than a bit flip in some source code that you are about to compile (which is far less likely to happen, due to that source code being a much "smaller target").
eklitzke 3y ago
This paper is ~15 years old. It was published in 2009, but the data was collected 2006-2008. As densities for things like RAM increase, error rates have to decrease commensurately to hide errors from end users. Additionally, Google runs hardware in their datacenters in a much more pathological way than most consumers will (e.g. they are more likely to run hardware much hotter than typical consumers). The point is that the figures in the paper are an interesting starting point for discussion, but they're not necessarily applicable to your laptop.
- wongarsu 3y ago
  > they are more likely to run hardware much hotter than typical consumers I'd expect laptops and phones with their limited cooling capacity to run their hardware much hotter than datacenters with their expensive HVAC systems
- p-e-w 3y ago
  I'm not convinced by this argument. In practice, errors on consumer hardware seem to be several orders of magnitude less common than implied by the paper. If they weren't, we'd be hearing anecdotes of one-character diffs appearing out of nowhere all the time, among many other issues. It's hard to see how this could be explained away with the slight increase in RAM densities since 2008, or different temperatures in data centers (plenty of individuals run their systems crazy hot, e.g. when the tower is stuffed with dust after years without cleaning).
  - alpaca128 3y ago
    Run any consumer device for more than 4 weeks and it'll slowly accumulate all kinds of weird bugs. I've seen smartphones unable to receive SMS because they simply ran too long without reboot, a Windows PC where selecting files wouldn't work anymore etc. > we'd be hearing anecdotes of one-character diffs appearing out of nowhere all the time On consumer devices it might simply be a small "attack" surface. A server running 24/7 serving countless requests would probably see more issues than a laptop running idle 95% of the time and loading a website every few minutes. Also a bit flip isn't guaranteed to cause a noticeable bug - and if it does it might not stand out from the usual noise of janky software oddities.
theginger 3y ago
They observed the majority of dimms had no errors. They had ecc ram so it's possible for a dimm that is going bad to carry on in service regularly producing correctable errors. If they were in your laptop you would start getting problems quite quickly and end up replacing.
OskarS 3y ago
If it's this common, it should be straight-forward to test as well, right? Allocate 1 gig of memory, do whatever mmap call you need to make sure it's not paged out, fill it with deterministically random stuff, then every hour check the contents and refill it. Keep running for a month. Seems odd to me that we have rely on some paper from 2009...
deleted 3y ago
[deleted]
duffyjp 3y ago
I'll run memtest86 for a full day when I build a system or get new ram. I've never seen an error outside of the situation where a stick has gone bad and it errors like crazy. I don't use ECC other than on my homelab server either. Tons of people do this and have no issues with bit flips or we'd be hearing about it.
- bcrl 3y ago
  Anecdotes are not real world statistics. Most people wouldn't identify a flipped bit of memory that caused a web page to glitch in their browser as a hardware problem. They'll just write it off as general enshittification of the web.
- sliken 3y ago
  In my experience, after handling many dozens of systems (out of 3000) with ECC errors, it's quite rare for the problem to be reproducible in memtest86. Running in production often triggers many errors per day. Our alarm threshold is over 100 correctable or 1 uncorrectable error per day. After a producing zero errors with 24 hours in memtest86, I usually see more errors when returned into production. Not sure if it's heat/cooling related, the access pattern, or maybe memtest86 tests the memory value, but not the memory address. I.e. if you fill memory with 0xdeadbeef you'll read 0xdeadbeef successfully even if the address is off. Dimms can "fail", but still have low error rates. Stories are pretty common, LinusT had one in the last year spent something like a week tracking it down, assumed it was a new kernel error. Tracking down memory errors with ECC is a painful process that involve tracking down numerous possible causes crashes. Sure bitflips are easy to ignore, but weird things do happen. Linux crashes, processes crash, suddenly things are weird, the desktop doesn't work like you expect. People assume that "the system is buggy" or it's related to patching, or some other user space error, but sometimes it's just a bitflip. Sadly there are few cases where a bit flip is obvious, but if you keep 10k files around for a decade it's pretty common to find a few corrupted. Various weird behaviors have been tracked down to memory issues. GCC for years had an error that nearly always was an internal compiler error ... triggered by a memory problem. One famous speed run of a game had something very weird happened, which was tracked to a single bit memory error. Sure can you get away with ECC, certainly. But the minimal price increase is in my opinion is quite reasonable. I'd much rather get a "single bit error corrected on dimm #3" dmesg, then occasional (or not so occasional) process, kernel, or filesystem crashes. I'm typing this from an 2015 system with a Xeon e3-1230 CPU that was cheaper than the equivalent i7, but has ECC support. Sure I spent a bit more on the motherboard and dimms, something around $120. Money well spent. I find system crashes quite disruptive, even 1 less per year is valuable to me.
TacticalCoder 3y ago
> How is that possible? I'd say that we live in a world of checksums, cryptographic checksums, retries, deterministic builds, distributed VCS (like Git), digital signatures, etc. which helps a lot with the very rare bitflips that do actually happen. > For example, in the average statically compiled codebase, flipping a single bit in source code being edited (with the code then being saved to disk) will make it fail to compile with high probability, which would be noticed immediately. I fully agree. If I set say "unstable" settings in my UEFI/BIOS for the RAM, I cannot compile, say, Emacs, without getting a segfault. > For example, in the average statically compiled codebase, flipping a single bit in source code being edited (with the code then being saved to disk) will make it fail to compile with high probability, which would be noticed immediately. That seems incongruent with the figure quoted above. I don't understand it either.
FuriouslyAdrift 3y ago
DDR5 includes on chip ECC in the standard (due to the high density). It's not the same as "full" ECC, though, as it does not have vision once the data leaves the module (on the wire, at the CPU, etc)

by p-e-w 3y ago

viraptor 3y ago

Yeah, that's not realistic. This is as much code as goes through a CI I'm working with many times a day. I'd constantly see errors on an unknown variable if this was a real rate.

teekert 3y ago
Or you are using ECC RAM?
bjornsing 3y ago
Aren’t you forgetting the time aspect? If you’re keeping that code in RAM for just a few seconds at most then the probability of a random bit flip is negligible, even if the hourly rate is high. But I agree this sounds unrealistic.
- codeflo 3y ago
  Presumably, most of the code doesn't change often, and consequently, should be kept in the filesystem cache and never actually read from disk. So the numbers could apply, I think.
Gordonjcp 3y ago
It's about one second of video at "editing quality" ;-)

jhugo 3y ago

One bit per GB of RAM, not per GB of processed source code (most of your RAM is likely empty most of the time, or used for stuff where a bit flip is less likely to be noticed). However, it still seems like a high estimate to me.

pmontra 3y ago
I have 32 GB of RAM. If I only use 1 GB of it it's still one bit flip in the RAM I'm using.
- jhugo 3y ago
  Eh? The "cosmic ray" (or similar upset) is roughly equally likely to affect any bit in the 32GB, so the chance of it appearing in the 1GB that you're using right now is 32 times smaller than the chance of it happening anywhere in your RAM.
zaphirplane 3y ago
Operating systems keep files in ram In addition to memory allocation till they need the ram, so I think it should be mostly full all the time
- mrspuratic 3y ago
  Unused RAM ~= wasted RAM... Recently I had to troubleshoot a linux process that started segfaulting randomly. Having dealt with non-ECC bit flips before I took a copy of the binary (i.e. from the filesystem cache), dropped the caches, and compared and disassembled the binaries: a single bit flip resulted in an illegal instruction. Cost of investigating issues >>> cost of ECC
- jhugo 3y ago
  But bit flips may not be noticed any time soon in the disk cache, and when they are later noticed, may be wrongly attributed to disk failure. It's a different failure mode than a bit flip in some source code that you are about to compile (which is far less likely to happen, due to that source code being a much "smaller target").

eklitzke 3y ago

This paper is ~15 years old. It was published in 2009, but the data was collected 2006-2008. As densities for things like RAM increase, error rates have to decrease commensurately to hide errors from end users. Additionally, Google runs hardware in their datacenters in a much more pathological way than most consumers will (e.g. they are more likely to run hardware much hotter than typical consumers). The point is that the figures in the paper are an interesting starting point for discussion, but they're not necessarily applicable to your laptop.

wongarsu 3y ago
> they are more likely to run hardware much hotter than typical consumers I'd expect laptops and phones with their limited cooling capacity to run their hardware much hotter than datacenters with their expensive HVAC systems
p-e-w 3y ago
I'm not convinced by this argument. In practice, errors on consumer hardware seem to be several orders of magnitude less common than implied by the paper. If they weren't, we'd be hearing anecdotes of one-character diffs appearing out of nowhere all the time, among many other issues. It's hard to see how this could be explained away with the slight increase in RAM densities since 2008, or different temperatures in data centers (plenty of individuals run their systems crazy hot, e.g. when the tower is stuffed with dust after years without cleaning).
- alpaca128 3y ago
  Run any consumer device for more than 4 weeks and it'll slowly accumulate all kinds of weird bugs. I've seen smartphones unable to receive SMS because they simply ran too long without reboot, a Windows PC where selecting files wouldn't work anymore etc. > we'd be hearing anecdotes of one-character diffs appearing out of nowhere all the time On consumer devices it might simply be a small "attack" surface. A server running 24/7 serving countless requests would probably see more issues than a laptop running idle 95% of the time and loading a website every few minutes. Also a bit flip isn't guaranteed to cause a noticeable bug - and if it does it might not stand out from the usual noise of janky software oddities.

theginger 3y ago

They observed the majority of dimms had no errors. They had ecc ram so it's possible for a dimm that is going bad to carry on in service regularly producing correctable errors. If they were in your laptop you would start getting problems quite quickly and end up replacing.

OskarS 3y ago

If it's this common, it should be straight-forward to test as well, right? Allocate 1 gig of memory, do whatever mmap call you need to make sure it's not paged out, fill it with deterministically random stuff, then every hour check the contents and refill it. Keep running for a month. Seems odd to me that we have rely on some paper from 2009...

deleted 3y ago

[deleted]

duffyjp 3y ago

I'll run memtest86 for a full day when I build a system or get new ram. I've never seen an error outside of the situation where a stick has gone bad and it errors like crazy. I don't use ECC other than on my homelab server either. Tons of people do this and have no issues with bit flips or we'd be hearing about it.

bcrl 3y ago
Anecdotes are not real world statistics. Most people wouldn't identify a flipped bit of memory that caused a web page to glitch in their browser as a hardware problem. They'll just write it off as general enshittification of the web.
sliken 3y ago
In my experience, after handling many dozens of systems (out of 3000) with ECC errors, it's quite rare for the problem to be reproducible in memtest86. Running in production often triggers many errors per day. Our alarm threshold is over 100 correctable or 1 uncorrectable error per day. After a producing zero errors with 24 hours in memtest86, I usually see more errors when returned into production. Not sure if it's heat/cooling related, the access pattern, or maybe memtest86 tests the memory value, but not the memory address. I.e. if you fill memory with 0xdeadbeef you'll read 0xdeadbeef successfully even if the address is off. Dimms can "fail", but still have low error rates. Stories are pretty common, LinusT had one in the last year spent something like a week tracking it down, assumed it was a new kernel error. Tracking down memory errors with ECC is a painful process that involve tracking down numerous possible causes crashes. Sure bitflips are easy to ignore, but weird things do happen. Linux crashes, processes crash, suddenly things are weird, the desktop doesn't work like you expect. People assume that "the system is buggy" or it's related to patching, or some other user space error, but sometimes it's just a bitflip. Sadly there are few cases where a bit flip is obvious, but if you keep 10k files around for a decade it's pretty common to find a few corrupted. Various weird behaviors have been tracked down to memory issues. GCC for years had an error that nearly always was an internal compiler error ... triggered by a memory problem. One famous speed run of a game had something very weird happened, which was tracked to a single bit memory error. Sure can you get away with ECC, certainly. But the minimal price increase is in my opinion is quite reasonable. I'd much rather get a "single bit error corrected on dimm #3" dmesg, then occasional (or not so occasional) process, kernel, or filesystem crashes. I'm typing this from an 2015 system with a Xeon e3-1230 CPU that was cheaper than the equivalent i7, but has ECC support. Sure I spent a bit more on the motherboard and dimms, something around $120. Money well spent. I find system crashes quite disruptive, even 1 less per year is valuable to me.

TacticalCoder 3y ago

> How is that possible? I'd say that we live in a world of checksums, cryptographic checksums, retries, deterministic builds, distributed VCS (like Git), digital signatures, etc. which helps a lot with the very rare bitflips that do actually happen. > For example, in the average statically compiled codebase, flipping a single bit in source code being edited (with the code then being saved to disk) will make it fail to compile with high probability, which would be noticed immediately. I fully agree. If I set say "unstable" settings in my UEFI/BIOS for the RAM, I cannot compile, say, Emacs, without getting a segfault. > For example, in the average statically compiled codebase, flipping a single bit in source code being edited (with the code then being saved to disk) will make it fail to compile with high probability, which would be noticed immediately. That seems incongruent with the figure quoted above. I don't understand it either.

FuriouslyAdrift 3y ago

DDR5 includes on chip ECC in the standard (due to the high density). It's not the same as "full" ECC, though, as it does not have vision once the data leaves the module (on the wire, at the CPU, etc)