4 ms·

Bit flips in RAM happen often enough to be an issue. I make a desktop program with a sizeable installation base and it uses a form paged on-disk storage for la

Bit flips in RAM happen often enough to be an issue. I make a desktop program with a sizeable installation base and it uses a form paged on-disk storage for larger data sets. The code has a fair amount of asserts in it and one specific assert would keep failing in a very tiny percentage of the cases. I kept reviewing the code and scratching my head before finally adding some extra logging for this case. Lo and behold, in all failed cases an argument passed to a function had one of the bits set for no apparent reason. I.e. the offsets of pages in the cache would be: 00000000000000 ... 00000000100000 ... 00000000200000 ... 00000000300000 ... But the args would come in as 80000000100000 00040000200000 01000000300000 ... Followed up with the people who reported these, asked to run an overnight RAM test and ALL of them found issues with their RAM. That was quite an eye-opener.

Someone 6y ago
You weren’t explicitly saying it, but some might infer it, so I think it’s worth mentioning that bit flips in RAM aren’t necessarily radiation-induced. Some other causes are heat and aging (both increase capacitor leakage, and DRAM cells are tiny capacitors), or lack of robustness in refresh logic.
- quercusa 6y ago
  Another cause, at least in the old days, was bad board design, in which badly-placed traces could induce currents in other traces. Of course this only happened two+ hours into the test.
s9w 6y ago
I remember a blog post by Riot (league of legends developer) also about this happening. But I couldn't find it again when I was searching for it. I can't imagine the havoc that regular bit flips must cause.
tyingq 6y ago
One of the more interesting ways to test this is to register a one character typo of a very popular domain where the typo is one bit off. Like, say, dacebook.com or facebooo.com Spin up a web server, and you'll get traffic, but typically only one of the requests from a specific client.
- sneak 6y ago
  it has to be a one-bit-error byte change in particular. https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinaburg_Bitsquatting_WP.pdf https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabu...
andi999 6y ago
Were all the machines which reported that in the same building (your software could been used in house). If so, better check for radiation levels, for health risks; I have never seen a RAM test fail actually (apart from a completely broken chip)
- edwintorok 6y ago
  As mentioned below it can deteriorate over time. A while ago I had a RAM stick fail. I noticed because GCC started segfaulting randomly and I haven't upgraded GCC recently. Sure enough a memtest revealed the fault. I think it didn't have enough cooling. I only purchased RAM with heatsinks since then, and I didn't have such problems anymore (also my latest case has much better airflow)
  - hinkley 6y ago
    I unplugged the wrong power cable once, and thereafter the RAID controller on that machine would report degradation after the system was running for a while. Finally ran a RAM error check, one bit in most of a stick was flipped. The last stick. So once the machine was all spooled up for a while, I’d get errors. This is also how I learned 3ware was using RAM instead of its own memory. What is he point of hardware RAID if main memory can screw it up? I have also heard of people swapping hardware on a defective machine only to learn that the power supply was bad, sending noise or brown power to the system. Swap in a new power supply and the components start to behave themselves.
- toast0 6y ago
  If you've got a system where you can tweak the ram timings and/or voltages (most desktops will let you do at least some of this), it's pretty easy to induce failures, if you haven't run into bad ram naturally. I've run enough systems with ECC ram to say the vast majority of ram works and will continue to work, but over time, a small amount will fail, again the vast majority is a single bit in the whole chip, but some will do multiple bits near each other (uncorrectable ecc error usually halts the system), or man single bit errors all over (many machine check errors per second makes the system go so slow, halting would be better).
brokenmachine 6y ago
I read a blog article years back by a game author who inserted CPU/RAM/Math sanity checks into the code (I forget what the reason he did this for was), and he was suprised with how often it detected hardware errors in customer's PCs. Wish I could find the article again, I forget the details but it was a fun read.

Someone 6y ago

You weren’t explicitly saying it, but some might infer it, so I think it’s worth mentioning that bit flips in RAM aren’t necessarily radiation-induced. Some other causes are heat and aging (both increase capacitor leakage, and DRAM cells are tiny capacitors), or lack of robustness in refresh logic.

quercusa 6y ago
Another cause, at least in the old days, was bad board design, in which badly-placed traces could induce currents in other traces. Of course this only happened two+ hours into the test.

s9w 6y ago

I remember a blog post by Riot (league of legends developer) also about this happening. But I couldn't find it again when I was searching for it. I can't imagine the havoc that regular bit flips must cause.

tyingq 6y ago

One of the more interesting ways to test this is to register a one character typo of a very popular domain where the typo is one bit off. Like, say, dacebook.com or facebooo.com Spin up a web server, and you'll get traffic, but typically only one of the requests from a specific client.

sneak 6y ago
it has to be a one-bit-error byte change in particular. https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinaburg_Bitsquatting_WP.pdf https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabu...

andi999 6y ago

Were all the machines which reported that in the same building (your software could been used in house). If so, better check for radiation levels, for health risks; I have never seen a RAM test fail actually (apart from a completely broken chip)

edwintorok 6y ago
As mentioned below it can deteriorate over time. A while ago I had a RAM stick fail. I noticed because GCC started segfaulting randomly and I haven't upgraded GCC recently. Sure enough a memtest revealed the fault. I think it didn't have enough cooling. I only purchased RAM with heatsinks since then, and I didn't have such problems anymore (also my latest case has much better airflow)
- hinkley 6y ago
  I unplugged the wrong power cable once, and thereafter the RAID controller on that machine would report degradation after the system was running for a while. Finally ran a RAM error check, one bit in most of a stick was flipped. The last stick. So once the machine was all spooled up for a while, I’d get errors. This is also how I learned 3ware was using RAM instead of its own memory. What is he point of hardware RAID if main memory can screw it up? I have also heard of people swapping hardware on a defective machine only to learn that the power supply was bad, sending noise or brown power to the system. Swap in a new power supply and the components start to behave themselves.
toast0 6y ago
If you've got a system where you can tweak the ram timings and/or voltages (most desktops will let you do at least some of this), it's pretty easy to induce failures, if you haven't run into bad ram naturally. I've run enough systems with ECC ram to say the vast majority of ram works and will continue to work, but over time, a small amount will fail, again the vast majority is a single bit in the whole chip, but some will do multiple bits near each other (uncorrectable ecc error usually halts the system), or man single bit errors all over (many machine check errors per second makes the system go so slow, halting would be better).

brokenmachine 6y ago

I read a blog article years back by a game author who inserted CPU/RAM/Math sanity checks into the code (I forget what the reason he did this for was), and he was suprised with how often it detected hardware errors in customer's PCs. Wish I could find the article again, I forget the details but it was a fun read.