15 ms·

I tried to observe bit flips and failed. I'm not sure if there was a problem with my methodology or my apartment was well shielded or what. My methodology was

by tzs 3y ago

I tried to observe bit flips and failed. I'm not sure if there was a problem with my methodology or my apartment was well shielded or what. My methodology was simple. On a Linux home server that had plenty of spare memory (non ECC RAM) I ran a process that simply alloced a large buffer and filled it with a pattern. It would then periodically scan through the buffer looking for changes to the pattern. I ran this for over a year which should have been long enough given the amount of RAM I was using and the rates that I found in the literature for cosmic ray induced bit flips resulted in several flips. My method would have missed a flip if the page it happened on had been paged out sometime in the past and that paged out copy still existed, and the bit flip happens between the time of the last scan and the time the kernel decides to discard that page. On the next scan it would page fault and load the good page. But the system was very lightly loaded and almost never actually had to page things out, so most of the time if a bit flipped it should have still been there by the next scan and so I don't think this explains why I saw no flips. A few years later I got a Mac Pro, which had ECC memory. I used that mostly at work from 2008-2017. I got another Mac Pro in 2009 which I used at home from 2009-2017. I'd occasionally look at the memory status in System Report which should say "ECC Errors" if the ECC had to fix any errors and only ever saw "OK". I'm not sure if that resets on boot and I only looked occasionally so if it does reset than it is quite likely I would have missed an error statuses.

dunham 3y ago
I recall seeing system log messages about ECC error correction on Sun Sparc machines back in the 90's. I don't know if it was radiation or flaky memory though.
Arch-TK 3y ago
I think it's possible that we've historically mis-attributed memory errors to cosmic rays when in fact they were caused by heat. If you ran that machine in a hotbox at 85°C for a year I think you probably would have experienced higher error rates. Bonus points if you also force the data to swap out a lot so it gets transferred through as many data paths as possible.
toast0 3y ago
My experience from a ~ 2000 server job with all ECC RAM is that errors cluster a lot. Easily 90%+ of the systems never had any ECC errors (correctable or no) in their whole life. Some systems would get one error a day. A small number got 100s per hour until we shut them down. Even less got thousands per hour. One in my whole career got like 50,000 in the time between when it started running very slowly (from processing ECC interrupts) and the next hourly reporting interval. You might get close to the literature numbers by taking that one instance and dividing it over the total ram byte hours. Although that was probably hardware failure and not comsic rays, but then you don't get a different ECC code for comsic rays. Also, there was an expectation that comsic ray induced bit errors would grow as ram circuit features shrunk, but it ended up not happening; reasons unknown or at least I never saw anything suggesting a reason. Getting errors in a small sample of RAM is unlikely, unless you specifically induce them by using debug features or misconfiguring your system (but some systems conspire to misconfigure themselves, making it easier to observe! a couple years ago, retail motherboards really liked setting the ram voltage too low)
- moffkalast 3y ago
  > 50,000 It's not 3.6 roentgen... Joking aside that's incredibly fascinating, I never thought that ECC memory has that much of a performance impact. Might be more optimal to just get a large jerry can of water and put that over the server as radiation shielding lol.
  - toast0 3y ago
    ECC interrupts were never a problem at reasonable counts. It's just this machine where the memory was falling apart where it was a problem. Our system was robust to a machine halting, but not so great at dealing with a machine running very very slow. Plus, it wasn't easy to connect and shut down the service (maybe we should have just killed it from IPMI, but this was the only time it happened, so learning experience). I had a similar issue one time where a Pentium III era server rebooted and came up with a comically small amount of memory, maybe 2-4 mb instead of 128 mb. That wasn't too bad, because it was a very lightly service; important to be on its own machine for reasons, but didn't need much. Just ran a little slow when it was running from swap. I think it did trigger a swap usage alert, and then it was like why is it swapping, why is it so slow, wait why doesn't it have any memory!?
prpl 3y ago
The incidence of cosmic rays is biased vertically, especially close to sea level where you get more muons than electrons, so that is one thing you would want to think about - you’d want to maximize your surface area toward that because your detection volume is so low. As a rule of thumb, you get 2 muons through your head a minute - but of course your head has a huge volume compared to memory chips.

by tzs 3y ago

dunham 3y ago

I recall seeing system log messages about ECC error correction on Sun Sparc machines back in the 90's. I don't know if it was radiation or flaky memory though.

Arch-TK 3y ago

I think it's possible that we've historically mis-attributed memory errors to cosmic rays when in fact they were caused by heat. If you ran that machine in a hotbox at 85°C for a year I think you probably would have experienced higher error rates. Bonus points if you also force the data to swap out a lot so it gets transferred through as many data paths as possible.

toast0 3y ago

My experience from a ~ 2000 server job with all ECC RAM is that errors cluster a lot. Easily 90%+ of the systems never had any ECC errors (correctable or no) in their whole life. Some systems would get one error a day. A small number got 100s per hour until we shut them down. Even less got thousands per hour. One in my whole career got like 50,000 in the time between when it started running very slowly (from processing ECC interrupts) and the next hourly reporting interval. You might get close to the literature numbers by taking that one instance and dividing it over the total ram byte hours. Although that was probably hardware failure and not comsic rays, but then you don't get a different ECC code for comsic rays. Also, there was an expectation that comsic ray induced bit errors would grow as ram circuit features shrunk, but it ended up not happening; reasons unknown or at least I never saw anything suggesting a reason. Getting errors in a small sample of RAM is unlikely, unless you specifically induce them by using debug features or misconfiguring your system (but some systems conspire to misconfigure themselves, making it easier to observe! a couple years ago, retail motherboards really liked setting the ram voltage too low)

moffkalast 3y ago
> 50,000 It's not 3.6 roentgen... Joking aside that's incredibly fascinating, I never thought that ECC memory has that much of a performance impact. Might be more optimal to just get a large jerry can of water and put that over the server as radiation shielding lol.
- toast0 3y ago
  ECC interrupts were never a problem at reasonable counts. It's just this machine where the memory was falling apart where it was a problem. Our system was robust to a machine halting, but not so great at dealing with a machine running very very slow. Plus, it wasn't easy to connect and shut down the service (maybe we should have just killed it from IPMI, but this was the only time it happened, so learning experience). I had a similar issue one time where a Pentium III era server rebooted and came up with a comically small amount of memory, maybe 2-4 mb instead of 128 mb. That wasn't too bad, because it was a very lightly service; important to be on its own machine for reasons, but didn't need much. Just ran a little slow when it was running from swap. I think it did trigger a swap usage alert, and then it was like why is it swapping, why is it so slow, wait why doesn't it have any memory!?

prpl 3y ago

The incidence of cosmic rays is biased vertically, especially close to sea level where you get more muons than electrons, so that is one thing you would want to think about - you’d want to maximize your surface area toward that because your detection volume is so low. As a rule of thumb, you get 2 muons through your head a minute - but of course your head has a huge volume compared to memory chips.