4 ms·

There's an excellent Radiolab podcast episode about how often cosmic rays cause computer errors in practice. It's engaging and educational: https://radiolab.org

There's an excellent Radiolab podcast episode about how often cosmic rays cause computer errors in practice. It's engaging and educational: https://radiolab.org/podcast/bit-flip https://radiolab.org/podcast/bit-flip I always knew about the theoretical cosmic ray bit flips. Before listening to this episode, I did not stop to think how often they actually cause problems.

remisharrock 3y ago
Am I the only one with exactly this episode that does not load ? All the others are ok ?
- archargelod 3y ago
  Same for me. After some searching I've found a mirror that still works [0]. [0] - https://podcastaddict.com/wissenschaft-auf-die-ohren/episode/150771599 https://podcastaddict.com/wissenschaft-auf-die-ohren/episode...
tzs 3y ago
I tried to observe bit flips and failed. I'm not sure if there was a problem with my methodology or my apartment was well shielded or what. My methodology was simple. On a Linux home server that had plenty of spare memory (non ECC RAM) I ran a process that simply alloced a large buffer and filled it with a pattern. It would then periodically scan through the buffer looking for changes to the pattern. I ran this for over a year which should have been long enough given the amount of RAM I was using and the rates that I found in the literature for cosmic ray induced bit flips resulted in several flips. My method would have missed a flip if the page it happened on had been paged out sometime in the past and that paged out copy still existed, and the bit flip happens between the time of the last scan and the time the kernel decides to discard that page. On the next scan it would page fault and load the good page. But the system was very lightly loaded and almost never actually had to page things out, so most of the time if a bit flipped it should have still been there by the next scan and so I don't think this explains why I saw no flips. A few years later I got a Mac Pro, which had ECC memory. I used that mostly at work from 2008-2017. I got another Mac Pro in 2009 which I used at home from 2009-2017. I'd occasionally look at the memory status in System Report which should say "ECC Errors" if the ECC had to fix any errors and only ever saw "OK". I'm not sure if that resets on boot and I only looked occasionally so if it does reset than it is quite likely I would have missed an error statuses.
- dunham 3y ago
  I recall seeing system log messages about ECC error correction on Sun Sparc machines back in the 90's. I don't know if it was radiation or flaky memory though.
- Arch-TK 3y ago
  I think it's possible that we've historically mis-attributed memory errors to cosmic rays when in fact they were caused by heat. If you ran that machine in a hotbox at 85°C for a year I think you probably would have experienced higher error rates. Bonus points if you also force the data to swap out a lot so it gets transferred through as many data paths as possible.
- toast0 3y ago
  My experience from a ~ 2000 server job with all ECC RAM is that errors cluster a lot. Easily 90%+ of the systems never had any ECC errors (correctable or no) in their whole life. Some systems would get one error a day. A small number got 100s per hour until we shut them down. Even less got thousands per hour. One in my whole career got like 50,000 in the time between when it started running very slowly (from processing ECC interrupts) and the next hourly reporting interval. You might get close to the literature numbers by taking that one instance and dividing it over the total ram byte hours. Although that was probably hardware failure and not comsic rays, but then you don't get a different ECC code for comsic rays. Also, there was an expectation that comsic ray induced bit errors would grow as ram circuit features shrunk, but it ended up not happening; reasons unknown or at least I never saw anything suggesting a reason. Getting errors in a small sample of RAM is unlikely, unless you specifically induce them by using debug features or misconfiguring your system (but some systems conspire to misconfigure themselves, making it easier to observe! a couple years ago, retail motherboards really liked setting the ram voltage too low)
  - moffkalast 3y ago
    > 50,000 It's not 3.6 roentgen... Joking aside that's incredibly fascinating, I never thought that ECC memory has that much of a performance impact. Might be more optimal to just get a large jerry can of water and put that over the server as radiation shielding lol.
mhitza 3y ago
Using cosmic rays as a practical dns hijacking tool (defcon) https://www.youtube.com/watch?v=aT7mnSstKGs https://www.youtube.com/watch?v=aT7mnSstKGs
dekhn 3y ago
I always assumed computers were "perfect"- at least, always executing their instructions according to a spec. FDIV and other bugs convinced me otherwise. Then, I worked on a system (a large processing accelerator) where some small fraction of machines had chips that would fail on a limited set of operations (producing the wrong result for a known input, vaidated by offline computation). Now I treat computers as statistically correct- you have to assume there is some level of corruption/error occurring due to hardware bugs or physical events.
- altruios 3y ago
  you would enjoy Dave Ackley's perspective on 'robust computing' vs 'efficient computing'. But his one of his ideas is allowing for error: which can speed up calculations if you can also reduce the error as the calculation goes on.
  - dekhn 3y ago
    Well... there's several things going wrong here and his perspective only seems to cover some of them. In the case of a scientific simulation, we can find algorithmic speedups that are "lossy"- an example would be approximations to n-body systems, where you need to calculate n-squared interactions (between all pairs). You can calculate all n-squared interactions which produces teh completely correct result. But since atoms that are far away don't interact strongly (falls off as 1 over r squared or more). So you can maintain a neighborlist- all atoms within a distance R- cheaper than you can calculate n-squared interactions, with some tiny error that is unknown. It's assumed in many cases the errror is neglible and the speedup is huge. Or you can switch to using a particle-mesh method which involves taking a fourier transform, doing some work in fourier space, then an inverse transform. The results are nlogn, and the error is small (and known). Speedup is signfiicant but takes much, much more computational skill and infrastructure. Next, the case I'm referring to of an accelerator running a tensorflow job, it's a totally different scenario- here, some random subset of machines will repeatedly return the wrong result- say, for a matrix-vector operation. Maybe garbage numbers, maybe all zero, maybe some infs. When that gets summed into your gradient, it often causes blow-up and the entire job terminates. It's not clear whether it makes sense to make high-performance jobs have to be robust- I see them as special cases where you're working hard to make sure the computing substrate is effectively 100% reliable (by sending/fixing those machines). Other folks have observed that some amount of small noise injection to the gradient can help training, but the sorts of errors I've seen almost immediately terminate the training job. I don't mind intentional noise addition, but noise due to hardware that is provably, reliably, and repeatedly miscalculating results? Not so much.
Nomadeon 3y ago
As we went from zero to 10K+ embedded systems (full PCs with significant RAM) the issues got weirder. The best was a one-off error log along the lines of "unknown type System.DateTime". Huh? That's a system defined type that just went missing. Never saw it again. Another at a different employer was a crash that occurred after a check condition that absolutely should have gated the crash from being reached. Single threaded. Simple microcontroller. Had to reflash it to flip the bit back. After doing the math on how much RAM we had in the wild vs. cosmic bit flip rates reported in super computers, we had to expect one flip per year. If it's a safety critical system, server or not, use ECC RAM!!
- repiret 3y ago
  I am of the opinion that far more than safety critical systems should use ECC. You should use ECC anytime bit flips might cost you more money then the ECC does, which is why I insist on ECC for my desktop computers.
- eschneider 3y ago
  Given a large enough installed base, any unlikely but possible problem will occur for some segment of the user population. Guaranteed. :/
repiret 3y ago
At work we make a hardware product, and among other things, it has a small microcontroller on it. We keep several hundred instances of our product to run automated integration tests. At that scale, we find SRAM bit-flips one of the microcontrollers every few weeks. We have a handful of software measures that are able to stop most instances of bit-flip from having a user-visible effect.
- bcrl 3y ago
  That kind of rate smells like a hardware bug. Insufficient decoupling caps maybe?
  - repiret 3y ago
    That seems plausible. I wasn’t deeply involved in the investigation, but the silicon vendor supports the cosmic ray hypothesis.
    - bcrl 3y ago
      The rate of cosmic rays impacting hardware should be extremely low, unless there's some kind of environmental factor (like being at a very high altitude). Or maybe there was a special train going by as in: https://www.jakepoz.com/debugging-behind-the-iron-curtain/ https://www.jakepoz.com/debugging-behind-the-iron-curtain/

remisharrock 3y ago

Am I the only one with exactly this episode that does not load ? All the others are ok ?

archargelod 3y ago
Same for me. After some searching I've found a mirror that still works [0]. [0] - https://podcastaddict.com/wissenschaft-auf-die-ohren/episode/150771599 https://podcastaddict.com/wissenschaft-auf-die-ohren/episode...

tzs 3y ago

I tried to observe bit flips and failed. I'm not sure if there was a problem with my methodology or my apartment was well shielded or what. My methodology was simple. On a Linux home server that had plenty of spare memory (non ECC RAM) I ran a process that simply alloced a large buffer and filled it with a pattern. It would then periodically scan through the buffer looking for changes to the pattern. I ran this for over a year which should have been long enough given the amount of RAM I was using and the rates that I found in the literature for cosmic ray induced bit flips resulted in several flips. My method would have missed a flip if the page it happened on had been paged out sometime in the past and that paged out copy still existed, and the bit flip happens between the time of the last scan and the time the kernel decides to discard that page. On the next scan it would page fault and load the good page. But the system was very lightly loaded and almost never actually had to page things out, so most of the time if a bit flipped it should have still been there by the next scan and so I don't think this explains why I saw no flips. A few years later I got a Mac Pro, which had ECC memory. I used that mostly at work from 2008-2017. I got another Mac Pro in 2009 which I used at home from 2009-2017. I'd occasionally look at the memory status in System Report which should say "ECC Errors" if the ECC had to fix any errors and only ever saw "OK". I'm not sure if that resets on boot and I only looked occasionally so if it does reset than it is quite likely I would have missed an error statuses.

dunham 3y ago
I recall seeing system log messages about ECC error correction on Sun Sparc machines back in the 90's. I don't know if it was radiation or flaky memory though.
Arch-TK 3y ago
I think it's possible that we've historically mis-attributed memory errors to cosmic rays when in fact they were caused by heat. If you ran that machine in a hotbox at 85°C for a year I think you probably would have experienced higher error rates. Bonus points if you also force the data to swap out a lot so it gets transferred through as many data paths as possible.
toast0 3y ago
My experience from a ~ 2000 server job with all ECC RAM is that errors cluster a lot. Easily 90%+ of the systems never had any ECC errors (correctable or no) in their whole life. Some systems would get one error a day. A small number got 100s per hour until we shut them down. Even less got thousands per hour. One in my whole career got like 50,000 in the time between when it started running very slowly (from processing ECC interrupts) and the next hourly reporting interval. You might get close to the literature numbers by taking that one instance and dividing it over the total ram byte hours. Although that was probably hardware failure and not comsic rays, but then you don't get a different ECC code for comsic rays. Also, there was an expectation that comsic ray induced bit errors would grow as ram circuit features shrunk, but it ended up not happening; reasons unknown or at least I never saw anything suggesting a reason. Getting errors in a small sample of RAM is unlikely, unless you specifically induce them by using debug features or misconfiguring your system (but some systems conspire to misconfigure themselves, making it easier to observe! a couple years ago, retail motherboards really liked setting the ram voltage too low)
- moffkalast 3y ago
  > 50,000 It's not 3.6 roentgen... Joking aside that's incredibly fascinating, I never thought that ECC memory has that much of a performance impact. Might be more optimal to just get a large jerry can of water and put that over the server as radiation shielding lol.

mhitza 3y ago

Using cosmic rays as a practical dns hijacking tool (defcon) https://www.youtube.com/watch?v=aT7mnSstKGs https://www.youtube.com/watch?v=aT7mnSstKGs

dekhn 3y ago

I always assumed computers were "perfect"- at least, always executing their instructions according to a spec. FDIV and other bugs convinced me otherwise. Then, I worked on a system (a large processing accelerator) where some small fraction of machines had chips that would fail on a limited set of operations (producing the wrong result for a known input, vaidated by offline computation). Now I treat computers as statistically correct- you have to assume there is some level of corruption/error occurring due to hardware bugs or physical events.

altruios 3y ago
you would enjoy Dave Ackley's perspective on 'robust computing' vs 'efficient computing'. But his one of his ideas is allowing for error: which can speed up calculations if you can also reduce the error as the calculation goes on.
- dekhn 3y ago
  Well... there's several things going wrong here and his perspective only seems to cover some of them. In the case of a scientific simulation, we can find algorithmic speedups that are "lossy"- an example would be approximations to n-body systems, where you need to calculate n-squared interactions (between all pairs). You can calculate all n-squared interactions which produces teh completely correct result. But since atoms that are far away don't interact strongly (falls off as 1 over r squared or more). So you can maintain a neighborlist- all atoms within a distance R- cheaper than you can calculate n-squared interactions, with some tiny error that is unknown. It's assumed in many cases the errror is neglible and the speedup is huge. Or you can switch to using a particle-mesh method which involves taking a fourier transform, doing some work in fourier space, then an inverse transform. The results are nlogn, and the error is small (and known). Speedup is signfiicant but takes much, much more computational skill and infrastructure. Next, the case I'm referring to of an accelerator running a tensorflow job, it's a totally different scenario- here, some random subset of machines will repeatedly return the wrong result- say, for a matrix-vector operation. Maybe garbage numbers, maybe all zero, maybe some infs. When that gets summed into your gradient, it often causes blow-up and the entire job terminates. It's not clear whether it makes sense to make high-performance jobs have to be robust- I see them as special cases where you're working hard to make sure the computing substrate is effectively 100% reliable (by sending/fixing those machines). Other folks have observed that some amount of small noise injection to the gradient can help training, but the sorts of errors I've seen almost immediately terminate the training job. I don't mind intentional noise addition, but noise due to hardware that is provably, reliably, and repeatedly miscalculating results? Not so much.

Nomadeon 3y ago

As we went from zero to 10K+ embedded systems (full PCs with significant RAM) the issues got weirder. The best was a one-off error log along the lines of "unknown type System.DateTime". Huh? That's a system defined type that just went missing. Never saw it again. Another at a different employer was a crash that occurred after a check condition that absolutely should have gated the crash from being reached. Single threaded. Simple microcontroller. Had to reflash it to flip the bit back. After doing the math on how much RAM we had in the wild vs. cosmic bit flip rates reported in super computers, we had to expect one flip per year. If it's a safety critical system, server or not, use ECC RAM!!

repiret 3y ago
I am of the opinion that far more than safety critical systems should use ECC. You should use ECC anytime bit flips might cost you more money then the ECC does, which is why I insist on ECC for my desktop computers.
eschneider 3y ago
Given a large enough installed base, any unlikely but possible problem will occur for some segment of the user population. Guaranteed. :/

repiret 3y ago

At work we make a hardware product, and among other things, it has a small microcontroller on it. We keep several hundred instances of our product to run automated integration tests. At that scale, we find SRAM bit-flips one of the microcontrollers every few weeks. We have a handful of software measures that are able to stop most instances of bit-flip from having a user-visible effect.

bcrl 3y ago
That kind of rate smells like a hardware bug. Insufficient decoupling caps maybe?
- repiret 3y ago
  That seems plausible. I wasn’t deeply involved in the investigation, but the silicon vendor supports the cosmic ray hypothesis.
  - bcrl 3y ago
    The rate of cosmic rays impacting hardware should be extremely low, unless there's some kind of environmental factor (like being at a very high altitude). Or maybe there was a special train going by as in: https://www.jakepoz.com/debugging-behind-the-iron-curtain/ https://www.jakepoz.com/debugging-behind-the-iron-curtain/