5 ms·

Cosmic rays are a cop out but a good plot device. They're so rare, but one could write a bit flip into a script for a novel or a tv show as the solution to any

Cosmic rays are a cop out but a good plot device. They're so rare, but one could write a bit flip into a script for a novel or a tv show as the solution to any computer mystery.

bick_nyers 4y ago
Aliens using cosmic ray generators to flip specific bits and set off nukes.
- tsujamin 4y ago
  Reminds me of the Sophons in The Three-Body Problem
asdff 4y ago
I really should have milked the "cosmic ray bitflip ate my homework" excuse in hindsight
jupp0r 4y ago
One more reason to use ZFS.
- Helmut10001 4y ago
  Surprising that only two people in all of the comments here suggest to use ZFS. It could have detected any broken data before it was moved to the backup system.
  - jupp0r 4y ago
    Not a lot of people are familiar with how well ZFS is engineered compared to most other file systems and how life changingly different it is.
shemnon42 4y ago
Cosmic rays are more common than you think. Google's early infrastructure was impacted by a supernova (because their nodes were so cheap). But something like NOTAM can handle these single bit flips without a problem. https://www.newyorker.com/magazine/2018/12/10/the-friendship-that-made-google-huge https://www.newyorker.com/magazine/2018/12/10/the-friendship... File this under "early Google's infrastructure was a low grade cosmic ray detector."
- defrost 4y ago
  You can expect 250 or so cosmic ray events per second in a 42 litre sodium iodide crystal pack at 100m above sea level. Source: 10 years airbourne geophysics, radiometric calibrations. Addendum: In-flight upset 154 km west of Learmonth, WA 7 October 2008 VH-QPA Airbus A330-303 [1] was a probable (but uncertain) example of cosmic ray events causing multiple spikes in one of three air data inertial reference units (ADIRUs) that also went on to cause a failure mode of the "best of three" reporting system leading to a pitch down in which [2] > 110 of the 303 passengers and nine of the 12 crew members were injured; 12 of the occupants were seriously injured and another 39 received hospital medical treatment. HOWEVER .. despite 250 events per second in a 42 litre volume, it took 128 million hours of unit operation to see a failure mode. It's a lot of billiard balls going through a lot of space and a high bar for "something bad" ( just the right bit flip ) to happen. [1] https://www.atsb.gov.au/sites/default/files/media/3532398/ao2008070.pdf https://www.atsb.gov.au/sites/default/files/media/3532398/ao... [2] https://www.atsb.gov.au/publications/investigation_reports/2008/aair/ao-2008-070 https://www.atsb.gov.au/publications/investigation_reports/2... ( Final Report TAB )
nix23 4y ago
>They're so rare, They are not rare, just because you don't detect them (consumer hardware) means nothing. >>study done by IBM in the 1990’s that referenced 1 cosmic ray bit flip per 256MB of memory per month https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-what-flips-your-bit/ https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-...
- lioeters 4y ago
  One cosmic-ray bit-flip per 256MB per month. Significantly more for computers on aircrafts at higher altitudes. I kinda wish I hadn't learned this fact, how fragile everything is. On the other hand, it makes me appreciate the importance of error tolerance and recovery. > Cosmic ray flux depends on altitude. Computers operated on top of mountains experience an order of magnitude higher rate of soft errors compared to sea level. The rate of upsets in aircraft may be more than 300 times the sea level upset rate. https://en.wikipedia.org/wiki/Soft_error#Cosmic_rays_creating_energetic_neutrons_and_protons https://en.wikipedia.org/wiki/Soft_error#Cosmic_rays_creatin...
  - nix23 4y ago
    >One cosmic-ray bit-flip per 256MB per month. Significantly more for computers on aircrafts at higher altitudes. Yes thats why you have max altitude for normal servers/hardware and it's also more disastrous on modern HW -> smaller transistors.
  - shadowgovt 4y ago
    The reason we all now wear our seatbelts through the entire flight is there was a cross-Pacific flight that had a sudden altitude drop where several people were injured from rebounding off the ceiling. When the news reported on it, they reported the cause was believed to have been an undetected massive downdraft that shoved the plane down. I lost the thread on that story before the investigation concluded the actual cause was autopilot error. It is, in some ways, more comforting to me to know that the issue wasn't novel atmospheric phenomena, but instead relatively-mundane cosmic radiation flipping one packet of data from the sensors to the avionics that the avionics lacked sufficient redundancy to detect or discard. As a result, the autopilot believed the plane had suddenly pitched 90 degrees and drastically corrected to escape stall.

Cosmic rays are a cop out but a good plot device. They're so rare, but one could write a bit flip into a script for a novel or a tv show as the solution to any computer mystery.

bick_nyers 4y ago

Aliens using cosmic ray generators to flip specific bits and set off nukes.

tsujamin 4y ago
Reminds me of the Sophons in The Three-Body Problem

asdff 4y ago

I really should have milked the "cosmic ray bitflip ate my homework" excuse in hindsight

jupp0r 4y ago

One more reason to use ZFS.

Helmut10001 4y ago
Surprising that only two people in all of the comments here suggest to use ZFS. It could have detected any broken data before it was moved to the backup system.
- jupp0r 4y ago
  Not a lot of people are familiar with how well ZFS is engineered compared to most other file systems and how life changingly different it is.

shemnon42 4y ago

Cosmic rays are more common than you think. Google's early infrastructure was impacted by a supernova (because their nodes were so cheap). But something like NOTAM can handle these single bit flips without a problem. https://www.newyorker.com/magazine/2018/12/10/the-friendship-that-made-google-huge https://www.newyorker.com/magazine/2018/12/10/the-friendship... File this under "early Google's infrastructure was a low grade cosmic ray detector."

defrost 4y ago
You can expect 250 or so cosmic ray events per second in a 42 litre sodium iodide crystal pack at 100m above sea level. Source: 10 years airbourne geophysics, radiometric calibrations. Addendum: In-flight upset 154 km west of Learmonth, WA 7 October 2008 VH-QPA Airbus A330-303 [1] was a probable (but uncertain) example of cosmic ray events causing multiple spikes in one of three air data inertial reference units (ADIRUs) that also went on to cause a failure mode of the "best of three" reporting system leading to a pitch down in which [2] > 110 of the 303 passengers and nine of the 12 crew members were injured; 12 of the occupants were seriously injured and another 39 received hospital medical treatment. HOWEVER .. despite 250 events per second in a 42 litre volume, it took 128 million hours of unit operation to see a failure mode. It's a lot of billiard balls going through a lot of space and a high bar for "something bad" ( just the right bit flip ) to happen. [1] https://www.atsb.gov.au/sites/default/files/media/3532398/ao2008070.pdf https://www.atsb.gov.au/sites/default/files/media/3532398/ao... [2] https://www.atsb.gov.au/publications/investigation_reports/2008/aair/ao-2008-070 https://www.atsb.gov.au/publications/investigation_reports/2... ( Final Report TAB )

nix23 4y ago

>They're so rare, They are not rare, just because you don't detect them (consumer hardware) means nothing. >>study done by IBM in the 1990’s that referenced 1 cosmic ray bit flip per 256MB of memory per month https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-what-flips-your-bit/ https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-...

lioeters 4y ago
One cosmic-ray bit-flip per 256MB per month. Significantly more for computers on aircrafts at higher altitudes. I kinda wish I hadn't learned this fact, how fragile everything is. On the other hand, it makes me appreciate the importance of error tolerance and recovery. > Cosmic ray flux depends on altitude. Computers operated on top of mountains experience an order of magnitude higher rate of soft errors compared to sea level. The rate of upsets in aircraft may be more than 300 times the sea level upset rate. https://en.wikipedia.org/wiki/Soft_error#Cosmic_rays_creating_energetic_neutrons_and_protons https://en.wikipedia.org/wiki/Soft_error#Cosmic_rays_creatin...
- nix23 4y ago
  >One cosmic-ray bit-flip per 256MB per month. Significantly more for computers on aircrafts at higher altitudes. Yes thats why you have max altitude for normal servers/hardware and it's also more disastrous on modern HW -> smaller transistors.
- shadowgovt 4y ago
  The reason we all now wear our seatbelts through the entire flight is there was a cross-Pacific flight that had a sudden altitude drop where several people were injured from rebounding off the ceiling. When the news reported on it, they reported the cause was believed to have been an undetected massive downdraft that shoved the plane down. I lost the thread on that story before the investigation concluded the actual cause was autopilot error. It is, in some ways, more comforting to me to know that the issue wasn't novel atmospheric phenomena, but instead relatively-mundane cosmic radiation flipping one packet of data from the sensors to the avionics that the avionics lacked sufficient redundancy to detect or discard. As a result, the autopilot believed the plane had suddenly pitched 90 degrees and drastically corrected to escape stall.