4 ms·

There is another mechanism called "Single Event Upset" (SEU) or "Single Event Effects" (SEE) (basically synonymous). This is due to cosmic rays. On the surfac

There is another mechanism called "Single Event Upset" (SEU) or "Single Event Effects" (SEE) (basically synonymous). This is due to cosmic rays. On the surface of the earth, the effect is mostly abated by the atmosphere - except for neutrons. As you go higher in the atmosphere (say on a mountaintop, or an airplane, or go into space) it becomes worse because of other charged particles that are no longer attenuated by the atmosphere. The typical issue at sea level is from neutrons hitting silicon atoms. If a neutron hits the neucleus in some area of the microprocessor circuitry, it suddenly recoils, basically causing an ionizing trail of several microns in length. Given transistors are now measured in 10s of nanometers, the ionizing path can cross many nodes in the circuit and create some sort of state change. Best case it happens in a single bit of a memory that has error correction and you never notice it. Worst case it causes latchup (power to ground short) in your processor and your CPU overheats and fries. Generally you would just notice it as a sudden error that causes the system to lock up, you'd reboot and it would come back up and be fine, leaving you with a vague thought of, "That was weird".

Florin_Andrei 11y ago
> This is due to cosmic rays. On the surface of the earth, the effect is mostly abated by the atmosphere - except for neutrons. So, if we had more hydrogen (either free or compounds) in the air this would not be the case, right? The column of air on top of your head is equivalent (in terms of mass) to a column of water 10 meters tall, with the same base section area. But the composition is quite different, of course - the only major component they have in common is oxygen.
- gh02t 11y ago
  Cosmic rays have energies in the tens to hundreds MeV range where pair production is the dominant attenuation mechanism. The probability of a photon inducing pair production in a material is roughly proportional to the square of the proton number, ergo cosmic rays don't give a shit about hydrogen (which has the smallest proton number possible). Even if the atmosphere was pure hydrogen gas at STP, the average distance traveled by a 20 MeV cosmic ray would be around 17km. http://physics.nist.gov/PhysRefData/XrayMassCoef/ElemTab/z01.html http://physics.nist.gov/PhysRefData/XrayMassCoef/ElemTab/z01... Hydrogen is pretty good at moderating neutrons down to thermal energies (eV range, ie room temperature) via elastic scattering, but gasses don't really have enough density to do a very good job. If you really want to protect something from neutrons you just coat it with boron. A mm coating of the stuff will keep out pretty much any common source of neutrons.
  - Florin_Andrei 11y ago
    Right, I was thinking about the neutrons. Maybe something like methane would be more efficient than pure hydrogen. Anyway, it's just a thought. It's very surprising to see how efficient boron is. I thought neutron shields (paraffin, water) are supposed to be very thick. Maybe boron does the job via a different mechanism?
    - gh02t 11y ago
      Boron is almost exclusively an absorber, so the boron nucleus captures the neutron and it basically disappears. Hydrogen is primarily a moderator, so it reduces the energy via elastic scattering, but a significant number of very low energy neutrons still escape (some absorption to produce deuterium also occurs). Thermal neutrons can still cause damage, but they're easy to block with a subsequent thin layer of lead or something like that. Boron's probability to capture a neutron is astronomically high, that's why you can get away with so little. Environmental sources of neutrons are actually pretty rare normally and most neutrons you do see will be pretty low energy and won't have a huge amount of penetrative power. A thin layer of boron will pretty much stop them. Pyrex (like the stuff baking dishes are made of, which is borosilicate glass - glass with boron added) is actually commonly used as control material in nuclear reactors.
harshreality 11y ago
US Patent 7309866 - Cosmic ray detectors for integrated circuit chips - Intel (applied for 2004-06, issued 2007-12) http://www.google.com/patents/US7309866 http://www.google.com/patents/US7309866 Is Intel using any such things in their commodity chips?
eridius 11y ago
How often does this sort of thing actually happen in real life? Or rather, what's the chance that some given computer will experience one of these events in its operational lifetime (or, if the chance is actually high enough, how many such events would it be expected to see on average given a lifespan of several years)?
- fezz 11y ago
  On CCD/CMOS sensors it can happen enough to be visible as dead columns or pixels. Pixels can be calibrated out. Most notably, gamma rays killing columns was the suspected cause of dead columns while filming Superman a few years back. As a result, cameras were shipped by ship rather than air to minimize this happening ( a bit of a reactionary tale to this particular incident and not generally what happens).
  - fezz 11y ago
    Forgot to mention... in talking with NASA about deploying cameras on the space station, the issue being able to fix pixel death by gamma ray is more relevant. Water is apparently one defense against it but not so practical. How much water would you need?
    - TheLoneWolfling 11y ago
      Water has a halving thickness of ~18cm for gamma rays. So a fair bit. Unfortunately, most other things are also of the same order of magnitude of ~20g/cm^2 - with gamma rays the single most important thing is just "how much mass is in the way". Which is exactly what you don't want.
- sliverstorm 11y ago
  Somewhere in the range that your laptop will almost certainly never see even a single event, but a very large datacenter or colo will have multiple events a month. There is a lot of disagreement on bitflips from ionizing radiation. They are unequivocally real, and unequivocally very rare. Even when they do happen, a large portion of the chip is dark a lot of the time, and a lot of the live data in the chip is simply thrown away and never used. (Think prefetching) Some bits, if flipped, will break something but will not corrupt the disk and the machine will be able to recover. Nobody really knows for certain exactly how big of a problem they are and how often they happen- it's all statistics, and it depends on things like where on the globe your computer is, what your building is made of, and what phase of the solar cycle we are in. It even depends on workload. Anybody who claims to know for certain...
hyperpallium 11y ago
As process nodes shrink and this happens more often, perhaps we'll eventually have to move to a more probability-based model of computation, giving up classical predictability.
- wsxcde 11y ago
  Well, this is what researchers were predicting 12-14 years ago. There was a lot of work in fault tolerant and probabilistic computer architectures as a result of this. (I believed the prediction and contributed to some of this work.) The prediction was that chips below 32nm wouldn't work reliably and the only option would be to use these exotic architectures. Well, here we are at 14nm and everything seems to be going okay.
elwell 11y ago
Next time a surprising bug pops up on my server I'll just blame it on cosmic rays and reboot.
- rhizome 11y ago
  "Sunspots" is a very old sysadmin joke.
  - rbritton 11y ago
    I actually have had one user take me seriously with that one. It's been a while, but I believe I also attributed her problem to the phase of the moon as well.
    - barrystaes 11y ago
      I tend to deduce and eventually blame either gravity or gremlins. I was never wrong.

Florin_Andrei 11y ago

> This is due to cosmic rays. On the surface of the earth, the effect is mostly abated by the atmosphere - except for neutrons. So, if we had more hydrogen (either free or compounds) in the air this would not be the case, right? The column of air on top of your head is equivalent (in terms of mass) to a column of water 10 meters tall, with the same base section area. But the composition is quite different, of course - the only major component they have in common is oxygen.

gh02t 11y ago
Cosmic rays have energies in the tens to hundreds MeV range where pair production is the dominant attenuation mechanism. The probability of a photon inducing pair production in a material is roughly proportional to the square of the proton number, ergo cosmic rays don't give a shit about hydrogen (which has the smallest proton number possible). Even if the atmosphere was pure hydrogen gas at STP, the average distance traveled by a 20 MeV cosmic ray would be around 17km. http://physics.nist.gov/PhysRefData/XrayMassCoef/ElemTab/z01.html http://physics.nist.gov/PhysRefData/XrayMassCoef/ElemTab/z01... Hydrogen is pretty good at moderating neutrons down to thermal energies (eV range, ie room temperature) via elastic scattering, but gasses don't really have enough density to do a very good job. If you really want to protect something from neutrons you just coat it with boron. A mm coating of the stuff will keep out pretty much any common source of neutrons.
- Florin_Andrei 11y ago
  Right, I was thinking about the neutrons. Maybe something like methane would be more efficient than pure hydrogen. Anyway, it's just a thought. It's very surprising to see how efficient boron is. I thought neutron shields (paraffin, water) are supposed to be very thick. Maybe boron does the job via a different mechanism?
  - gh02t 11y ago
    Boron is almost exclusively an absorber, so the boron nucleus captures the neutron and it basically disappears. Hydrogen is primarily a moderator, so it reduces the energy via elastic scattering, but a significant number of very low energy neutrons still escape (some absorption to produce deuterium also occurs). Thermal neutrons can still cause damage, but they're easy to block with a subsequent thin layer of lead or something like that. Boron's probability to capture a neutron is astronomically high, that's why you can get away with so little. Environmental sources of neutrons are actually pretty rare normally and most neutrons you do see will be pretty low energy and won't have a huge amount of penetrative power. A thin layer of boron will pretty much stop them. Pyrex (like the stuff baking dishes are made of, which is borosilicate glass - glass with boron added) is actually commonly used as control material in nuclear reactors.

harshreality 11y ago

US Patent 7309866 - Cosmic ray detectors for integrated circuit chips - Intel (applied for 2004-06, issued 2007-12) http://www.google.com/patents/US7309866 http://www.google.com/patents/US7309866 Is Intel using any such things in their commodity chips?

eridius 11y ago

How often does this sort of thing actually happen in real life? Or rather, what's the chance that some given computer will experience one of these events in its operational lifetime (or, if the chance is actually high enough, how many such events would it be expected to see on average given a lifespan of several years)?

fezz 11y ago
On CCD/CMOS sensors it can happen enough to be visible as dead columns or pixels. Pixels can be calibrated out. Most notably, gamma rays killing columns was the suspected cause of dead columns while filming Superman a few years back. As a result, cameras were shipped by ship rather than air to minimize this happening ( a bit of a reactionary tale to this particular incident and not generally what happens).
- fezz 11y ago
  Forgot to mention... in talking with NASA about deploying cameras on the space station, the issue being able to fix pixel death by gamma ray is more relevant. Water is apparently one defense against it but not so practical. How much water would you need?
  - TheLoneWolfling 11y ago
    Water has a halving thickness of ~18cm for gamma rays. So a fair bit. Unfortunately, most other things are also of the same order of magnitude of ~20g/cm^2 - with gamma rays the single most important thing is just "how much mass is in the way". Which is exactly what you don't want.
sliverstorm 11y ago
Somewhere in the range that your laptop will almost certainly never see even a single event, but a very large datacenter or colo will have multiple events a month. There is a lot of disagreement on bitflips from ionizing radiation. They are unequivocally real, and unequivocally very rare. Even when they do happen, a large portion of the chip is dark a lot of the time, and a lot of the live data in the chip is simply thrown away and never used. (Think prefetching) Some bits, if flipped, will break something but will not corrupt the disk and the machine will be able to recover. Nobody really knows for certain exactly how big of a problem they are and how often they happen- it's all statistics, and it depends on things like where on the globe your computer is, what your building is made of, and what phase of the solar cycle we are in. It even depends on workload. Anybody who claims to know for certain...

hyperpallium 11y ago

As process nodes shrink and this happens more often, perhaps we'll eventually have to move to a more probability-based model of computation, giving up classical predictability.

wsxcde 11y ago
Well, this is what researchers were predicting 12-14 years ago. There was a lot of work in fault tolerant and probabilistic computer architectures as a result of this. (I believed the prediction and contributed to some of this work.) The prediction was that chips below 32nm wouldn't work reliably and the only option would be to use these exotic architectures. Well, here we are at 14nm and everything seems to be going okay.

elwell 11y ago

Next time a surprising bug pops up on my server I'll just blame it on cosmic rays and reboot.

rhizome 11y ago
"Sunspots" is a very old sysadmin joke.
- rbritton 11y ago
  I actually have had one user take me seriously with that one. It's been a while, but I believe I also attributed her problem to the phase of the moon as well.
  - barrystaes 11y ago
    I tend to deduce and eventually blame either gravity or gremlins. I was never wrong.