3 ms·

Soft Errors Are Hard Problems (2009)

11 points by ivank 12y ago

georgecmu 12y ago
Alpha particle strike - two protons plus two electrons, emitted when a heavy radioactive element decays into a lighter element. Alpha particle is the same as a Helium nucleus, so it has 2 more neutrons and 2 fewer electrons than the article claims. Cosmic ray strike - a high energy neutron (or other particle) emitted by the Sun. On the contrary, cosmic rays are very high energy particles mainly originating outside the Solar system. These particles are gradually absorbed by the Earth's atmosphere, so they are more of a problem in orbit and at high altitude. What does it mean for a particle to be "gradually absorbed"?
- thedufer 12y ago
  > What does it mean for a particle to be "gradually absorbed"? I assume it means that the number of particles decreases smoothly as altitude decreases, as opposed to largely being absorbed at distinct altitudes (as one might expect the radiation belts to do, perhaps).
PhantomGremlin 12y ago
There are many techniques that can be used to eliminate soft errors. These techniques used to be quite common in the '60s and '70s when hardware was much less reliable overall.[1] There were companies like Tandem Computers [2] whose rai·son d'être was fault tolerance. But then two things happened. First, chips themselves became much more reliable. Second, semiconductor vendors began adding fault tolerance to standard ICs. E.g. Intel adds ECC to their internal caches. So, IIRC from reading Dark Pools[3], cheap Linux boxes became so reliable that upstart stock exchanges were just running on random white box server hardware located in people's offices. No need to buy million dollar computers any more. No need to buy the highly fault tolerant Itanium. [1] http://en.wikipedia.org/wiki/Fault_tolerant http://en.wikipedia.org/wiki/Fault_tolerant [2] http://en.wikipedia.org/wiki/Tandem_Computers http://en.wikipedia.org/wiki/Tandem_Computers [3] http://en.wikipedia.org/wiki/Scott_Patterson_%28author%29#Dark_Pools http://en.wikipedia.org/wiki/Scott_Patterson_%28author%29#Da...
tzs 12y ago
I used to run a home Linux desktop and a home Linux server. Neither had ECC RAM. I spent something like two years trying to spot a soft error. My detector was simple. I had a process that allocated a fixed size buffer, filled it with a pattern, and then went into a loop. The loop would check to see if the pattern was undisturbed and then sleep 60 seconds. I had this process running on both machines pretty much continuously. Neither ever detected an error in the pattern. I was a bit disappointed. I've been using a 2008 Mac Pro at work since 2008, and a 2009 Mac Pro at home since 2009. Both have ECC RAM. I've never seen an indication that the ECC has actually had to correct anything [1]. [1] I'm not fully sure how I would see if the ECC has corrected a transient error. I've read that it would show up in the DIMM status in system profiler, but I don't check that often, so I could easily miss a transient error.
- PhantomGremlin 12y ago
  > Neither ever detected an error You didn't have a very big sample size. Here's what Google had to say about the topic based on their systems[1]: Our first observation is that memory errors are not rare events. About a third of all machines in the fleet experience at least one memory error per year [1] http://research.google.com/pubs/pub35162.html http://research.google.com/pubs/pub35162.html
  - baruch 12y ago
    I don't have a fleet of systems like Google but on the several few thousand servers I look over, I see the number of ECC errors is lower than that. It is however a strong power law distribution, many servers (>70%) see no errors at all, and a few have so many that they are unusable, some are in between and have undetected performance or other impact.

11 points by ivank 12y ago

Alpha particle strike - two protons plus two electrons, emitted when a heavy radioactive element decays into a lighter element. Alpha particle is the same as a Helium nucleus, so it has 2 more neutrons and 2 fewer electrons than the article claims. Cosmic ray strike - a high energy neutron (or other particle) emitted by the Sun. On the contrary, cosmic rays are very high energy particles mainly originating outside the Solar system. These particles are gradually absorbed by the Earth's atmosphere, so they are more of a problem in orbit and at high altitude. What does it mean for a particle to be "gradually absorbed"?

thedufer 12y ago
> What does it mean for a particle to be "gradually absorbed"? I assume it means that the number of particles decreases smoothly as altitude decreases, as opposed to largely being absorbed at distinct altitudes (as one might expect the radiation belts to do, perhaps).

PhantomGremlin 12y ago

There are many techniques that can be used to eliminate soft errors. These techniques used to be quite common in the '60s and '70s when hardware was much less reliable overall.[1] There were companies like Tandem Computers [2] whose rai·son d'être was fault tolerance. But then two things happened. First, chips themselves became much more reliable. Second, semiconductor vendors began adding fault tolerance to standard ICs. E.g. Intel adds ECC to their internal caches. So, IIRC from reading Dark Pools[3], cheap Linux boxes became so reliable that upstart stock exchanges were just running on random white box server hardware located in people's offices. No need to buy million dollar computers any more. No need to buy the highly fault tolerant Itanium. [1] http://en.wikipedia.org/wiki/Fault_tolerant http://en.wikipedia.org/wiki/Fault_tolerant [2] http://en.wikipedia.org/wiki/Tandem_Computers http://en.wikipedia.org/wiki/Tandem_Computers [3] http://en.wikipedia.org/wiki/Scott_Patterson_%28author%29#Dark_Pools http://en.wikipedia.org/wiki/Scott_Patterson_%28author%29#Da...

tzs 12y ago

I used to run a home Linux desktop and a home Linux server. Neither had ECC RAM. I spent something like two years trying to spot a soft error. My detector was simple. I had a process that allocated a fixed size buffer, filled it with a pattern, and then went into a loop. The loop would check to see if the pattern was undisturbed and then sleep 60 seconds. I had this process running on both machines pretty much continuously. Neither ever detected an error in the pattern. I was a bit disappointed. I've been using a 2008 Mac Pro at work since 2008, and a 2009 Mac Pro at home since 2009. Both have ECC RAM. I've never seen an indication that the ECC has actually had to correct anything [1]. [1] I'm not fully sure how I would see if the ECC has corrected a transient error. I've read that it would show up in the DIMM status in system profiler, but I don't check that often, so I could easily miss a transient error.

PhantomGremlin 12y ago
> Neither ever detected an error You didn't have a very big sample size. Here's what Google had to say about the topic based on their systems[1]: Our first observation is that memory errors are not rare events. About a third of all machines in the fleet experience at least one memory error per year [1] http://research.google.com/pubs/pub35162.html http://research.google.com/pubs/pub35162.html
- baruch 12y ago
  I don't have a fleet of systems like Google but on the several few thousand servers I look over, I see the number of ECC errors is lower than that. It is however a strong power law distribution, many servers (>70%) see no errors at all, and a few have so many that they are unusable, some are in between and have undetected performance or other impact.