2 ms·

A bunch of years ago Cisco had an issue with some RAM in a new switch model, I think it was in the 65xx. They where crashing randomly but only in certain places

A bunch of years ago Cisco had an issue with some RAM in a new switch model, I think it was in the 65xx. They where crashing randomly but only in certain places in the world. Cisco spent tons of money on this. No idea. They brought in a physics professor. The devices with the most issues were located in countries up near the artic circle. Cosmic Rays caused a bit flip in this particular set of RAM due to something in its design. Sorry for the light details, it's been years. I also worked at a switch manufacturer. We had some ASICs from one of the big companies. Had crashes that we could not explain at all. We knew it was not us. Proved that bits where flipping in the switch ASIC. Turn out they had forgot to spec low alpha solder. Alpha partics will not go through your skin, but when it is layered right on to the chip....oops.

srcmap 8y ago
I used to work for two switch chip companies also. I know both of them has SW/logic in the switch chips guard against random errors in SRAM - "Alpha particles". I have seen detail test report that ran the system in nuclear lab and graph out the level of radiation level vs impact on the system error/recoverable error rate. I knew big customers (such as Cisco) can ask for such test reports and they do get it. It would be very interesting experiment to position some modem/old (7, 14, 28 nm chips) electronics (cell phones, raspberry Pi + solar panels) to various distant near nuclear accident site, monitor them remotely, classify and log the failure rate over the months/ years.
- myrandomcomment 8y ago
  We would take the switches to the labs and have the "shoot" various types of particles other radiation sources at it to check resistance in both HW and SW. Depending on what you did sometime it takes months get get the device back as it has to "cool down". It was pretty neat.
russdill 8y ago
Xilinx tracks a ton of data in regard to their products, packaging, and bit upsets: https://forums.xilinx.com/t5/PLD-Blog-Archived/Package-Generated-Alpha-Particles-They-are-a-Problem-and-Here-is/ba-p/54029 https://forums.xilinx.com/t5/PLD-Blog-Archived/Package-Gener...
jhallenworld 8y ago
Low alpha solder used to come from lead ballast lost at sea. This was supposedly a valuable resource for deep sea treasure hunters.

srcmap 8y ago

I used to work for two switch chip companies also. I know both of them has SW/logic in the switch chips guard against random errors in SRAM - "Alpha particles". I have seen detail test report that ran the system in nuclear lab and graph out the level of radiation level vs impact on the system error/recoverable error rate. I knew big customers (such as Cisco) can ask for such test reports and they do get it. It would be very interesting experiment to position some modem/old (7, 14, 28 nm chips) electronics (cell phones, raspberry Pi + solar panels) to various distant near nuclear accident site, monitor them remotely, classify and log the failure rate over the months/ years.

myrandomcomment 8y ago
We would take the switches to the labs and have the "shoot" various types of particles other radiation sources at it to check resistance in both HW and SW. Depending on what you did sometime it takes months get get the device back as it has to "cool down". It was pretty neat.

russdill 8y ago

Xilinx tracks a ton of data in regard to their products, packaging, and bit upsets: https://forums.xilinx.com/t5/PLD-Blog-Archived/Package-Generated-Alpha-Particles-They-are-a-Problem-and-Here-is/ba-p/54029 https://forums.xilinx.com/t5/PLD-Blog-Archived/Package-Gener...

jhallenworld 8y ago

Low alpha solder used to come from lead ballast lost at sea. This was supposedly a valuable resource for deep sea treasure hunters.