10 ms·
Cosmic rays causing 30k network malfunctions in Japan each year
- neonological 5y agohttps://www.wnycstudios.org/podcasts/radiolab/articles/bit-flip https://www.wnycstudios.org/podcasts/radiolab/articles/bit-f... A more comprehensive an interesting take on the cosmic ray phenomenon. It’s actually a global issue that is accounted for in certain electronics.
- fennecfoxen 5y ago> Soft errors occur when the data in an electronic device is corrupted after neutrons, produced when cosmic rays hit oxygen and nitrogen in the earth's atmosphere, collide with the semiconductors within the equipment. Uhhhhhhh. Neutrons? Doubt goes here. I was pretty sure charged particles and ionizing radiation are to blame for messing up your electronics.
- fennecs 5y agoSupposedly yes neutrons [0 they give a link under causes]. I guess neutrons are more penetrating than protons, can pass through chips/buildings etc more easily, and even though weak interaction, it ends up more significant contribution than protons etc that interact more easily. [0] https://en.m.wikipedia.org/wiki/Soft_error https://en.m.wikipedia.org/wiki/Soft_error
- zaarn 5y agoNeutron radiation is in fact ionizing radiation. It is much more dangerous than other types, as it readily passes most materials, yet dosage is absorbed. Absorbed neutrons cause particles to (almost always) become radioactive, which causes havoc as those new radioactive particles decay. The other common types of radiation (alpha, beta and gamma) do not cause things to become radioactive (usually), instead they are a threat if the actually radioactive material accumulates as dust on your skin (for example). Neutron radiation is much more dangerous because it seeds radioactivity deep in the body, so you can't clean it off by washing or waiting for the absorbed particles to be flushed back out of the body naturally within a few days.
- yummypaint 5y agoOne of the ways high energy neutrons interact is by scattering protons elastically. Their masses are essentially the same so the neutron can transfer its momentum very efficiently. These scattered protons (perhaps from plastic in the packaging etc) are then the charged particles that flip bits. These protons are generally lower energy than cosmic ray protons, which means they deposit MORE energy per distance traveled, so they can be worse than cosmics directly.
- stuartcw 5y agoI have always wondered whether new ideas are seeded by cosmic rays disturbing neurons in the brain..
- adenozine 5y agoThat's actually mind blowing to think about. Thanks for sharing this.
- SketchySeaBeast 5y ago> That's actually mind blowing to think about. That would be too many cosmic rays. And the "flipped the wrong neuron" possibility is much scarier in my mind. And probably in everyone else's brain as well.
- deepsun 5y agoThat'd mean people underground (or under thick layer of concrete, like in skyscrapers) are not creative.
- dleslie 5y agoSounds like we have a hypothesis and a test environment; now we just need a test for creativity and we can start doing science!
- supernova87a 5y agoBut grad students are often sent to work in the basement, and yet they need to have the most ideas!
- moistbar 5y agoI guess you could say they'd be... Living under a rock. Yeah, yeah, I'm leaving. Put the weapons away...
- AlanSE 5y agoI suspect that modern processors are closer to quantum limits than what neurons are.
- shiftpgdn 5y agoKind of a fun ancedote: In "the old days" when you'd run a server room in your office we had a very large HPC cluster + a significant amount of storage and other one off servers on the top floor of a mid rise office building. I eventually moved it all to a former nuclear fallout facility where our systems were three floors down and under a gigantic pool of water. Error rates and random crashes fell off IMMEDIATELY. I believe Microsoft reported similar findings with their submerged data center.
- kevin_nisbet 5y agoYea, when looking into these myself I did float in conversation that we'd need build a giant pool over our data centers. It was more of a joke though and never went anywhere. Although our error rate was more like 1-2 per week on the equipment I was looking at.
- sandworm101 5y agoPools are expensive. The same weight of gravel or sand would produce a similar effect. Put the server farm under a multi-level car park structure.
- shiftpgdn 5y agoWater is one of the most effective radiation shields we have for cosmic rays. NASA has contemplated using a water shield for Mars astronauts. There is a reason most fallout shelters are built under a pool when possible.
- sandworm101 5y agoNasa has also contemplated using human feces as shielding. The mixed use, dual purpose systems for spaceflight are a special case. What really matters is mass. Sand is cheaper than pool water.
- kevin_nisbet 5y ago
- belter 5y agoThey have to do two things :-) 1) Use ECC memory 2) Go underground "One experiment measured the soft error rate at the sea level to be 5,950 failures in time (FIT = failures per billion hours) per DRAM chip. When the same test setup was moved to an underground vault, shielded by over 50 feet (15 m) of rock that effectively eliminated all cosmic rays, zero soft errors were recorded.[6] In this test, all other causes of soft errors are too small to be measured, compared to the error rate caused by cosmic rays." "Soft Errors" https://en.wikipedia.org/wiki/Soft_error#Cosmic_rays_creating_energetic_neutrons_and_protons https://en.wikipedia.org/wiki/Soft_error#Cosmic_rays_creatin...
- contravariant 5y agoThat's a bit more than just 'go underground'. I've seen people dig for cables but I've never seen them dig a hole 15m deep.
- tyingq 5y agoPolyethylene is supposedly good at blocking cosmic rays. It would be funny to me if the fix was just a drop ceiling full of old grocery bags.
- robbmorganf 5y agoHave a source? That seems like an awesome way to recycle grocery bags.
- 1-6 5y agohttps://www.nature.com/articles/s41598-017-01707-2 https://www.nature.com/articles/s41598-017-01707-2
- tyingq 5y agoThe peer comment posted a source that has lots of references. Though most of them are tests in space, where I assume the thickness requirements would rule out grocery bags. I am curious how thick a layer of HDPE you would need on earth to make any notable difference.
- KWxIUElW8Xt0tD9 5y agoHow many are caused by Windows? :-)
- dole 5y agoCosmic bitflip is like the Godwin's Law of BOFH excuses.
- arbitrage 5y agoAhh, the ol' Cisco get-out-of-jail-free card. Headline would be more accurate if it said "30K unidentified bugs blamed on unverifiable phenomenon each year in Japan".
- 1-6 5y agoI'm speculating here but it would be more likely that software bugs cause more errors than cosmic rays.
- dnautics 5y agoFound the rust programmer.
- marcosdumay 5y agoThe rust programmer is the one out there ending that phrase with "and this a problem that must be solved". ... Well, the rust programmer and the nuclear engineer, possibly.
- lovemenot 5y agoIncorrect. According to the article NTT experimented with neutrons (simulated cosmic rays) in a lab. NTT systematically set out to be able to distinguish these different kinds of error, and you go ahead and ignore that effort.
- notum 5y agoCosmic rays are, or rather were, the reason why manufactures avoid shipping high end cameras by plane, cosmic rays kill pixels in imaging sensors: https://youtu.be/98FZ8C6HneE?t=479 https://youtu.be/98FZ8C6HneE?t=479 It's also the reason why all ISS footage has dead pixels on it.
- andy_ppp 5y agoVery interesting, so should people who travel with photography equipment take night flights if possible?
- notum 5y agoRays hit our planet regardless of Sun, if anything it might block a few. They are caused by ultra-energetic cosmic events, such as star collisions, black hole feeding cycles. Cosmic rays are just very potent photons, capable of knocking out electrons from an atom (meaning they are ionizing) causing havoc in precision electronics and, well, our DNA.
- moftz 5y agoCosmic rays can also be high energy ions. Most of the photons come from the sun and the ions can come from outside the universe and the sun.
- andy_ppp 5y agoWhat is the tolerance for cosmic rays for a human... I ask because on the journey to Mars you’ll get quite a few I guess and also on the planet’s surface after you’re there with such a thin atmosphere?
- tinco 5y agoThis is actually a thing people are criticizing Musk for, he is being accused of sending people to Mars with insufficient shielding against cosmic rays. In my opinion it's a bit silly argument, as there's a whole bunch of other risks and quality of life sacrifices made by the persons who are going to undertake that journey. Some raised chance of cancer is probably the least of their worries.
- 1-6 5y agoAs we move to 5nm chips, I wonder if there's a greater likelihood of errors from background radiation.
- sosuke 5y agoThis is a great place to ask. Are our brains susceptible to this kind of interference and if so do you think the structure evolved to handle some corruption of this kind?
- Serenntop 5y agoWe could reduce the cosmic ray flux by thickening the atmosphere. A small increase in normal air density would give us a significant reduction in cosmic rays. Talk to the terraformers. They've given a lot of thought to how to make air.
- jstrieb 5y agoThis reminds me of Bitsquatting, which is like typosquatting but uses domains one bit-flip away instead of one mistyped letter away.[0] In a small experiment, for some popular domains, the author reportedly got a lot of hits. In the associated DEFCON talk, the author notes that even if you use ECC memory, it's unlikely that the DRAM in your hard drives does too.[1] 0: http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html 1: https://www.youtube.com/watch?v=9WcHsT97suU https://www.youtube.com/watch?v=9WcHsT97suU
- hyko 5y agoIs appeal to cosmic rays a candidate for an engineering logical fallacy?
- devwastaken 5y agoI'm curious if anyone has ran a year long memtest and seen what happens. That, and if they can perhaps measure the actual amount of particles coming into contact with the system.
- dekhn 5y agoMany computer problems are probabilistic in nature with very low occurrence, and it can be very challenging to come to any real conclusions about the cause and outcome of various process failures. While I believe cosmic rays are a problem, the issue I'm dealing with now is bad silicon- when you make a bunch of chips (millions), some fraction of them will compute a few operations incorrectly sometimes. Post-manufacture validation doesn't catch all problems and ultimately some machines slip in the serving fleet. ML People who train on this hardware report more NaNs causing training failure than expected due to software bugs. It's extremely challenging to debug because most ML codes are very robust to small amounts of injected noise, especially gaussian independent noise (there's literature showing that introducing random numbers in training often helps training go faster). This is a fascinating area and there aren't a lot of people who can really make forward progress in it.
- ChrisArchitect 5y agomeanwhile, here is a 230 megapixel photo of the sun I saw just this week: https://twitter.com/AJamesMcCarthy/status/1379461549119574016 https://twitter.com/AJamesMcCarthy/status/137946154911957401...