11 ms·
For a brief period, the Windows kernel tried to deal with gamma rays
- dekhn 8y agoIf you have a large enough fleet, and log your ECC errors, you have actually built a not-very-sensitive and very expensive scientific instrument- a cosmic ray detector. Physics is awesome.
- pavel_lishin 8y agoAn app was created to use smartphones for this very purpose! https://news.wisc.edu/physicist-turns-smartphones-into-pocket-cosmic-ray-detectors/ https://news.wisc.edu/physicist-turns-smartphones-into-pocke...
- aaaaaaaaaab 8y agoThere used to be a distributed earthquake-detector app too, which correlated HDD seek error statistics from a large number of users.
- russdill 8y agoAs long as you aren't high altitude or near the poles, I think most of your upset events will be from trace radioactive elements in your packaging.
- notacoward 8y agoTo answer the question in the OP: yes, the processor cache might be more susceptible than RAM, if the RAM is ECC. I've heard many stories about bit-flips causing serious problems at higher-elevation sites. Apparently a major installation at either NCAR or UCAR was delayed by a month fighting such problems. While I haven't actually confirmed any of these stories first hand, I've heard enough to believe that a little paranoia is justified.
- blattimwind 8y agoInternal data buses and caches are typically ECC'd. In fact, DRAM and the memory bus are most likely the only non-ECC parts in desktop computers.
- bboreham 8y agoThe way I remember it, when this happened on our Sun E10Ks, the RAM was ECC but the cache was not. Maybe I misremember. It was 20 years ago.
- mjevans 8y agoThat was my thought process as well, that an early silicon spin / chip wasn't ECC because it would change state frequently and the circuit budget wasn't worth a soft error; but the external RAM might be ECC because a longer running state might not have such a refresh/detection of error. I agree that these days I'm unaware of any "serious" CPU where the data isn't at least protected by at least a parity bit on chip. RISC-V : spec mentions ECC, but in a phrasing that makes it clearly optional. https://github.com/riscv/riscv-v-spec/blob/master/v-spec.adoc https://github.com/riscv/riscv-v-spec/blob/master/v-spec.ado... "Many processors use error correction codes in the on-chip cache, including the Intel Itanium and Xeon[28] processors, the AMD Athlon, Opteron, all Zen-[29] and Zen+-based[30] processors (EPYC, EPYC Embedded, Ryzen and Ryzen Threadripper), and the DEC Alpha 21264.[23][31]" https://en.wikipedia.org/wiki/ECC_memory#Cache https://en.wikipedia.org/wiki/ECC_memory#Cache I prefer to always use ECC and layers of defenses where persistence of data is in question. Jeff Atwood's reference does point to other options, assuming that validation is baked in to the storage process and distributed in such a way as to identify and correct errors across a distributed infrastructure rather than a single system; the distributed nature means it could be more resilient and the validation of data at rest / comparing results is arguably a higher level of integrity than just ECC can provide. https://blog.codinghorror.com/to-ecc-or-not-to-ecc/ https://blog.codinghorror.com/to-ecc-or-not-to-ecc/
- blattimwind 8y agoWay back in the late 90s IBM had a problem with alpha-source-contaminated plastics in their SRAM chips. Those chips were used as caches in Sun SPARC processor modules. IBM told some customers, but not Sun. This caused random bitflips in the processor cache, leading to assorted failures and crashes in what was supposed to be reliable UNIX servers.
- effnorwood 8y agoIt did not
- fpgaminer 8y agoSo ... here's what I'm thinking, as a complete layman with respect to how radiation affects memory devices. RAM is DRAM, i.e. dynamic RAM. It has to get automatically refreshed relatively frequently. So, maybe (again, me being a layman) what happens is that usually gamma rays hit a DRAM cell, but haven't imparted enough energy to cause a flip. A millisecond later the cell gets refreshed erasing what little influence the gamma ray had. No harm done. A flip would only occur if enough particles hit the cell within the refresh time frame. That's of course possible, but more rare. Contrast this with processor cache. On-die cache is most likely SRAM, Static RAM. It doesn't get refreshed. So the slight voltage errors caused by gamma rays can slowly build up over time. Perhaps this normally isn't an issue, because even though the cache is SRAM and doesn't get refreshed automatically, it'll get "refreshed" by virtue of being cache. i.e. as long as the processor is busy the cache is constantly getting re-written with new cache lines. But that won't hold true when the processor is asleep. The cache will be sitting idle, making itself susceptible to accumulated charges. Thus the likelihood of a gamma flip is greatly increased. All of that crude logic aside there's one caveat: > he workaround was removed once the problem was fixed in microcode or in a later processor stepping. So ... either everything I said is a load of bollocks and actually this was a processor bug that some CPU engineer mistook as gamma flips, or maybe my theory is correct and they changed the CPU to occasionally wake up and "refresh" its cache automatically. The mystery remains...
- jimrandomh 8y agoNo, SRAM doesn't work that way. DRAM needs refreshing because it's made out of capacitors; if left alone, their charge will stay the same. Whereas SRAM is made of flip-flops; if the charge in an SRAM cell moves towards an intermediate state in between 0 and 1, it will naturally move back to 0 or 1 (whichever is closer) with no refreshing needed, by virtue of its structure.
- ip26 8y agoDRAM needs refreshing because it's made out of capacitors; if left alone, their charge will stay the same. No; capacitors slowly leak and lose their charge. This is why DRAM remains powered when your computer sleeps, eternally refreshing.
- baybal2 8y agoFor people concerned, discover the wonderful thing called 8T SRAM
- nategri 8y agoA pedantic point (good thing I'm on HN), but I wonder if they didn't actually mean muons and not gamma rays?
- jermaustin1 8y agoI doubt think the programmer knew one way or the other, just doing what they were told by the manufacturer, and gamma ray is probably easier to explain to anyone after the 1950s/60s than muon. I mean after all, did any superhero get their powers from muons?
- ketralnis 8y agoOnly Muon Man, but he decayed in 2.2μs so he wasn't very well known
- 21 8y agoApparently 95% of "bit flipping cosmic rays" are neutrons: > At the Earth's surface approximately 95% of the particles capable of causing soft errors are energetic neutrons with the remainder composed of protons and pions. https://en.wikipedia.org/wiki/Soft_error#Cosmic_rays_creating_energetic_neutrons_and_protons https://en.wikipedia.org/wiki/Soft_error#Cosmic_rays_creatin...
- planteen 8y agoSounds about right. I've been a part of a radiation test giving a SoC gamma exposure for total irradiating dose measurement. I didn't see a single upset the whole test.
- nategri 8y agoGreat find. And somewhat surprising because hadrons mostly don't survive to sea level. Too much integrated cross-section with the atmosphere.
- epaulson 8y agoI've heard stories from the supercomputing folks about trying to put their machine rooms underneath parking structures, to get the added protection of layers and layers of concrete overhead.
- xpaulbettsx 8y agoI found this code as an intern at Microsoft, while the manufacturer is hidden in the post, I'll give you a clue - the company starts with "I" and ends with "ntel"
- worldlinx 8y agoUnless you know what you are doing, you might want to delete that. Probably goes against the NDA you've signed, no?
- xpaulbettsx 8y agoThis was over a decade ago, I'm not too worried about it.
- nbsd4life 8y agoThe suggestion to invalidate cache is also in ACPI 6.2A 16.1.1.2 Example 2: S1 Sleeping State Implementation (page 783) http://www.uefi.org/sites/default/files/resources/ACPI%206_2_A_Sept29.pdf http://www.uefi.org/sites/default/files/resources/ACPI%206_2...
- asveikau 8y agoIt's pretty obvious that when people at Microsoft say "processor vendor" it's a euphemism for "Intel". Not as true anymore now that they support arm but... Still kinda true.
- tjoff 8y agoYou do know that the 64 bit version of Windows XP was made for AMD processors before intel had any. We also had Transmeta. I bet all of them have some kinks just as different models from the same vendor have different bugs etc. that require special handling. Meltdown/Spectre is just a recent and very visible artifact that would demonstrate this.
- asveikau 8y ago
- spullara 8y agoIf you don't believe in bit flips, try this! http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html I did that for a bit on cloudfront.net and got dozens of them in a short amount of time.
- amichal 8y agobitflips are also single char typos in ascii. I'm guessing that is what you saw
- Kenji 8y agoRead the article before commenting, please. He addressed that.
- detaro 8y agoI think the article gives good reasons to believe they are not typos, why do you think otherwise?
- amichal 8y agoWhile i like the sibling comment's network errors answer better then mine I didn't see any reasoning other then a unfounded statement[1] in the article or on a quick skim for "typo" in the referenced whitepaper: [1] "These requests were not typos or other manually entered URLs"
- bgirard 8y agoHere's a better quote: "There are many frequently resolved domains that few people type or know." There's a lot more details in the article.
- eli 8y ago"All of these requests used only four domains in the HTTP Host header, as shown in Table 4. Three out of four domains contain more than one bit error, ruling out a simple mistype of fbbdn.net for fbcdn.net. "
- kurtisc 8y agoMicrosoft allow commented-out code in their kernel?
- herogreen 8y agoCould it be that the manufacturer was aware of a bug and chosed to circumvent it by using gamma rays as a pretext ?
- JdeBP 8y agoThat's the implication of the mentions of processor steppings, that almost everyone is ignoring in favour of discussing a more exciting subject. (-:
- discoball 8y agoMaybe the cache should be made using Silicon on Sapphire for radiation hardening? But I am not sure if multi-process silicon fabrication is viable.
- rubenbe 8y agoBit flips are real. I used to see them on my (admittedly low end) webserver. Eg. There were occasional errors like "myOfunction not found". A quick glance on a ASCII table shows that the original function name "my_function" is indeed one bitflip away (0x4F vs 0x5F)
- jandrese 8y agoIf you were seeing errors like that with any regularity there's almost certainly a hardware issue on your server. Cosmic ray bitflips are really rare, and tend to be basically invisible until you're monitoring an entire datacenter.
- datenwolf 8y agoInvalidating the caches is kind of a cringe inducing approach on this (actual) problem. Especially in HPC radiation related single event upsets have become a real problem. If you do the math, all the silicon area devoted to memory (DRAM, caches, registers) adds up, and what you've got is essentially particle detector. Compared to the effective volume of a purpose designed one (ATLAS, CMS, Super Kaminokade, etc.) rather small, but a particle detector nevertheless. A couple of months / years ago, there was an article (also linked here on HN, IIRC) that did a few back of the envelope calculations regarding expected event rates. IIRC it was something on the order of 1 event per day per 10^12 transistors. (EDIT: not the one I thought of but blows the same horn: http://energysfe.ufsc.br/slides/Paolo-Rech-260917.pdf http://energysfe.ufsc.br/slides/Paolo-Rech-260917.pdf ) Also radiation hardened software has been researched (and still is). Essentially the idea is to not only have redundant, error correcting memory, but also redundant, error correcting computation. NASA has some publications on that. e.g. https://ti.arc.nasa.gov/m/pub-archive/1075h/1075%20(Mehlitz).pdf https://ti.arc.nasa.gov/m/pub-archive/1075h/1075%20(Mehlitz)...
- Nokinside 8y agoCould physicists and astronomers use all this distributed particle detection for research? Like a app in a phone or desktop that sends reports back with location and time information.
- ajb 8y agoMost bit flip events are apparently due to alpha particles from radioactive decay in the package (source - worked for a company which used 'low alpha compound' in some of our ICs to put off having to implement HW error correction) , which would be a big confounding signal. I imaging physics experiments work hard to avoid this, or at least correct for it.
- mikeash 8y agoThus the demand for low-background steel, mostly recovered from pre-atomic-age ships: https://en.wikipedia.org/wiki/Low-background_steel https://en.wikipedia.org/wiki/Low-background_steel
- myrandomcomment 8y agoA bunch of years ago Cisco had an issue with some RAM in a new switch model, I think it was in the 65xx. They where crashing randomly but only in certain places in the world. Cisco spent tons of money on this. No idea. They brought in a physics professor. The devices with the most issues were located in countries up near the artic circle. Cosmic Rays caused a bit flip in this particular set of RAM due to something in its design. Sorry for the light details, it's been years. I also worked at a switch manufacturer. We had some ASICs from one of the big companies. Had crashes that we could not explain at all. We knew it was not us. Proved that bits where flipping in the switch ASIC. Turn out they had forgot to spec low alpha solder. Alpha partics will not go through your skin, but when it is layered right on to the chip....oops.
- srcmap 8y agoI used to work for two switch chip companies also. I know both of them has SW/logic in the switch chips guard against random errors in SRAM - "Alpha particles". I have seen detail test report that ran the system in nuclear lab and graph out the level of radiation level vs impact on the system error/recoverable error rate. I knew big customers (such as Cisco) can ask for such test reports and they do get it. It would be very interesting experiment to position some modem/old (7, 14, 28 nm chips) electronics (cell phones, raspberry Pi + solar panels) to various distant near nuclear accident site, monitor them remotely, classify and log the failure rate over the months/ years.
- myrandomcomment 8y agoWe would take the switches to the labs and have the "shoot" various types of particles other radiation sources at it to check resistance in both HW and SW. Depending on what you did sometime it takes months get get the device back as it has to "cool down". It was pretty neat.
- russdill 8y agoXilinx tracks a ton of data in regard to their products, packaging, and bit upsets: https://forums.xilinx.com/t5/PLD-Blog-Archived/Package-Generated-Alpha-Particles-They-are-a-Problem-and-Here-is/ba-p/54029 https://forums.xilinx.com/t5/PLD-Blog-Archived/Package-Gener...
- yongjik 8y agoI once saw a postmortem where a server process mysteriously tried to delete whole data (fortunately no actual data was lost). After much confusion, the conclusion was that a cosmic ray flipped a single bit in a register, making it point to 8 bytes past the correct address in C++ virtual function table. As a result, instead of calling UpdateRow(), the process executed DeleteTable(). Of course cosmic rays don't exactly leave a trace, so we will never know.
- weberc2 8y agoWhen I was fresh out of college, I worked as a contractor for a prominent agricultural equipment manufacturer. I was responsible for building out the touch-screen interface for the radio (a Qt app). I was told by an engineer who worked for the equipment manufacturer that my application wasn't good enough because needed to be able to operate correctly in the face of arbitrary bit flips "from lightning strikes"--I kindly asked her to show me the requirements which was sufficient to get her to relent, but that was still the wackiest request I've ever received.
- russdill 8y agoThe requirement request is a great way to push back on feature creep. There's a lot of cargo culting that goes on in the "protection against bit-flips". You sometimes have to go a step further and ask what error rate are you required to be below. Once you have that number, you can start asking what your current error rate is without mitigations, and how much a given mitigation will reduce your error rate.
- vvanders 8y agoMy favorite entry in that problem space is metastability[1]. Do you interface two different clock domains(which is basically most things)? Guess what all of your computing is built on the "chance" that bits won't flip. Granted, statistics make this pretty solid but kinda blew my mind when I first stumbled across it. [1] https://en.wikipedia.org/wiki/Metastability_(electronics) https://en.wikipedia.org/wiki/Metastability_(electronics)
- russdill 8y agoYup, a large portion of hardware design is based on getting below a required maximum failure rate. For metastability, you just keep adding more flip-flops. BTW, the cache invalidation request may be due to this. They figured they could more easily reach their time between failure interval if they could discount time during S1.
- drharby 8y ago
- trelliscoded 8y agoEmbedded people deal with this all the time. One class of solutions involves a checker task running continuously, which verifies the integrity of the data structures, kind of like a poor man's ECC. Really important code generally does everything three times, so there's a tie breaker in case there's a temporary fault in code or memory. I've seen this done with macros in ways that result in pretty wild code, like running a computation three times, storing each of those results three times, and then comparing the resulting nine outputs three times. That was in a diving related application, so it's not crazy to do all that work over and over since it had to be right. Complex embedded systems like your cellphone's baseband processor usually just give up at some point and suicide a task or even the whole OS if they detect a problem. For a while I had a Qualcomm debugger attached to the internal cell modem I had in a netbook I was working on, and the baseband crashed all the time due to hardware faults. I thought I had a bad chip for a while until I realized it never happened when I left it in an underground parking lot.
- ZakNichol 8y agoThis. Cache is the least of their worries in aerospace. It's common to see satellite IC's dosed up so high on the ol' Gamma that the silicon MOSFET junctions themselves start disintegrating.
- jacobush 8y agoOr for really serious stuff, 3 processors running 3 implementations from the same spec, voting.
- nobrains 8y agoRelevant: https://en.wikipedia.org/wiki/Timothy_C._May https://en.wikipedia.org/wiki/Timothy_C._May May is most noted for having identified the cause of the "alpha particle problem", which was affecting the reliability of integrated circuits as device features reached a critical size where a single alpha particle could change the state of a stored value