11 ms·
Ask HN: Why don't transistors in microchips fail?
Considering that a Quad-core + GPU Core i7 Haswell has 1.4e9 transistors inside, even given a really small probability of one of them failing, wouldn't this be catastrophic?
Wouldn't a single transistor failing mean the whole chip stops working? Or are there protections built-in so only performance is lost over time?
- deleted 11y ago[deleted]
- mchannon 11y agoGenerally, yes, a failing transistor can be a fatal problem. This relates to "chip yield" on a waferfull of chips. Faults don't always manifest themselves as a binary pass/fail result; as chip temperatures increase, transistors that have faults will "misfire" more often. As long as this temperature is high enough, these lower-grade chips can be sold as lower-end processors that never in practice reach these temperatures. Am not aware of any redundancy units in current microprocessor offerings but it would not surprise me; Intel did something of this nature with their 80386 line but it was more of a labeling thing ("16 BIT S/W ONLY"). Solid state drives, on the other hand, are built around this protection; when a block fails after so many read/write cycles, the logic "TRIM"s that portion of the virtual disk, diminishing its capacity but keeping the rest of the device going.
- ajross 11y ago> Am not aware of any redundancy units in current microprocessor offerings Sure there are. That's why Intel sells chips with a thousand different cache sizes, for example. Bad bit in the cache? Just turn that block off. Likewise for whole cores in some of the bigger chips, I believe.
- frik 11y agoThe Xbox360 had four cores (IBM Power) but only three cores were enabled. And older AMD CPUs were released with 6 enabled cores out of 8 cores.
- GTP 11y agoWow, that's interesting! Do you know if somebody tried to enable one of this "extra cores"?
- jotm 11y agoHaha, of course they did. http://www.tomsguide.com/forum/id-2142984/amd-phenom-710-4th-core-unlock.html http://www.tomsguide.com/forum/id-2142984/amd-phenom-710-4th... AMD used to software-disable the 4th core on the Phenoms, but then they switched to disabling them by hardware, so that rendered any software means useless. AMD only disabled the cores to save on manufacturing - most of the time, these cores were not working properly. But as the process got better, people got 4 or more cores for the price of 2-3 :-D
- jevinskie 11y agoThe PS3 disabled 1/8 SPUs for yield and you can try to reenable it: http://ps3devwiki.com/ps3/Unlocking_the_8th_SPE http://ps3devwiki.com/ps3/Unlocking_the_8th_SPE Good luck if it turns out the disabled SPU is actually bad!
- ixtli 11y agoI seem to recall the ps4's Cell shipping with 2 out of 8 cores disabled because of the failure rate per wafer.
- scott_s 11y agoI spoke with the project lead for the XBox 360 chip, and I asked him: Why three cores? It's a strange number. He said it was to make Christmas.
- provemewrong 11y agoAMD was particularly fond of selling dual and even tripple core CPUs that were quad core chips with some cores disabled. And people where unclocking these cores for profit, because they speculated that AMD disables core for the whole batch even if it was a single unit from it that failed the test, so there could be possibly plenty of good cores disabled along the way.
- ajross 11y agoYes, they can fail. Lots and lots of them fail immediately due to manufacturing defects. And over time, electromigration (where dopant atoms get kicked out of position by interaction with electron momentum) will slowly degrade their performance. And sometimes they fail due to specific events like an overheat or electrostatic discharge. But the failure rate after initial burn-in is phenomenally low. They're solid state devices, after all, and the only moving parts are electrons.
- jflatow 11y agoAs far as I understand, chips are designed with many redundant pathways for exactly this reason. I think its a rather important part of the design process, but someone who knows better should probably say.
- valarauca1 11y agoNot exactly. Reduancy is designed into chip hardware not to account for cosmic rays striking the wrong trace, but for manufacturing defects. For example Nvidia runs it's GTX 980 and GTX 970 production at the same time. The only difference is GTX970's can have up to 2 of their compute units non-functional. This is very commonly done in the industry. If you remember Phenom Dual, Tri, and Quadcores. Which were the same chip, just it was expected that only 5% of produced chips would be fully featured quad cores, the rest would be sold as other core counts. This was done with 27xx series i5, which were 34xx core i7's but with hyper threading disabled due to issues with yields on dye shrinks. If a single transistor fails, normally the whole thing dies.
- bri3d 11y agoThey're designed for manufacturing defects - this process is called DFM (Design For Manufacturing) and usually revolves around performing a Critical Area Analysis for defects of a given size. Design software is used to determine what would happen if a speck of dust of a certain diameter landed on the die in given locations. Then, the critical areas are spaced out and moved around to attempt to balance design constraints with yield. For large, expensive parts or parts in which a single common defect could easily blow the whole yield (for example DRAM, especially when embedded, CPUs with lots of cache or cores, and so on), regions (or rows and columns of memory) are generally fused off so that if one specific region fails qualification, it can be disabled without discarding the whole chip. This is the source of most 3-core CPUs, as well as the difference between most models in a single CPU family (they're often binned off based on how much of their L2 cache actually works). However, once parts are manufactured and qualified, they're pretty much done. Some hardware has BISR (Built In Self Repair) but as far as I know it's not particularly common outside of DRAM.
- rhino369 11y agoExtremely good R&D done by semiconductor companies. It's frankly amazing how good they are.
- joelaaronseely 11y agoThere is another mechanism called "Single Event Upset" (SEU) or "Single Event Effects" (SEE) (basically synonymous). This is due to cosmic rays. On the surface of the earth, the effect is mostly abated by the atmosphere - except for neutrons. As you go higher in the atmosphere (say on a mountaintop, or an airplane, or go into space) it becomes worse because of other charged particles that are no longer attenuated by the atmosphere. The typical issue at sea level is from neutrons hitting silicon atoms. If a neutron hits the neucleus in some area of the microprocessor circuitry, it suddenly recoils, basically causing an ionizing trail of several microns in length. Given transistors are now measured in 10s of nanometers, the ionizing path can cross many nodes in the circuit and create some sort of state change. Best case it happens in a single bit of a memory that has error correction and you never notice it. Worst case it causes latchup (power to ground short) in your processor and your CPU overheats and fries. Generally you would just notice it as a sudden error that causes the system to lock up, you'd reboot and it would come back up and be fine, leaving you with a vague thought of, "That was weird".
- Florin_Andrei 11y ago> This is due to cosmic rays. On the surface of the earth, the effect is mostly abated by the atmosphere - except for neutrons. So, if we had more hydrogen (either free or compounds) in the air this would not be the case, right? The column of air on top of your head is equivalent (in terms of mass) to a column of water 10 meters tall, with the same base section area. But the composition is quite different, of course - the only major component they have in common is oxygen.
- gh02t 11y agoCosmic rays have energies in the tens to hundreds MeV range where pair production is the dominant attenuation mechanism. The probability of a photon inducing pair production in a material is roughly proportional to the square of the proton number, ergo cosmic rays don't give a shit about hydrogen (which has the smallest proton number possible). Even if the atmosphere was pure hydrogen gas at STP, the average distance traveled by a 20 MeV cosmic ray would be around 17km. http://physics.nist.gov/PhysRefData/XrayMassCoef/ElemTab/z01.html http://physics.nist.gov/PhysRefData/XrayMassCoef/ElemTab/z01... Hydrogen is pretty good at moderating neutrons down to thermal energies (eV range, ie room temperature) via elastic scattering, but gasses don't really have enough density to do a very good job. If you really want to protect something from neutrons you just coat it with boron. A mm coating of the stuff will keep out pretty much any common source of neutrons.
- kabdib 11y agoOh, they do fail. The last time I worked with some hardware folks speccing a system-on-a-chip, they were modeling device lifetime versus clock speed. "Hey software guys, if we reduce the clock rate by ten percent we get another three years out of the chip." Or somesuch, due to electromigration and other things, largely made worse by heat. Since it was a gaming console, we wound up at some kind of compromise that involved guessing what the Competition would also be doing with their clock rate.
- zokier 11y agoThat is very interesting that they do that for cheap massproduced consumer goods. I mean I can understand doing such tradeoffs in very expensive stuff that is expected to last for decades (industrial machines, space probes etc), but that the manufacturer cares enough about the lifetime (beyond the minimum warranty period) of their goods is somewhat surprising in this day and age.
- luchadorvader 11y agoI wouldn't think it was that surprising. Businesses, especially consoles, want to be seen as reliable and good quality in order to develop a strong relationship with their customers. If it commonly fails right after the minimum warranty, then why would someone want to buy something again that is unreliable? They wouldn't and would go right to the competition.
- mhb 11y agoSomeone more cynical might conclude that they are interested in the MTBF because failures on the early end of the tail that ARE within the warranty period will cost them money.
- kabdib 11y agoWhy be cynical? That's engineering. Real engineering is about materials, cost and time (if you leave one of those out, you're either doing research or just aimlessly puttering around ... maybe both :-) ). There are other dimensions, too. In the consumer market, engineering is also about the reliability that you want your customers to experience. Companies that make console hardware want you to be happy; they're not going to ship you a hunk of hardware that generates a Warranty Expired Interrupt at one-year-and-one-day because they want you to keep buying games and services. Having to buy a new console is a hassle. Maybe you'll buy the competitor's console instead, who knows? On the other hand, console margins (and yes, generally they have margins these days and are not sold at a loss) are razor thin. There are knife-fights in meetings over three cent changes to components because at production scale those pennies rapidly turn into millions of dollars. You don't make a console with a reliability of 20 years because it would cost way too much and be obsolete long before the failure curve started to inflect. So you optimize the product lifetime for user expectation of value, how long you think the technology will remain relevant, what the market will bear, and a bunch of other things (chip sourcing, cost of manufacture, architectural headroom and so on). This involves hard-won experience, spreadsheets, testing, figuring out how vendors are lying to you, plane trips to godforsaken industrial parks, and fist-fights in hallways. It's awesome :-) Are there companies selling stuff that will break the moment the warranty is over, or earlier? Sure; they are betting that actually getting warranty service is so inconvenient that you won't bother. On the other hand, having watched (from the outside!) the execution of a product-recall-scale warranty, I was impressed with the company's professionalism, and it didn't strike me as a company that didn't care about repeat customers. It sucks on both sides for something like this to happen, but there's very little cyncism involved.
- RogerL 11y agoOthers have answered why, here is the 'what would happen'. Heat your CPU up by pointing a hair dryer at it (you may want to treat this as a thought experiment as you could destroy your computer). At some point it begins to fail because transistors are pushed past theiroperating conditions. Another way to push it to failure is to overclock. The results are ... variable. Sometimes you won't notice the problems, computations will just come out wrong. Sometimes the computer will blue screen or spontaneously reboot. And so on. Just depends where the failure occurs, and if the currently running software depends on that part of the chip. If a transistor responsible for instruction dispatch fails it's probably instant death. If a transistor responsible for helping in computing the least significant bit of a sin() computation, well, you may never notice it.
- fr0styMatt2 11y agoI remember playing around with overclocking my old Pentium 4 and how it would boot fine into Windows, but then you'd run Prime95 on it and the benchmark would start failing because the FPU was returning incorrect results.
- nickpsecurity 11y agoThey're extremely simple, have no moving parts, and the materials/processes of semiconductor fabs optimize to ensure they get done right. The whole chip will often fail if transistors are fabbed incorrectly and rest end up in errata sheets where you work around them. Environmental effects are reduced with Silicon-on-Insulator (SOI), rad-hard methods, immunity-aware programming, and so on. Architectures such as Tandem's NonStop assumed there'd be plenty of failures and just ran things in lockstep with redundant components. So, simplicity and hard work by fab designers is 90+% of it. There's whole fields and processes dedicated to the rest.
- nhaehnle 11y agoErrata are usually caused by bugs in the logical design of the chip, not in the manufacturing or physical behavior of transistors. If you have a source for an errata that was issued due to systematically buggy transistors, I'd be curious to hear that story!
- nickpsecurity 11y agoCant the erratic behavior of the transistors (eg flawed ones overheating) cause their logical function to fail? Past that question, Ive still plenty to learn on hardware and will take your word for it about errata sheets. Sounds right given the things described in them.
- rdc12 11y ago"Cant the erratic behavior of the transistors (eg flawed ones overheating)" Yes but that is either a manafacturing defect (if persistant) or a transient error, or running out of speced tolerances. Or simply details of reality. Whereas a logical design flaw, is more the actual design/implemenation, is fundementally wrong more akin to a bug.
- nickpsecurity 11y agoI gotcha. Appreciate the tip.
- zokier 11y agoSlightly related thing is RAM random bit errors. There was an interesting article published few years ago where some guy registered domains that differed by one bit from some popular domains and recorded the traffic that hit them. Kinda scary to think what else is wrong in your RAM then... Too bad that ECC is still restricted to servers and serious workstations. http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html
- buren 11y agoCool! The author of the post you linked, also linked to this paper https://www.cs.princeton.edu/~appel/papers/memerr.pdf https://www.cs.princeton.edu/~appel/papers/memerr.pdf. From the paper: "..experimental study showing that soft memory errors can lead to serious security vulnerabilities in Java and .NET virtual machines"
- rgovind 11y agoThe first author in the princeton paper is my brother!!
- spiritplumber 11y agoThis is why we usually slightly underclock stuff that has to live on boats.
- cskau 11y agoBoats specifically because there's a higher turnaround time to replace failing hardware?
- spiritplumber 11y agoYeah, also corrosion and voltage spikes due to old/untuned gensets or even taking a lightning strike on the hull.
- gibrown 11y agoAs a former hardware engineer who worked on automated test equipment that tested ASICs (and did ASIC dev), there are a lot of different methods used to avoid this. As others mentioned, most of these problems are caught when testing the chips. Most of the transistors on a chip are actually used for caching or RAM, and in those cases the chips have built in methods for disabling the portions of memory that are non-functional. I don't recall any instances of CPUs/firmware doing this dynamically, but I wouldn't be surprised if there are. A lot of chips have some self diagnostics. Most ASICs also have extra transistors sprinkled around so they can bypass and fix errors in the manufacturing process. Making chips is like printing money where some percentage of your money is defective. It pays to try and fix them after printing. Also, as someone who has ordered lots of parts there are many cases where you put a part into production and then find an abnormally high failure rate. I once did a few months of high temperature and vibration testing on our boards to try and discover these sorts of issues, and then you spend a bunch of time convincing the manufacturer that their parts are not meeting spec. Fun times... thanks for the trip down memory lane.
- mcshicks 11y agoIn my previous job I did failure analysis for returned ASICs for a high volume commercial application. As the parent says in fact because a modern ASIC has structural coverage with built in self test for the overwhelming majority of digital circuits, you don't ship a lot of parts with bad transistors. The vast majority of ASICs that I saw that were returned had failed due to electrical over stress, which is polite way of saying the customer blew up the part. But even more to the point, in modern electronics the ratio of problems caused by either software or assembly level issues is very high compared to ASIC level hw problems. High volume ASICs are designed with relatively large margins to ensure good product yield. Further the cost of investigating and root causing an issue to the transistor level is very high, that unless you have measurable trend defect rate, any actual random failure not related to design would probably be attributed to something else first. That is not to say that there are no HW design level issues with ASICs, but generally when they are discovered you would try and change the low level software to either make the problem not happen ever, or make it very very infrequent. You might also simply screen (test) the parts so you don't ship any that exhibit the undesired behavior. So it's not that such an event that the OP mentions can never happen, it's just that it's so infrequent compared to all the other types of problems that can happen in modern electronics that unless you have a very specific reason to believe it's being caused by hardware you would never be able to differentiate it from a random software bug.
- Gibbon1 11y agoTransistors don't fail for the same reason the 70 year old wires in my house don't fail. The electrons flowing through the transistors doesn't disturb the molecular structure of the doped silicon.
- jacquesm 11y agoSorry, but that's just plain wrong: https://en.wikipedia.org/wiki/Electromigration https://en.wikipedia.org/wiki/Electromigration At the scale of your house wiring the effect is not so noticeable but for integrated circuits it is definitely a factor. As for your house wiring, if it is really 70 years old you might want to worry about the insulation, not the copper.
- Gibbon1 11y agoWhat's wrong about it? Transistors don't work by electromigration.
- panax 11y agoElectromigration will move atoms on the interconnects between transistors and eventually cause an open.
- Gibbon1 11y agoHe said transistors, not IC's. And electromigration is a problem with metal interconnects at high current densities. So you talking solemnly about a failure type that can occur with on some integrated circuits under some conditions, or when the designers screwed up. But which doesn't actually happen much in practice. The persons actually question is roughly why do transistors last so long compared to other types of mechanisms. No one in the comments made an attempt to answer that, at all.
- tzs 11y agoSpeaking of the effects of component failure on chips, a couple years ago researchers demonstrated self-healing chips [1]. Large parts of the chips could be destroyed and the remaining components would reconfigure themselves to find an alternative way to accomplish their task. [1] http://www.caltech.edu/news/creating-indestructible-self-healing-circuits-38815 http://www.caltech.edu/news/creating-indestructible-self-hea...
- panax 11y agoThe 10 series Altera FPGAs are going to be a little like this, although it is more for SEUs and also IP segmentation/security. You will have lots of little islands, all which can detect configuration errors and report them, and even recover using partial reconfiguration. Maybe someday we can make it smart enough to replace and reroute around physically bad LEs automatically.
- intrasight 11y agoWhen I was studying EE, a professor said on this subject that about 20% of the transistors in a chip are used for self-diagnostics. Manufacturing failures are a given. The diagnostics tell the company what has failed, and they segment the chips into different product/price classes based upon what works and what doesn't. After being deployed into a product, I assume that chips would follow a standard Bathtub Curve: https://en.wikipedia.org/wiki/Bathtub_curve https://en.wikipedia.org/wiki/Bathtub_curve As geometries fall, the effects of "wear" at the atomic level will go up.
- pjc50 11y agoI think the proportion is nowhere near that high for most ASICs - JTAG boundary scan adds a few percent to give you this testability, maybe 1-5%.
- Taniwha 11y agoFull chip scan provide logic that links EVERY flipflop in a design into one or more scan chain (not just boundary scan which checks pin bonding) - mostly it's a mux on each input - and is more like 5%ish You test by scanning in a bit pattern, issuing a single clock andscanning out the result. Smart software generates the minimal set of test vectors that tests every wire and gate between the flops. Chip testers are expensive (millionish) so minimising tester time minimises chip cost - we make special testign logic for things like srams
- panax 11y agoYour dual core CPU is often the same quad core die with two of the cores disabled because one of the cores has a defect and is sold at a discount to increase yield.
- greenNote 11y agoAs stated, two big variables are clock rate and feature size, which both effect mean time between failures (MTBF). Being more conservative increases this metric. I know from working in a fab that there are many electrical inspection steps along the process, so failures are caught during the manufacturing process (reducing the chance that you see them in the final product). Once the chip is packaged, and assuming that it is operated in a nominal environment, then failures are not that common.
- Nomentatus 11y agoNearly all chips experienced transistor failures, rendering them useless, back in the day. Intel is the monster it is because they were the guys who first found out how to sorta "temper" chips to vastly reduce that failure rate (most failures were gross enough to be instant, back then, and Intel started with memory chips.) Because their heat treatment left no visible mark, Intel didn't patent it, but kept it as a trade secret giving them an incredible economic advantage, for many years. They all but swept the field. I've no doubt misremembered some details.
- Gravityloss 11y agoThey do fail. Linus Torvalds talked about this in 2007 http://yarchive.net/comp/linux/cpu_reliability.html http://yarchive.net/comp/linux/cpu_reliability.html
- wsxcde 11y agoOthers have already mentioned one failure mechanism that causes transistor degradation over time: electromigration. Other important aging mechanisms are negative-bias temperature instability (NBTI) and hot carrier injection (HCI). I've seem papers claim the dual of NBTI - PBTI - is now an issue in the newest process nodes. This seems to be a nice overview of aging effects: http://spectrum.ieee.org/semiconductors/processors/transistor-aging http://spectrum.ieee.org/semiconductors/processors/transisto....
- MichaelCrawford 11y agoThey do. That's why our boxen have power-on self tests.
- msandford 11y ago> Considering that a Quad-core + GPU Core i7 Haswell has 1.4e9 transistors inside, even given a really small probability of one of them failing, wouldn't this be catastrophic? Yes, generally speaking it would be. Depending on where it is inside the chip. > Wouldn't a single transistor failing mean the whole chip stops working? Or are there protections built-in so only performance is lost over time? Not necessarily. It might be somewhere that never or rarely gets used, in which case the failure won't make the chip stop working. It might mean that you start seeing wrong values on a particular cache line, or that your branch prediction gets worse (if it's in the branch predictor) or that your floating point math doesn't work quite right anymore. But most of the failures are either manufacturing errors meaning that the chip NEVER works right, or they're "infant mortality" meaning that the chip dies very soon after it's packaged up and tested. So if you test long enough, you can prevent this kind of problem from making it to customers. Once the chip is verified to work at all, and it makes it through the infant mortality period, the lifetime is actually quite good. There are a few reasons: 1. there are no moving parts so traditional fatigue doesn't play a role 2. all "parts" (transisotrs) are encased in multiple layers of silicon dioxide so that you can lay the metal layers down 3. the whole silicon die is encased yet again in another package which protects the die from the atmosphere 4. even if it was exposed to the atmosphere, and the raw silicon oxidized, it would make silicon dioxide, which is a protective insulator 5. there is a degradation curve for the transistors, but the manufacturers generally don't push up against the limits too hard because it's fairly easy and cheap to underclock and the customer doesn't really know what they're missing 6. since most people don't stress their computers too egregiously this merely slows down the slide down the degradation curve as it's largely governed by temperature, and temperature is generated by a) higher voltage required for higher clock speed and b) more utilization of the CPU Once you add all these up you're left with a system that's very, very robust. The failure rates are serious but only measured over decades. If you tried to keep a thousand modern CPUs running very hot for decades you'd be sorely disappointed in the failure rate. But for the few years that people use a computer and the relative low load that they place on them (as personal computers) they never have a big enough sample space to see failures. Hard drives and RAM fail far sooner, at least until SSDs start to mature.
- deleted 11y ago[deleted]
- jsudhams 11y agoSo would that mean we need to ensure the systems in critical area (not nuclear or some but banks and transaction critical) be tech refereshed mandatory at 4/5 years? Especially when 7nm production starts.
- 2bluesc 11y agoIn 2011, Intel released the 6 series chipset with an incorrectly sized transistor that would ultimately fail if used extensively. A massive recall followed. http://www.anandtech.com/show/4142/intel-discovers-bug-in-6series-chipset-begins-recall http://www.anandtech.com/show/4142/intel-discovers-bug-in-6s...