7 ms·

I don't work in environment where I get to deal with hardware failures, so pardon my ignorance, but has anyone seen a failed CPU piece which has failed during n

by zxcdw 13y ago

I don't work in environment where I get to deal with hardware failures, so pardon my ignorance, but has anyone seen a failed CPU piece which has failed during normal operation? I am under an impression that it is very rare for a CPU itself to fail so that it would need to be replaced. The only times I've even heard about failing CPUs has been if they've been overclocked or insufficiently cooled(add in overvolting, and you get both :)) or physical damage during mounting/unmounting or otherwise handling hardware. And even then the failure has usually been elsewhere than the CPU itself. Of course I am not saying it'd be unheard of, but for me frankly, right now it is.

zhemao 13y ago
I think the MTBF is generally longer than people would normally go without replacing their CPU. Also, CPUs are generally designed to degrade more gracefully. For instance, they may have circuity that scales the frequency down as delays get longer. Also, in multicore CPUs, there are generally some spare cores that will get swapped in if a previously in-use core breaks.
- anon_cownerd 13y ago
  > Also, in multicore CPUs, there are generally some spare cores that will get swapped in if a previously in-use core breaks. That sounds like a huge cost to bear. Looking at e.g. a Haswell die photo [1], there are just four physical cores present for a four-core part. With that die area per core, you would take a ~15-20% area hit (that translates to 15-20% cost) just to have a spare core in case one failed some years later. I have heard of manufacturers selling otherwise "defective" parts where a core or cache slice has a defect by relabeling as a part with fewer cores. But that's a manufacture-time decision, not a dynamic reconfiguration in the field. [1] http://cdn2.wccftech.com/wp-content/uploads/2013/05/Intel-Haswell-Core.jpg http://cdn2.wccftech.com/wp-content/uploads/2013/05/Intel-Ha...
  - ssafejava 13y ago
    I think the GP was on the right track, but somewhat confused. They obviously don't do this on all models, but some dual-core models are disabled quad-cores. Remember the Athlon X3? That was a binned chip that usually was created from X4s with a broken core. Most buyers didn't mind, and some of them got lucky and were able to re-enable the disabled core. It seems like the GP might be suggesting that a quad-core CPU will swap in another core when one dies. That doesn't happen. But the binning process allows them to still sell slightly defective silicon with disabled parts (cores, cache), which saves money. On a related note, a lot of GPUs actually do have a few dozen execution units that are disabled by default and can be swapped in after stress testing at the factory. I believe some can even do that in the wild, but I could be wrong.
    - zhemao 13y ago
      Oh whoops, I guess my VLSI professor lied to me. Maybe it only happens with certain multiprocessors. But yeah, there are definitely disabled cores in a lot of processors for the reasons you mentioned.
      - cantfindmypass 13y ago
        ISTR the PS3's Cell processor had eight physical cores to increase yeild.
        zhemao 13y ago
        You're mostly right https://en.wikipedia.org/wiki/PlayStation_3#Technical_specifications https://en.wikipedia.org/wiki/PlayStation_3#Technical_specif... The cell processor had a main PowerPC core and eight floating-point SIMD co-processors called synergistic processing elements (SPE). For the PS3, one of the eight SPEs was disabled and another was reserved for the operating system, leaving the other six for developers.
naner 13y ago
Yes, CPUs can fail just like any other hardware component. On desktop systems the most common case is you'll try and boot the system and just be presented with blank video or a beep code. On server systems with multiple CPUs there usually will be an error reported via blinking light or the little info LCD on the front of the system. In some cases the damage is actually visible on the CPU (e.g. discoloration of some of the gold contact points on the bottom). CPUs under normal operation fail less frequently than most other components in my experience.
grumps 13y ago
I use to work in situations where we had to account for failures. We had lab equipment that would just run all that time, and with terrible wiring and we were horrible to it too. We left covers off, piled stuff on top of it and just Frankenstein the hell out of all of it. I even managed to flash a new OS/App at the same time of a power failure but it still lived.... Failures were more prominent in memory...but they did happen. We also sent equipment through environmental testing that would force failures. I don't recall of hearing of any CPU failures. Although most of our equipment was DSP & FPGA based but there were some tiny 'lil CPU's there.
tonyarkles 13y ago
I had a Celeron that had its cache go bad once. It would work "fine" in Windows, but Linux would report that the CPU was throwing some kind of exception. If you went into the BIOS and disabled the cache it would be stable, but with the cache on it would crash after a day or two. Swapped out the CPU and the machine lived a great life for a long time.
- zymhan 13y ago
  > but Linux would report that the CPU was throwing some kind of exception Does this mean it didn't boot into Linux?
  - tonyarkles 13y ago
    It booted but dmesg was very noisy! It would actually run for a day or two, serving up a modest PHP load, before crapping out.
gnoway 13y ago
I have seen it happen one time that I can remember, where I was sure it was the CPU. We had 8 dual socket 5400-era Xeon servers in a VMware cluster. Whenever 64bit Windows 2008+ virtual machines were started on or were vmotioned to one of the hosts, they would bluescreen. We did not experience this behavior at all, then one day we did. I replaced both CPUs and the problem disappeared. I have to assume it was one of the CPUs. It's entirely possible that they overheated, but if they did it was due to poor cooling; we did not overvolt or overclock these machines.
aimhb 13y ago
It was posted here on HN a few days ago, but you might find the DEFCON talk on single bit domain errors relevant: https://www.youtube.com/watch?v=ZPbyDSvGasw https://www.youtube.com/watch?v=ZPbyDSvGasw In summary, some data centers are run hotter than recommended, which leads to a lot of mostly ignored domain resolution errors, which leads to a security risk.
dntrkv 13y ago
What a crazy coincidence. I was actually just helping a friend troubleshoot his dead PC and I told him to test his ram, video card, motherboard, powersupply, in that order, and not to bother with the CPU since I have never had one fail on me. It ended up being the CPU that went out. I go on Hacker News a couple hours later and see this post. heh.
- ac29 13y ago
  https://en.wikipedia.org/wiki/Baader-Meinhof_phenomenon https://en.wikipedia.org/wiki/Baader-Meinhof_phenomenon
  - josephagoss 13y ago
    Is this the same effect that when told to count yellow cars whilst driving, you end up noticing far more yellow cars than you thought were on road. Or how you notice how much more frequent your model of car is on the roads?
    - olalonde 13y ago
      Also when you learn a new word in a different language, it seems like suddenly everyone is using that word all the time.
  - option_greek 13y ago
    Are there any explanations for this phenomenon. I keep experiencing it all the time :)
    - ehsanu1 13y ago
      Me too, but I had always figured it as survivorship bias. That is, there are many such things that I newly hear of, but most of them don't get repeated mentions right way. The few of them that do catch my attention, and it seems like it happens a lot. But it actually happens only for a few of the many, many new things I hear about all the time. Similar to my friend who thought that every time he looked at a dead street lamp, it would turn on. :) Another possible factor is that I find I pay more attention to things I've recently learned about more closely when I come across them, but ignore them when I either know them very well or don't know them at all. I don't know if this is true for anyone else.
ChuckMcM 13y ago
" has anyone seen a failed CPU piece which has failed during normal operation?" Several. But I'm lucky in that I worked for NetApp for 5 years which have several million NetApp filers in the field that were all calling home when they had issues, and Google which has a very large number of CPUs all around the planet doing their bidding. With visibility into a population like that you see faults that are once in a billion happen about once a month :-). Two general kinds of failures though, the more common one is a system machine check (the internal logic detected the fault condition and put the CPU into the machine check state) which happens when 3 or more bits go sideways in the various RAM inside the mesh of execution units. Nominally ECC protected it can detect but not correct multi-bit errors. Power it off, power it on, and restart from a known condition and it's good as new. The more rare occurrence is that something in the CPU fails which results in the CPU not coming out of RESET even, or immediately going into a machine check state. When you find those Intel often wants you to send it back to them so they can do failure analysis on it. The most common root cause analysis for those is some moderate electrostatic damage which took a while to finally finish the process of failing. Some of the more interesting papers at the ISSCC are sometimes on lifetime expectancy of small geometry transistors. They are a lot more susceptible to damage and disruption due to cosmic rays and other environmental agents.
- dale-cooper 13y ago
  Interesting. How would these failures manifest themselves for a user?
  - ithkuil 13y ago
    In the best case your app crashes or misbehaves randomly not unlike bad RAM (yes it can happen even with ECC). In the worst case there is a subtle numeric error that could potentially percolate to users. Usually it's about floating point issues, since usually you don't use floats to access arrays or do pointer arithm, issues in that unit may not cause an early failure and hence go undetected.
  - ChuckMcM 13y ago
    Simple case is that your machine just halts. If you have a way of looking at them, it might puts a bios code on the LEDs. On servers the baseboard management chip (BMC) will usually record a machine check event as well.
zurn 13y ago
It's not easy to get a CPU chip failure reliably diagnosed in the field. Even if you manage to do the trial and error component swapping dance pointing to the CPU, you don't get very good confidence. Might be that new CPU taxes the power feed less or there was misapplied cooling paste or a bad contact in the pins etc.
forrestthewoods 13y ago
I think you're only considering catastrophic unable to boot failure. A CPU could be going bad with the effects less obvious. Microsoft released research that shows enthusiast overclocking has a clear increase in hardware faults. [1] http://www.extremetech.com/gaming/131739-microsoft-analyzes-over-a-million-pc-failures-results-shatter-enthusiast-myths http://www.extremetech.com/gaming/131739-microsoft-analyzes-... [2] http://research.microsoft.com/pubs/144888/eurosys84-nightingale.pdf http://research.microsoft.com/pubs/144888/eurosys84-nighting...
- alelefant 13y ago
  While that research is certainly interesting, the poster above you acknowledged that overclocking could result in failures.
rodgerd 13y ago
From one of my personal machines: [14865975.000023] Machine check events logged MCE 0 CPU 0 BANK 2 ADDR 1438280 TIME 1384859595 Wed Nov 20 00:13:15 2013 STATUS d40040000000011a MCGSTATUS 0 MCGCAP 104 APICID 0 SOCKETID 0 CPUID Vendor AMD Family 6 Model 8 This will, of course, be getting replaced shortly, preferably before it does any real damage. Given it's a 2002-era chip, 11 years of service isn't exactly terrible. I've seen quite a few UltraSPARC chips (especially IIIs) go over the years at work, and often had a shit of a time trying to get Sun to accept them as faulty and replace them.
cnvogel 13y ago
A machine here at work logged cache related machine check excel exceptions, at a rate of roughly 1/day but not regularly or deterministic. Not related to load or temperature and even after clocking lower than spec'ed. Changing the CPU fixed it. Those were correctable errors, prime95 or memtest did not detect anything.
vanderZwan 13y ago
My dad had a laptop which would not boot unless he put it in the fridge first for half an hour. As long as he didn't reboot everything then worked "fine". Does that count?
- raverbashing 13y ago
  That's probably a bad contact or a tiny fissure that makes contact again when cooled And the failed part is either needed only on bootup or when the current gets going it doesn't stop until it's powered off again
  - vanderZwan 13y ago
    Thanks for clearing that up - it's always been bothering what it could possibly be (that and the "how on earth did he figure that out in the first place?")
    - danudey 13y ago
      The cold would shrink the motherboard to re-connect the contacts; it might have been fixed by doing the old Xbox/nvidia trick of putting it in the oven (which would soften the solder, causing it to shift and re-connect). With the Xbox, you could apparently even just wrap it in a towel and let its own heat do the trick.
- serans 13y ago
  how on earth did he come up with putting the laptop in the fridge?
Shorel 13y ago
I saw a 386 let the magic blue smoke out.
cafard 13y ago
A couple of times: Sparc chips that did some notable damage on their way out. One was pretty old, one not so much.
valarauca1 13y ago
I had this problem once when overclocking an AMD phenon. The short story is (I don't know the whole cause) the on-board crypto units stopped being random. Which wasn't a real problem for 'some' day-to-day use. This was in the mid to later 00's, so https wasn't quiet everywhere yet. The problem manifested slowly. When ever I'd connect to HTTPS, my browser would crash. One sound card would phone home for an update, and my computer would crash. Randomly certain games would crash when ever anti-cheat software attempted to run. It was just odd, and took a few days of hunting to find out what was actually going wrong.

by zxcdw 13y ago

zhemao 13y ago

I think the MTBF is generally longer than people would normally go without replacing their CPU. Also, CPUs are generally designed to degrade more gracefully. For instance, they may have circuity that scales the frequency down as delays get longer. Also, in multicore CPUs, there are generally some spare cores that will get swapped in if a previously in-use core breaks.

anon_cownerd 13y ago
> Also, in multicore CPUs, there are generally some spare cores that will get swapped in if a previously in-use core breaks. That sounds like a huge cost to bear. Looking at e.g. a Haswell die photo [1], there are just four physical cores present for a four-core part. With that die area per core, you would take a ~15-20% area hit (that translates to 15-20% cost) just to have a spare core in case one failed some years later. I have heard of manufacturers selling otherwise "defective" parts where a core or cache slice has a defect by relabeling as a part with fewer cores. But that's a manufacture-time decision, not a dynamic reconfiguration in the field. [1] http://cdn2.wccftech.com/wp-content/uploads/2013/05/Intel-Haswell-Core.jpg http://cdn2.wccftech.com/wp-content/uploads/2013/05/Intel-Ha...
- ssafejava 13y ago
  I think the GP was on the right track, but somewhat confused. They obviously don't do this on all models, but some dual-core models are disabled quad-cores. Remember the Athlon X3? That was a binned chip that usually was created from X4s with a broken core. Most buyers didn't mind, and some of them got lucky and were able to re-enable the disabled core. It seems like the GP might be suggesting that a quad-core CPU will swap in another core when one dies. That doesn't happen. But the binning process allows them to still sell slightly defective silicon with disabled parts (cores, cache), which saves money. On a related note, a lot of GPUs actually do have a few dozen execution units that are disabled by default and can be swapped in after stress testing at the factory. I believe some can even do that in the wild, but I could be wrong.
  - zhemao 13y ago
    Oh whoops, I guess my VLSI professor lied to me. Maybe it only happens with certain multiprocessors. But yeah, there are definitely disabled cores in a lot of processors for the reasons you mentioned.
    - cantfindmypass 13y ago
      ISTR the PS3's Cell processor had eight physical cores to increase yeild.
      - zhemao 13y ago
        You're mostly right https://en.wikipedia.org/wiki/PlayStation_3#Technical_specifications https://en.wikipedia.org/wiki/PlayStation_3#Technical_specif... The cell processor had a main PowerPC core and eight floating-point SIMD co-processors called synergistic processing elements (SPE). For the PS3, one of the eight SPEs was disabled and another was reserved for the operating system, leaving the other six for developers.

naner 13y ago

Yes, CPUs can fail just like any other hardware component. On desktop systems the most common case is you'll try and boot the system and just be presented with blank video or a beep code. On server systems with multiple CPUs there usually will be an error reported via blinking light or the little info LCD on the front of the system. In some cases the damage is actually visible on the CPU (e.g. discoloration of some of the gold contact points on the bottom). CPUs under normal operation fail less frequently than most other components in my experience.

grumps 13y ago

I use to work in situations where we had to account for failures. We had lab equipment that would just run all that time, and with terrible wiring and we were horrible to it too. We left covers off, piled stuff on top of it and just Frankenstein the hell out of all of it. I even managed to flash a new OS/App at the same time of a power failure but it still lived.... Failures were more prominent in memory...but they did happen. We also sent equipment through environmental testing that would force failures. I don't recall of hearing of any CPU failures. Although most of our equipment was DSP & FPGA based but there were some tiny 'lil CPU's there.

tonyarkles 13y ago

I had a Celeron that had its cache go bad once. It would work "fine" in Windows, but Linux would report that the CPU was throwing some kind of exception. If you went into the BIOS and disabled the cache it would be stable, but with the cache on it would crash after a day or two. Swapped out the CPU and the machine lived a great life for a long time.

zymhan 13y ago
> but Linux would report that the CPU was throwing some kind of exception Does this mean it didn't boot into Linux?
- tonyarkles 13y ago
  It booted but dmesg was very noisy! It would actually run for a day or two, serving up a modest PHP load, before crapping out.

gnoway 13y ago

I have seen it happen one time that I can remember, where I was sure it was the CPU. We had 8 dual socket 5400-era Xeon servers in a VMware cluster. Whenever 64bit Windows 2008+ virtual machines were started on or were vmotioned to one of the hosts, they would bluescreen. We did not experience this behavior at all, then one day we did. I replaced both CPUs and the problem disappeared. I have to assume it was one of the CPUs. It's entirely possible that they overheated, but if they did it was due to poor cooling; we did not overvolt or overclock these machines.

aimhb 13y ago

It was posted here on HN a few days ago, but you might find the DEFCON talk on single bit domain errors relevant: https://www.youtube.com/watch?v=ZPbyDSvGasw https://www.youtube.com/watch?v=ZPbyDSvGasw In summary, some data centers are run hotter than recommended, which leads to a lot of mostly ignored domain resolution errors, which leads to a security risk.

dntrkv 13y ago

What a crazy coincidence. I was actually just helping a friend troubleshoot his dead PC and I told him to test his ram, video card, motherboard, powersupply, in that order, and not to bother with the CPU since I have never had one fail on me. It ended up being the CPU that went out. I go on Hacker News a couple hours later and see this post. heh.

ac29 13y ago
https://en.wikipedia.org/wiki/Baader-Meinhof_phenomenon https://en.wikipedia.org/wiki/Baader-Meinhof_phenomenon
- josephagoss 13y ago
  Is this the same effect that when told to count yellow cars whilst driving, you end up noticing far more yellow cars than you thought were on road. Or how you notice how much more frequent your model of car is on the roads?
  - olalonde 13y ago
    Also when you learn a new word in a different language, it seems like suddenly everyone is using that word all the time.
- option_greek 13y ago
  Are there any explanations for this phenomenon. I keep experiencing it all the time :)
  - ehsanu1 13y ago
    Me too, but I had always figured it as survivorship bias. That is, there are many such things that I newly hear of, but most of them don't get repeated mentions right way. The few of them that do catch my attention, and it seems like it happens a lot. But it actually happens only for a few of the many, many new things I hear about all the time. Similar to my friend who thought that every time he looked at a dead street lamp, it would turn on. :) Another possible factor is that I find I pay more attention to things I've recently learned about more closely when I come across them, but ignore them when I either know them very well or don't know them at all. I don't know if this is true for anyone else.

ChuckMcM 13y ago

" has anyone seen a failed CPU piece which has failed during normal operation?" Several. But I'm lucky in that I worked for NetApp for 5 years which have several million NetApp filers in the field that were all calling home when they had issues, and Google which has a very large number of CPUs all around the planet doing their bidding. With visibility into a population like that you see faults that are once in a billion happen about once a month :-). Two general kinds of failures though, the more common one is a system machine check (the internal logic detected the fault condition and put the CPU into the machine check state) which happens when 3 or more bits go sideways in the various RAM inside the mesh of execution units. Nominally ECC protected it can detect but not correct multi-bit errors. Power it off, power it on, and restart from a known condition and it's good as new. The more rare occurrence is that something in the CPU fails which results in the CPU not coming out of RESET even, or immediately going into a machine check state. When you find those Intel often wants you to send it back to them so they can do failure analysis on it. The most common root cause analysis for those is some moderate electrostatic damage which took a while to finally finish the process of failing. Some of the more interesting papers at the ISSCC are sometimes on lifetime expectancy of small geometry transistors. They are a lot more susceptible to damage and disruption due to cosmic rays and other environmental agents.

dale-cooper 13y ago
Interesting. How would these failures manifest themselves for a user?
- ithkuil 13y ago
  In the best case your app crashes or misbehaves randomly not unlike bad RAM (yes it can happen even with ECC). In the worst case there is a subtle numeric error that could potentially percolate to users. Usually it's about floating point issues, since usually you don't use floats to access arrays or do pointer arithm, issues in that unit may not cause an early failure and hence go undetected.
- ChuckMcM 13y ago
  Simple case is that your machine just halts. If you have a way of looking at them, it might puts a bios code on the LEDs. On servers the baseboard management chip (BMC) will usually record a machine check event as well.

zurn 13y ago

It's not easy to get a CPU chip failure reliably diagnosed in the field. Even if you manage to do the trial and error component swapping dance pointing to the CPU, you don't get very good confidence. Might be that new CPU taxes the power feed less or there was misapplied cooling paste or a bad contact in the pins etc.

forrestthewoods 13y ago

I think you're only considering catastrophic unable to boot failure. A CPU could be going bad with the effects less obvious. Microsoft released research that shows enthusiast overclocking has a clear increase in hardware faults. [1] http://www.extremetech.com/gaming/131739-microsoft-analyzes-over-a-million-pc-failures-results-shatter-enthusiast-myths http://www.extremetech.com/gaming/131739-microsoft-analyzes-... [2] http://research.microsoft.com/pubs/144888/eurosys84-nightingale.pdf http://research.microsoft.com/pubs/144888/eurosys84-nighting...

alelefant 13y ago
While that research is certainly interesting, the poster above you acknowledged that overclocking could result in failures.

rodgerd 13y ago

From one of my personal machines: [14865975.000023] Machine check events logged MCE 0 CPU 0 BANK 2 ADDR 1438280 TIME 1384859595 Wed Nov 20 00:13:15 2013 STATUS d40040000000011a MCGSTATUS 0 MCGCAP 104 APICID 0 SOCKETID 0 CPUID Vendor AMD Family 6 Model 8 This will, of course, be getting replaced shortly, preferably before it does any real damage. Given it's a 2002-era chip, 11 years of service isn't exactly terrible. I've seen quite a few UltraSPARC chips (especially IIIs) go over the years at work, and often had a shit of a time trying to get Sun to accept them as faulty and replace them.

cnvogel 13y ago

A machine here at work logged cache related machine check excel exceptions, at a rate of roughly 1/day but not regularly or deterministic. Not related to load or temperature and even after clocking lower than spec'ed. Changing the CPU fixed it. Those were correctable errors, prime95 or memtest did not detect anything.

vanderZwan 13y ago

My dad had a laptop which would not boot unless he put it in the fridge first for half an hour. As long as he didn't reboot everything then worked "fine". Does that count?

raverbashing 13y ago
That's probably a bad contact or a tiny fissure that makes contact again when cooled And the failed part is either needed only on bootup or when the current gets going it doesn't stop until it's powered off again
- vanderZwan 13y ago
  Thanks for clearing that up - it's always been bothering what it could possibly be (that and the "how on earth did he figure that out in the first place?")
  - danudey 13y ago
    The cold would shrink the motherboard to re-connect the contacts; it might have been fixed by doing the old Xbox/nvidia trick of putting it in the oven (which would soften the solder, causing it to shift and re-connect). With the Xbox, you could apparently even just wrap it in a towel and let its own heat do the trick.
serans 13y ago
how on earth did he come up with putting the laptop in the fridge?

Shorel 13y ago

I saw a 386 let the magic blue smoke out.

cafard 13y ago

A couple of times: Sparc chips that did some notable damage on their way out. One was pretty old, one not so much.

valarauca1 13y ago

I had this problem once when overclocking an AMD phenon. The short story is (I don't know the whole cause) the on-board crypto units stopped being random. Which wasn't a real problem for 'some' day-to-day use. This was in the mid to later 00's, so https wasn't quiet everywhere yet. The problem manifested slowly. When ever I'd connect to HTTPS, my browser would crash. One sound card would phone home for an update, and my computer would crash. Randomly certain games would crash when ever anti-cheat software attempted to run. It was just odd, and took a few days of hunting to find out what was actually going wrong.