7 ms·

Sorry, unable to believe: 2-4% failure rate for CPU's? That's for detected/known failures: what about random, unable-to-reproduce, hardly noticed the data skip

by w10-1 2y ago

Sorry, unable to believe: 2-4% failure rate for CPU's? That's for detected/known failures: what about random, unable-to-reproduce, hardly noticed the data skip failures? Have I been living in a fantasy bubble where CPU's do exactly what you asked of them (and errors come from not holding it right)?

gjsman-1000 2y ago
4% seems very high to me; but CPU errors happen with relative frequency, and design mistakes are common. If you ever run “cat /proc/cpuinfo” on, say, Skylake - Linux will happily tell you it has 5-6 workarounds active for hardware mistakes. CPUs are still pretty darn reliable. Think about how many GHz your CPU runs at, multiplied many instructions per cycle there are, and then calculate the failure rate if there was just 1 mistake per minute. Nothing on earth would compare.
- upon_drumhead 2y ago
  Mechanical hard drives are absolutely on the same, or higher, level of reliability. It's mind boggling what we can achieve when we really focus on quality outcomes.
  - deleted 2y ago
    [deleted]
- veqq 2y ago
  Wow, you weren't joking: > apic_c1e spectre_v1 spectre_v2 spec_store_bypass swapgs taa itlb_multihit srbds mmio_stale_data retbleed eibrs_pbrsb gds bhi
  - asveikau 2y ago
    I mean, a bunch of those are timing issues in speculative execution, you could make an argument that it's working as designed but people didn't anticipate the existence of timing exploits. I'd call that different from computation errors.
    - gjsman-1000 2y ago
      As the original comment suggested, about 5-6 of these are not related to timing exploits (or at least, not the Meltdown/Spectre variants which claim so many patches to their name). Summaries from LKML and Kernel.org: > apic_c1e Both ACPI and MP specifications require that the APIC id in the respective tables must be the same as the APIC id in CPUID. The kernel retrieves the physical package id from the APIC id during the ACPI/MP table scan and builds the physical to logical package map. There exist Virtualbox and Xen implementations which violate the spec. As a result the physical to logical package map, which relies on the ACPI/MP tables does not work on those systems, because the CPUID initialized physical package id does not match the firmware id. This causes system crashes and malfunction due to invalid package mappings. The only way to cure this is to sanitize the physical package id after the CPUID enumeration and yell when the APIC ids are different. If the physical package IDs differ use the package information from the ACPI/MP tables so the existing logical package map just works. > taa TAA is a hardware vulnerability that allows unprivileged speculative access to data which is available in various CPU internal buffers by using asynchronous aborts within an Intel TSX transactional region. > itlb_multihit iTLB multihit is an erratum where some processors may incur a machine check error, possibly resulting in an unrecoverable CPU lockup, when an instruction fetch hits multiple entries in the instruction TLB. This can occur when the page size is changed along with either the physical address or cache type. A malicious guest running on a virtualized system can exploit this erratum to perform a denial of service attack. > srbds SRBDS is a hardware vulnerability that allows MDS techniques to infer values returned from special register accesses. Special register accesses are accesses to off core registers. According to Intel's evaluation, the special register reads that have a security expectation of privacy are RDRAND, RDSEED and SGX EGETKEY. > mmio_stale_data Processor MMIO Stale Data Vulnerabilities are a class of memory-mapped I/O (MMIO) vulnerabilities that can expose data. The sequences of operations for exposing data range from simple to very complex. Because most of the vulnerabilities require the attacker to have access to MMIO, many environments are not affected. System environments using virtualization where MMIO access is provided to untrusted guests may need mitigation. These vulnerabilities are not transient execution attacks. However, these vulnerabilities may propagate stale data into core fill buffers where the data can subsequently be inferred by an unmitigated transient execution attack. Mitigation for these vulnerabilities includes a combination of microcode update and software changes, depending on the platform and usage model. Some of these mitigations are similar to those used to mitigate Microarchitectural Data Sampling (MDS) or those used to mitigate Special Register Buffer Data Sampling (SRBDS).
- cwbriscoe 2y ago
  Here is for my Ryzen 7700X: sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso
  - amluto 2y ago
    FWIW, “sysret_ss_attrs” is a workaround for a design error in AMD’s x86_64 implementation. One might argue that AMD is right because they designed AMD64 in the first place, but IMO this is silly, and AMD’s design is unjustifiable. (I’m the one who characterized this issue on Linux and wrote the test case and the workaround.)
layer8 2y ago
This is consistent with their report for 2019-2021, preceding the issues with the 13th/14th gen: https://www.pugetsystems.com/labs/articles/most-reliable-pc-hardware-of-2021-2279/#Processors_CPUs https://www.pugetsystems.com/labs/articles/most-reliable-pc-... Apparently it was much better in 2018: https://www.pugetsystems.com/labs/articles/most-reliable-pc-hardware-of-2018-1322/#CPU_Processor https://www.pugetsystems.com/labs/articles/most-reliable-pc-... On the other hand, 2011 did show 1.5%: https://www.pugetsystems.com/labs/articles/most-reliable-pc-hardware-of-2014-616/#CPU_Processor https://www.pugetsystems.com/labs/articles/most-reliable-pc-... GPU failure rates also weren't great 10-15 years ago, in particular for AMD: https://www.pugetsystems.com/labs/articles/video-card-failure-rates-by-generation-563/ https://www.pugetsystems.com/labs/articles/video-card-failur...
- moffkalast 2y ago
  Had three ATI Radeons back in the day, and that's only because the first two died under warranty one after the other lmao. it was even worse before AMD bought them.
Panzer04 2y ago
I wonder if this buckets things like motherboard problems under the same causes. Those numbers do seem very high in general. I guess it would be a pretty useless comparison if they weren't carefully filtering for CPU-only failure though..
Aurornis 2y ago
> Have I been living in a fantasy bubble where CPU's do exactly what you asked of them (and errors come from not holding it right)? These are gaming CPUs clocked right up to the threshold of stability (and in some cases, past it) Server CPUs with ECC RAM are significantly better. However, if you haven’t experienced much CPU instability, you may not have operated at scale where it appears. Get into the scale where operations occur across 100,000s of CPUs and weird things happen constantly.
- userbinator 2y ago
  One of the previous discussions of this instability noted that it was happening to the server equivalents (which are after all the same die) at stock settings as well.
  - Dalewyn 2y ago
    I remember hearing that server motherboards also played a role in overclocking out of the box, which is frankly fucking stupid. I don't recall anything about Raptor Lake-based Xeons suffering from degradation.
    - userbinator 2y ago
      Unless they disable Turbo Boost (which is horrible for performance, but great if you want benchmark consistency), the CPU will automatically overclock until it reaches the limits, adjusting both voltage and frequency. All the evidence I've seen points to electromigration as a cause of this degradation, and IMHO excessively aggressive automatic overvolting by Intel's microcode is to blame. There is actually a simple experiment which can determine whether that is true --- remove the fan from the heatsink, or even let the CPU run without a heatsink. As the CPU will automatically throttle once it reaches its designed maximum temperature (and AFAIK that is a hardcoded limit), it will lower its frequency and voltage to maintain that temperature. If this results in a stable CPU, while the one that has great cooling becomes more unstable, it confirms the hypothesis. There are numerous stories of machines where the heatsink was not in contact with the CPU for some reason, yet they remained perfectly stable (but slow) for many years. I can also say that I've had an 8th-gen i7 running at 100% 24x7 with all power and turbo limits disabled, with its temperature constantly at the design limit of 100C, and it has also remained stable for over 5 years.
Reason077 2y ago
> "Sorry, unable to believe: 2-4% failure rate for CPU's?" It seems high to me too. I can't recall ever having a CPU fail and I must have used/owned hundreds of them in my lifetime. But presumably, failure rates in data centres where CPUs are run 24/7 at high temperatures etc are higher than in consumer applications? Interesting that failure rates seem to peak in the summer months, too, and this didn't seem to be explained in the article. Perhaps the data center's cooling is working less effectively in summer?
- sqeaky 2y ago
  How many CPUs have you owned? Compare this to Rolling a die in a tabletop game. How often do you roll a natural one or snake eyes? On a 20-sided die a natural one only comes up 5% of the time. I don't think I'm quite at having owned 40 (desktop) CPUs, but I've had failed CPUs a number of times. On an old Athlon XP 3000+ it had bad CPU cache and was able to change it from crashing repeatedly to working by disabling a large swath of the L2 cache in a bio setting. I am just now retiring a workstation that had an AMD Ryzen 5950x that was generally unstable, that would pass memtest and all the diagnostics I know how to run would pass, but about once a month would print a message on all the consoles I had open about an MCE kernel exception detected on some CPU number at random. One time I had one of those budget TriCore CPUs that had the 4th core turned on in the BIOS by accident, and that generally caused the ton of issues when I figured out it was being detected as a quad core I went in and disabled that last core and it went back to working just fine. I'm sure at least one or two of my Intel machines failed similarly, I've had a number of dead CPUs including a few Pentium 4s back when I used to fool around with overclocking settings, and at least a few dead server chips. And I know that I've had a few systems that simply wouldn't boot even when all the parts were fresh out of the box but would when I'd swap out one part or another, and sometimes that failed part was a CPU whether it be AMD or intel. Oh, I also get to cheat no clue how many dead Solaris CPUs I have seen! Those big Mainframe imitation systems have the ability to hot swap CPUs and it was pretty rare to have a cabinet sized computer where every piece of hardware was fully operational, at least in my role as a legacy maintainer of such things. I bet they worked much better when they rolled off the factory line.
  - 2y ago
high_na_euv 2y ago
https://research.google/pubs/cores-that-dont-count/ https://research.google/pubs/cores-that-dont-count/
philjohn 2y ago
It's pretty widely publicised that even on older Intel CPU's there is a non zero number of servers giving unexpected results at scale - https://arxiv.org/pdf/2102.11245 https://arxiv.org/pdf/2102.11245
whizzter 2y ago
I broke the first CPU I've bought for my own money, it was a PPro 180mhz back in early 1997 (2 days before Intel introduced MMX). I ran it overclocked at 200mhz for a good while until I started getting stability issues, had to downclock to 133mhz to use it after that (even 180 was unstable) and bought a new computer once I started my first "real" job. Seen other HW issues, memory or motherboard(soldered memory) on my ex's laptop that affected only certain adress ranges, memtest86 has been my go-to to check computer health when random crap starts happening since then and I've replaced at least one memory stick on another machine thanks to it.

by w10-1 2y ago

gjsman-1000 2y ago

4% seems very high to me; but CPU errors happen with relative frequency, and design mistakes are common. If you ever run “cat /proc/cpuinfo” on, say, Skylake - Linux will happily tell you it has 5-6 workarounds active for hardware mistakes. CPUs are still pretty darn reliable. Think about how many GHz your CPU runs at, multiplied many instructions per cycle there are, and then calculate the failure rate if there was just 1 mistake per minute. Nothing on earth would compare.

upon_drumhead 2y ago
Mechanical hard drives are absolutely on the same, or higher, level of reliability. It's mind boggling what we can achieve when we really focus on quality outcomes.
- deleted 2y ago
  [deleted]
veqq 2y ago
Wow, you weren't joking: > apic_c1e spectre_v1 spectre_v2 spec_store_bypass swapgs taa itlb_multihit srbds mmio_stale_data retbleed eibrs_pbrsb gds bhi
- asveikau 2y ago
  I mean, a bunch of those are timing issues in speculative execution, you could make an argument that it's working as designed but people didn't anticipate the existence of timing exploits. I'd call that different from computation errors.
  - gjsman-1000 2y ago
    As the original comment suggested, about 5-6 of these are not related to timing exploits (or at least, not the Meltdown/Spectre variants which claim so many patches to their name). Summaries from LKML and Kernel.org: > apic_c1e Both ACPI and MP specifications require that the APIC id in the respective tables must be the same as the APIC id in CPUID. The kernel retrieves the physical package id from the APIC id during the ACPI/MP table scan and builds the physical to logical package map. There exist Virtualbox and Xen implementations which violate the spec. As a result the physical to logical package map, which relies on the ACPI/MP tables does not work on those systems, because the CPUID initialized physical package id does not match the firmware id. This causes system crashes and malfunction due to invalid package mappings. The only way to cure this is to sanitize the physical package id after the CPUID enumeration and yell when the APIC ids are different. If the physical package IDs differ use the package information from the ACPI/MP tables so the existing logical package map just works. > taa TAA is a hardware vulnerability that allows unprivileged speculative access to data which is available in various CPU internal buffers by using asynchronous aborts within an Intel TSX transactional region. > itlb_multihit iTLB multihit is an erratum where some processors may incur a machine check error, possibly resulting in an unrecoverable CPU lockup, when an instruction fetch hits multiple entries in the instruction TLB. This can occur when the page size is changed along with either the physical address or cache type. A malicious guest running on a virtualized system can exploit this erratum to perform a denial of service attack. > srbds SRBDS is a hardware vulnerability that allows MDS techniques to infer values returned from special register accesses. Special register accesses are accesses to off core registers. According to Intel's evaluation, the special register reads that have a security expectation of privacy are RDRAND, RDSEED and SGX EGETKEY. > mmio_stale_data Processor MMIO Stale Data Vulnerabilities are a class of memory-mapped I/O (MMIO) vulnerabilities that can expose data. The sequences of operations for exposing data range from simple to very complex. Because most of the vulnerabilities require the attacker to have access to MMIO, many environments are not affected. System environments using virtualization where MMIO access is provided to untrusted guests may need mitigation. These vulnerabilities are not transient execution attacks. However, these vulnerabilities may propagate stale data into core fill buffers where the data can subsequently be inferred by an unmitigated transient execution attack. Mitigation for these vulnerabilities includes a combination of microcode update and software changes, depending on the platform and usage model. Some of these mitigations are similar to those used to mitigate Microarchitectural Data Sampling (MDS) or those used to mitigate Special Register Buffer Data Sampling (SRBDS).
cwbriscoe 2y ago
Here is for my Ryzen 7700X: sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso
- amluto 2y ago
  FWIW, “sysret_ss_attrs” is a workaround for a design error in AMD’s x86_64 implementation. One might argue that AMD is right because they designed AMD64 in the first place, but IMO this is silly, and AMD’s design is unjustifiable. (I’m the one who characterized this issue on Linux and wrote the test case and the workaround.)

layer8 2y ago

This is consistent with their report for 2019-2021, preceding the issues with the 13th/14th gen: https://www.pugetsystems.com/labs/articles/most-reliable-pc-hardware-of-2021-2279/#Processors_CPUs https://www.pugetsystems.com/labs/articles/most-reliable-pc-... Apparently it was much better in 2018: https://www.pugetsystems.com/labs/articles/most-reliable-pc-hardware-of-2018-1322/#CPU_Processor https://www.pugetsystems.com/labs/articles/most-reliable-pc-... On the other hand, 2011 did show 1.5%: https://www.pugetsystems.com/labs/articles/most-reliable-pc-hardware-of-2014-616/#CPU_Processor https://www.pugetsystems.com/labs/articles/most-reliable-pc-... GPU failure rates also weren't great 10-15 years ago, in particular for AMD: https://www.pugetsystems.com/labs/articles/video-card-failure-rates-by-generation-563/ https://www.pugetsystems.com/labs/articles/video-card-failur...

moffkalast 2y ago
Had three ATI Radeons back in the day, and that's only because the first two died under warranty one after the other lmao. it was even worse before AMD bought them.

Panzer04 2y ago

I wonder if this buckets things like motherboard problems under the same causes. Those numbers do seem very high in general. I guess it would be a pretty useless comparison if they weren't carefully filtering for CPU-only failure though..

Aurornis 2y ago

> Have I been living in a fantasy bubble where CPU's do exactly what you asked of them (and errors come from not holding it right)? These are gaming CPUs clocked right up to the threshold of stability (and in some cases, past it) Server CPUs with ECC RAM are significantly better. However, if you haven’t experienced much CPU instability, you may not have operated at scale where it appears. Get into the scale where operations occur across 100,000s of CPUs and weird things happen constantly.

userbinator 2y ago
One of the previous discussions of this instability noted that it was happening to the server equivalents (which are after all the same die) at stock settings as well.
- Dalewyn 2y ago
  I remember hearing that server motherboards also played a role in overclocking out of the box, which is frankly fucking stupid. I don't recall anything about Raptor Lake-based Xeons suffering from degradation.
  - userbinator 2y ago
    Unless they disable Turbo Boost (which is horrible for performance, but great if you want benchmark consistency), the CPU will automatically overclock until it reaches the limits, adjusting both voltage and frequency. All the evidence I've seen points to electromigration as a cause of this degradation, and IMHO excessively aggressive automatic overvolting by Intel's microcode is to blame. There is actually a simple experiment which can determine whether that is true --- remove the fan from the heatsink, or even let the CPU run without a heatsink. As the CPU will automatically throttle once it reaches its designed maximum temperature (and AFAIK that is a hardcoded limit), it will lower its frequency and voltage to maintain that temperature. If this results in a stable CPU, while the one that has great cooling becomes more unstable, it confirms the hypothesis. There are numerous stories of machines where the heatsink was not in contact with the CPU for some reason, yet they remained perfectly stable (but slow) for many years. I can also say that I've had an 8th-gen i7 running at 100% 24x7 with all power and turbo limits disabled, with its temperature constantly at the design limit of 100C, and it has also remained stable for over 5 years.

Reason077 2y ago

> "Sorry, unable to believe: 2-4% failure rate for CPU's?" It seems high to me too. I can't recall ever having a CPU fail and I must have used/owned hundreds of them in my lifetime. But presumably, failure rates in data centres where CPUs are run 24/7 at high temperatures etc are higher than in consumer applications? Interesting that failure rates seem to peak in the summer months, too, and this didn't seem to be explained in the article. Perhaps the data center's cooling is working less effectively in summer?

sqeaky 2y ago
How many CPUs have you owned? Compare this to Rolling a die in a tabletop game. How often do you roll a natural one or snake eyes? On a 20-sided die a natural one only comes up 5% of the time. I don't think I'm quite at having owned 40 (desktop) CPUs, but I've had failed CPUs a number of times. On an old Athlon XP 3000+ it had bad CPU cache and was able to change it from crashing repeatedly to working by disabling a large swath of the L2 cache in a bio setting. I am just now retiring a workstation that had an AMD Ryzen 5950x that was generally unstable, that would pass memtest and all the diagnostics I know how to run would pass, but about once a month would print a message on all the consoles I had open about an MCE kernel exception detected on some CPU number at random. One time I had one of those budget TriCore CPUs that had the 4th core turned on in the BIOS by accident, and that generally caused the ton of issues when I figured out it was being detected as a quad core I went in and disabled that last core and it went back to working just fine. I'm sure at least one or two of my Intel machines failed similarly, I've had a number of dead CPUs including a few Pentium 4s back when I used to fool around with overclocking settings, and at least a few dead server chips. And I know that I've had a few systems that simply wouldn't boot even when all the parts were fresh out of the box but would when I'd swap out one part or another, and sometimes that failed part was a CPU whether it be AMD or intel. Oh, I also get to cheat no clue how many dead Solaris CPUs I have seen! Those big Mainframe imitation systems have the ability to hot swap CPUs and it was pretty rare to have a cabinet sized computer where every piece of hardware was fully operational, at least in my role as a legacy maintainer of such things. I bet they worked much better when they rolled off the factory line.
- 2y ago

high_na_euv 2y ago

https://research.google/pubs/cores-that-dont-count/ https://research.google/pubs/cores-that-dont-count/

philjohn 2y ago

It's pretty widely publicised that even on older Intel CPU's there is a non zero number of servers giving unexpected results at scale - https://arxiv.org/pdf/2102.11245 https://arxiv.org/pdf/2102.11245

whizzter 2y ago

I broke the first CPU I've bought for my own money, it was a PPro 180mhz back in early 1997 (2 days before Intel introduced MMX). I ran it overclocked at 200mhz for a good while until I started getting stability issues, had to downclock to 133mhz to use it after that (even 180 was unstable) and bought a new computer once I started my first "real" job. Seen other HW issues, memory or motherboard(soldered memory) on my ex's laptop that affected only certain adress ranges, memtest86 has been my go-to to check computer health when random crap starts happening since then and I've replaced at least one memory stick on another machine thanks to it.