6 ms·

Linux 6.1 will make it a bit easier to help spot faulty CPUs

CJefferson 4y ago
This reminds me of Raymond Chen's blog post about overclocked CPUs ( https://devblogs.microsoft.com/oldnewthing/20050412-47/?p=35923 https://devblogs.microsoft.com/oldnewthing/20050412-47/?p=35... ), where Microsoft were getting reports of crashes in CPU instructions which should be unable to cause segfaults, like xor register with itself. I imagine if you are getting such crashes, it would be useful to see exactly where they are coming from!
- TonyTrapp 4y ago
  Another classic blog post on this topic that I like coming back to: https://www.codeofhonor.com/blog/whose-bug-is-this-anyway https://www.codeofhonor.com/blog/whose-bug-is-this-anyway - specifically the "Your computer is broken" section. Apparently Guild Wars ran some computations in the background with a known result, and if those computations didn't yield the correct result, the game would inform the user about a potential hardware fault.
  - ece 4y ago
    If undervolting is the new overclocking, you'd have to wonder if that changes anything.
    - justsomehnguy 4y ago
      For some years I run Win2003 on my home PC, both in overclocked and underclocked modes (Tualeron 1300 if this tells you anything). In the underclocked variant (cooled only by one 80mm fan pretty far from the CPU) I pretty quickly started to recognize the program crashes from the CPU overheating alone. In both cases the reason is too much heat undissipated, but underclocking, in some extreme situations, exhibit the same "undef" behaviour as the overclocked systems.
      - brnt 4y ago
        I had the Tualatin Celeron 1300 too (good times!), but I've never heard them referred to as Tualeron until today!
- nonane 4y ago
  Somewhat similar: I worked on the !chkimg (1) Windbg extension at Microsoft. This little extension was run on incoming kernel dumps to see if it can identify crashes due to bad RAM. It analyzed incoming memory dump files - sent via Windows Error Reporting. If the dump had any code sections for Microsoft code, it would compare that with the binaries that Microsoft officially shipped. If it found evidence of code corruption, it would try to classify the pattern - bad RAM had a distinctive pattern of corruption. We also found other interesting things - i.e. lots of anti-virus patching the kernel code and causing alot of problems (this was way back before PatchGuard). 1. https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/-chkimg https://docs.microsoft.com/en-us/windows-hardware/drivers/de...
tester756 4y ago
is it related to that? >Cores that don't count >https://research.google/pubs/pub50337/ https://research.google/pubs/pub50337/ >We are accustomed to thinking of computers as fail-stop, es- pecially the cores that execute instructions, and most system software implicitly relies on that assumption. During most of the VLSI era, processors that passed manufacturing tests and were operated within specifications have insulated us from this fiction. As fabrication pushes towards smaller feature sizes and more elaborate computational structures, and as increasingly specialized instruction-silicon pairings are intro- duced to improve performance, we have observed ephemeral computational errors that were not detected during manu- facturing tests. These defects cannot always be mitigated by techniques such as microcode updates, and may be correlated to specific components within the processor, allowing small code changes to effect large shifts in reliability. Worse, these failures are often “silent” – the only symptom is an erroneous computation. We refer to a core that develops such behavior as “mercu- rial.” Mercurial cores are extremely rare, but in a large fleet of servers we can observe the disruption they cause, often enough to see them as a distinct problem – one that will re- quire collaboration between hardware designers, processor vendors, and systems software architects. >Violations of lock semantics leading to application data-corruption and crashes.

This reminds me of Raymond Chen's blog post about overclocked CPUs ( https://devblogs.microsoft.com/oldnewthing/20050412-47/?p=35923 https://devblogs.microsoft.com/oldnewthing/20050412-47/?p=35... ), where Microsoft were getting reports of crashes in CPU instructions which should be unable to cause segfaults, like xor register with itself. I imagine if you are getting such crashes, it would be useful to see exactly where they are coming from!

TonyTrapp 4y ago
Another classic blog post on this topic that I like coming back to: https://www.codeofhonor.com/blog/whose-bug-is-this-anyway https://www.codeofhonor.com/blog/whose-bug-is-this-anyway - specifically the "Your computer is broken" section. Apparently Guild Wars ran some computations in the background with a known result, and if those computations didn't yield the correct result, the game would inform the user about a potential hardware fault.
- ece 4y ago
  If undervolting is the new overclocking, you'd have to wonder if that changes anything.
  - justsomehnguy 4y ago
    For some years I run Win2003 on my home PC, both in overclocked and underclocked modes (Tualeron 1300 if this tells you anything). In the underclocked variant (cooled only by one 80mm fan pretty far from the CPU) I pretty quickly started to recognize the program crashes from the CPU overheating alone. In both cases the reason is too much heat undissipated, but underclocking, in some extreme situations, exhibit the same "undef" behaviour as the overclocked systems.
    - brnt 4y ago
      I had the Tualatin Celeron 1300 too (good times!), but I've never heard them referred to as Tualeron until today!
nonane 4y ago
Somewhat similar: I worked on the !chkimg (1) Windbg extension at Microsoft. This little extension was run on incoming kernel dumps to see if it can identify crashes due to bad RAM. It analyzed incoming memory dump files - sent via Windows Error Reporting. If the dump had any code sections for Microsoft code, it would compare that with the binaries that Microsoft officially shipped. If it found evidence of code corruption, it would try to classify the pattern - bad RAM had a distinctive pattern of corruption. We also found other interesting things - i.e. lots of anti-virus patching the kernel code and causing alot of problems (this was way back before PatchGuard). 1. https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/-chkimg https://docs.microsoft.com/en-us/windows-hardware/drivers/de...

tester756 4y ago

is it related to that? >Cores that don't count >https://research.google/pubs/pub50337/ https://research.google/pubs/pub50337/ >We are accustomed to thinking of computers as fail-stop, es- pecially the cores that execute instructions, and most system software implicitly relies on that assumption. During most of the VLSI era, processors that passed manufacturing tests and were operated within specifications have insulated us from this fiction. As fabrication pushes towards smaller feature sizes and more elaborate computational structures, and as increasingly specialized instruction-silicon pairings are intro- duced to improve performance, we have observed ephemeral computational errors that were not detected during manu- facturing tests. These defects cannot always be mitigated by techniques such as microcode updates, and may be correlated to specific components within the processor, allowing small code changes to effect large shifts in reliability. Worse, these failures are often “silent” – the only symptom is an erroneous computation. We refer to a core that develops such behavior as “mercu- rial.” Mercurial cores are extremely rare, but in a large fleet of servers we can observe the disruption they cause, often enough to see them as a distinct problem – one that will re- quire collaboration between hardware designers, processor vendors, and systems software architects. >Violations of lock semantics leading to application data-corruption and crashes.