6 ms·

Well, it's not like Intel CPUs don't have similar bugs. See Skylake for example - the list of known errata starts on page 27 and continues on through page 63 :

by jjuhl 10y ago

Well, it's not like Intel CPUs don't have similar bugs. See Skylake for example - the list of known errata starts on page 27 and continues on through page 63 : http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/desktop-6th-gen-core-family-spec-update.pdf http://www.intel.com/content/dam/www/public/us/en/documents/...

yeukhon 10y ago
Interesting. I guess unlike software, these sorts of bugs can't be fixed without producing a new one right? Do CPU and GPU manufactures do any types of fuzzing?
- jjuhl 10y ago
  Microcode is software running on your CPU. And many problems can be fixed with a microcode update, but not all - see the workarounds listed in the PDF I linked - note especially how many say "None identified".
  - yeukhon 10y ago
    Okay, thanks for clarification. I was thinking about the floating bug which they had to recall the defective processors: https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug
    - justsid 10y ago
      Hard to say if something like this would be patchable via a microcode update. But regardless, back then your CPU didn’t run software to run your software, so a new hardware revision was in order.
- deleted 10y ago
  [deleted]
- monocasa 10y ago
  CPU ones definitely fuzz, and formally verify parts of their chips. Unfortunately fuzzing ultimately has a random component, which doesn't really prove that you got all of these bugs.
  - valarauca1 10y ago
    (To addon to parent poster) Unfortunately fuzzing ultimately has a random component An instruction which accepts 2 registers and returns 1 register has a 192bit problem space to validate. This complexity is present in an instruction as simple as `add`. As AVX2 instructions which accepts 3 registers and outputs 1 has a 1024bit problem space to validate. This occurred in FMA3 with a ~512bit problem space. Repeat for _every_ instruction (HUNDREDS). You can see how a few bugs slip though the cracks. The problem space is as large as some cryptographic functions!! I'm honestly surprised we don't see more of them.
    - cwzwarich 10y ago
      The specific result produced by the data path is probably not very relevant in the case of a lockup. The control path involved with instruction decoding, register renaming, out-of-order execution, SMT, etc. is generally the cause of issues like this. With interactions between different blocks of the CPU and the size of some of the data structures involved, the full verification space is much, much larger.
      - valarauca1 10y ago
        This isn't a lock up. If you read the source (TechPowerUp is terrible why it isn't banned is beyond me) it is actually an _illegal instruction_ error. Which just crashes the current application. Also if SMT is not disable the error doesn't occur. Also if the chip isn't over clocked the error doesn't occur. So it is very clearly power related.
        wongarsu 10y ago
        Quoting from http://forum.hwbot.org/showthread.php?t=167605 http://forum.hwbot.org/showthread.php?t=167605: > this always hard freezes the computer: >- At all clock speeds. >- When running single-threaded, it happens to any core that I pin it to. It should be at worst an illegal instruction, but instead the whole core freezes, even on underclocked computers.
- rrdharan 10y ago
  They do tremendous amounts of validation. I believe random generation of input data is part of that. Here's an old heavily cited paper from Intel on the topic; I'm sure their state of the art has advanced considerably in the intervening 17 years since its publication: http://dl.acm.org/citation.cfm?id=623013 http://dl.acm.org/citation.cfm?id=623013
  - greglindahl 10y ago
    "crashme" is one venerable program that does this kind of fuzzing -- I managed to find a bug in a cpu with it once, which does not speak well of their QA department.
    - xenadu02 10y ago
      A former coworker was a QA manager at Intel. He said it was an explicit decision to cut back on validation and QA, which is why he wasn't at Intel anymore. The general feeling was that they had "overreacted" to the Pentium FDIV bug and needed to move faster. YMMV, I have no inside knowledge.
    - i336_ 10y ago
      I'm very curious what CPU it was, and what the bug was.
      - greglindahl 10y ago
        Broadcom Sibyte 1250. I had pre-release silicon, and there was a known bug that prefetch would occasionally hang the chip. I wanted to have a little fun goofing around, so I modified crashme to replace prefetches in the randomly generated code with a noop. A few minutes later, I hung the chip. If I identified the right erratum later, it was: if there is a branch in the delay slot of a branch in a delay slot which is mispredicted, the cpu hangs. It was fixed by making this sequence throw an illegal instruction exception. (It was undefined behavior already, I think.)
        i336_ 10y ago
- deleted 10y ago
  [deleted]
- matthew-wegner 10y ago
  In this case the fix is adding a resistor: "If your system does not use SERIRQ and BIOS puts SERIRQ in Quiet-Mode, then the weak external pull up resistor is not required. All other cases must implement an external pull-up resistor, 8.2k to 10k, tied to 3.3V" https://www-ssl.intel.com/content/dam/www/public/us/en/documents/specification-updates/atom-c2000-family-spec-update.pdf https://www-ssl.intel.com/content/dam/www/public/us/en/docum... Depending on the motherboard, those pins may be exposed already. Here's what the fix looks like a Synology unit: https://www.reddit.com/r/synology/comments/609u1l/c2538_clock_fix_confirmed_ds2415/ https://www.reddit.com/r/synology/comments/609u1l/c2538_cloc...
  - yuhong 10y ago
    So I assume that the problem is that SERIRQ is left floating causing too much of a load on the LPC clock it depends on, right?
zeusk 10y ago
or the recent C2xxx bricking 'fiasco' http://www.guru3d.com/news-story/intel-atom-c2000-chips-are-bricking-products.html http://www.guru3d.com/news-story/intel-atom-c2000-chips-are-... https://www.reddit.com/r/wallstreetbets/comments/5sqg30/pot_intel_shitstorm_incoming_intel_atom_c2xxx/ https://www.reddit.com/r/wallstreetbets/comments/5sqg30/pot_...
- jsheard 10y ago
  Intel's chipsets aren't immune either, the first revision of P67 and H67 had a flaw that caused the SATA ports to slowly overvolt themselves to death. They ended up having to recall a huge number of motherboards.
  - seanp2k2 10y ago
    Yep, I had to RMA a P8P67Pro because of this https://en.wikipedia.org/wiki/Sandy_Bridge#Cougar_Point_chipset_flaw https://en.wikipedia.org/wiki/Sandy_Bridge#Cougar_Point_chip...
djsumdog 10y ago
It's crazy how today we can just release microcode updates to patch the CPU, compared to the original Pentium bug which requires the operating system to be patched.
- gsnedders 10y ago
  To be fair, there are definitely occasional bugs that require OS patches.
- flamedoge 10y ago
  I'm presuming it's just bios update to disable certain optimizations within the CPU.. meaning they were already in place to be A/B tested.. just flipping bit on/off.
  - deleted 10y ago
    [deleted]
  - vanjoe 10y ago
    I don't think this is true. I'm not an expert on application scale CPUs, but I think they actually have ROMs that control state machines that execute the instructions. This is similar to how regular instructions are stored in memory.
    - xenadu02 10y ago
      My understanding is that there are several forms of microcode updates. Sometimes the microcode can do simple patches like inserting a nop between problematic instruction combinations. This usually doesn't have much (if any) negative performance implications. The instruction decoder (again AFAIK) is fairly programmable. However if there is a more serious flaw in the actual silicon the microcode must rewrite the problem instructions to emulate the correct behavior and that can make it extremely slow - think emulating floating point but not quite as bad.
    - therein 10y ago
      Correct, look at this page for an example: https://www.bedroomlan.org/projects/mcasm https://www.bedroomlan.org/projects/mcasm // Minimum Microcode file. cond OP:4; cond uaddr:3; signal /MEMIO = ....1; signal MREAD = ...1.; signal MWRITE = ..1..; field REGMOVE = XX___; signal MAR <- PC = 01...; signal IR <- MEM = 10...; start OP=XXXX; // A tiny fetch instruction /MEMIO, MREAD, MAR <- PC; // hold; // Same as MEMIO, MREAD, MAR <- PC hold, -MREAD, IR <- MEM; // Same as MEMIO, MAR <- PC, IR <- MEM // End of file.

by jjuhl 10y ago

yeukhon 10y ago

Interesting. I guess unlike software, these sorts of bugs can't be fixed without producing a new one right? Do CPU and GPU manufactures do any types of fuzzing?

jjuhl 10y ago
Microcode is software running on your CPU. And many problems can be fixed with a microcode update, but not all - see the workarounds listed in the PDF I linked - note especially how many say "None identified".
- yeukhon 10y ago
  Okay, thanks for clarification. I was thinking about the floating bug which they had to recall the defective processors: https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug
  - justsid 10y ago
    Hard to say if something like this would be patchable via a microcode update. But regardless, back then your CPU didn’t run software to run your software, so a new hardware revision was in order.
deleted 10y ago
[deleted]
monocasa 10y ago
CPU ones definitely fuzz, and formally verify parts of their chips. Unfortunately fuzzing ultimately has a random component, which doesn't really prove that you got all of these bugs.
- valarauca1 10y ago
  (To addon to parent poster) Unfortunately fuzzing ultimately has a random component An instruction which accepts 2 registers and returns 1 register has a 192bit problem space to validate. This complexity is present in an instruction as simple as `add`. As AVX2 instructions which accepts 3 registers and outputs 1 has a 1024bit problem space to validate. This occurred in FMA3 with a ~512bit problem space. Repeat for _every_ instruction (HUNDREDS). You can see how a few bugs slip though the cracks. The problem space is as large as some cryptographic functions!! I'm honestly surprised we don't see more of them.
  - cwzwarich 10y ago
    The specific result produced by the data path is probably not very relevant in the case of a lockup. The control path involved with instruction decoding, register renaming, out-of-order execution, SMT, etc. is generally the cause of issues like this. With interactions between different blocks of the CPU and the size of some of the data structures involved, the full verification space is much, much larger.
    - valarauca1 10y ago
      This isn't a lock up. If you read the source (TechPowerUp is terrible why it isn't banned is beyond me) it is actually an _illegal instruction_ error. Which just crashes the current application. Also if SMT is not disable the error doesn't occur. Also if the chip isn't over clocked the error doesn't occur. So it is very clearly power related.
      - wongarsu 10y ago
        Quoting from http://forum.hwbot.org/showthread.php?t=167605 http://forum.hwbot.org/showthread.php?t=167605: > this always hard freezes the computer: >- At all clock speeds. >- When running single-threaded, it happens to any core that I pin it to. It should be at worst an illegal instruction, but instead the whole core freezes, even on underclocked computers.
rrdharan 10y ago
They do tremendous amounts of validation. I believe random generation of input data is part of that. Here's an old heavily cited paper from Intel on the topic; I'm sure their state of the art has advanced considerably in the intervening 17 years since its publication: http://dl.acm.org/citation.cfm?id=623013 http://dl.acm.org/citation.cfm?id=623013
- greglindahl 10y ago
  "crashme" is one venerable program that does this kind of fuzzing -- I managed to find a bug in a cpu with it once, which does not speak well of their QA department.
  - xenadu02 10y ago
    A former coworker was a QA manager at Intel. He said it was an explicit decision to cut back on validation and QA, which is why he wasn't at Intel anymore. The general feeling was that they had "overreacted" to the Pentium FDIV bug and needed to move faster. YMMV, I have no inside knowledge.
  - i336_ 10y ago
    I'm very curious what CPU it was, and what the bug was.
    - greglindahl 10y ago
      Broadcom Sibyte 1250. I had pre-release silicon, and there was a known bug that prefetch would occasionally hang the chip. I wanted to have a little fun goofing around, so I modified crashme to replace prefetches in the randomly generated code with a noop. A few minutes later, I hung the chip. If I identified the right erratum later, it was: if there is a branch in the delay slot of a branch in a delay slot which is mispredicted, the cpu hangs. It was fixed by making this sequence throw an illegal instruction exception. (It was undefined behavior already, I think.)
      - i336_ 10y ago
deleted 10y ago
[deleted]
matthew-wegner 10y ago
In this case the fix is adding a resistor: "If your system does not use SERIRQ and BIOS puts SERIRQ in Quiet-Mode, then the weak external pull up resistor is not required. All other cases must implement an external pull-up resistor, 8.2k to 10k, tied to 3.3V" https://www-ssl.intel.com/content/dam/www/public/us/en/documents/specification-updates/atom-c2000-family-spec-update.pdf https://www-ssl.intel.com/content/dam/www/public/us/en/docum... Depending on the motherboard, those pins may be exposed already. Here's what the fix looks like a Synology unit: https://www.reddit.com/r/synology/comments/609u1l/c2538_clock_fix_confirmed_ds2415/ https://www.reddit.com/r/synology/comments/609u1l/c2538_cloc...
- yuhong 10y ago
  So I assume that the problem is that SERIRQ is left floating causing too much of a load on the LPC clock it depends on, right?

zeusk 10y ago

or the recent C2xxx bricking 'fiasco' http://www.guru3d.com/news-story/intel-atom-c2000-chips-are-bricking-products.html http://www.guru3d.com/news-story/intel-atom-c2000-chips-are-... https://www.reddit.com/r/wallstreetbets/comments/5sqg30/pot_intel_shitstorm_incoming_intel_atom_c2xxx/ https://www.reddit.com/r/wallstreetbets/comments/5sqg30/pot_...

jsheard 10y ago
Intel's chipsets aren't immune either, the first revision of P67 and H67 had a flaw that caused the SATA ports to slowly overvolt themselves to death. They ended up having to recall a huge number of motherboards.
- seanp2k2 10y ago
  Yep, I had to RMA a P8P67Pro because of this https://en.wikipedia.org/wiki/Sandy_Bridge#Cougar_Point_chipset_flaw https://en.wikipedia.org/wiki/Sandy_Bridge#Cougar_Point_chip...

djsumdog 10y ago

It's crazy how today we can just release microcode updates to patch the CPU, compared to the original Pentium bug which requires the operating system to be patched.

gsnedders 10y ago
To be fair, there are definitely occasional bugs that require OS patches.
flamedoge 10y ago
I'm presuming it's just bios update to disable certain optimizations within the CPU.. meaning they were already in place to be A/B tested.. just flipping bit on/off.
- deleted 10y ago
  [deleted]
- vanjoe 10y ago
  I don't think this is true. I'm not an expert on application scale CPUs, but I think they actually have ROMs that control state machines that execute the instructions. This is similar to how regular instructions are stored in memory.
  - xenadu02 10y ago
    My understanding is that there are several forms of microcode updates. Sometimes the microcode can do simple patches like inserting a nop between problematic instruction combinations. This usually doesn't have much (if any) negative performance implications. The instruction decoder (again AFAIK) is fairly programmable. However if there is a more serious flaw in the actual silicon the microcode must rewrite the problem instructions to emulate the correct behavior and that can make it extremely slow - think emulating floating point but not quite as bad.
  - therein 10y ago
    Correct, look at this page for an example: https://www.bedroomlan.org/projects/mcasm https://www.bedroomlan.org/projects/mcasm // Minimum Microcode file. cond OP:4; cond uaddr:3; signal /MEMIO = ....1; signal MREAD = ...1.; signal MWRITE = ..1..; field REGMOVE = XX___; signal MAR <- PC = 01...; signal IR <- MEM = 10...; start OP=XXXX; // A tiny fetch instruction /MEMIO, MREAD, MAR <- PC; // hold; // Same as MEMIO, MREAD, MAR <- PC hold, -MREAD, IR <- MEM; // Same as MEMIO, MAR <- PC, IR <- MEM // End of file.