4 ms·

Simple instructions for freezing a Skylake Processor

86 points by yuhong 11y ago

MichaelBurge 11y ago
That's interesting that a BIOS upgrade can fix this. Maybe the BIOS is able to disable an instruction set like AVX?
- ah- 11y ago
  This talk from the 32c3 just two weeks ago looks at exactly that problem. They have "chicken bits" (not sure if that's the technical term :)) which can turn off certain features, and they can patch microcode. https://media.ccc.de/v/32c3-7171-when_hardware_must_just_work https://media.ccc.de/v/32c3-7171-when_hardware_must_just_wor... Software design and testing is hard, but what happens when each bug fix can cost months of delay and millions of dollars? In this talk we’ll take a behind-the-scenes look at the challenges in the design of a very complex, yet critical piece of hardware: the modern x86 CPU.
  - jeswin 11y ago
    Great video. Very informative. Thanks.
  - rincebrain 11y ago
    Killbits is the one I've heard most commonly - as the talk says, most modern processors have a bunch because it's impossible to test every possible edge case a priori. (Good recent examples include the AMD lockup bug that Matt Dillon found, and the nasty counter-example of the Intel Quark segfault bug, which can't be worked around so easily because the Quark doesn't have killbits. "Oops.")
    - wyager 11y ago
      >most modern processors have a bunch because it's impossible to test every possible edge case a priori. Only with the sort of engineering most current CPU manufacturers use. It's possible to have formally verified hardware; it's just expensive with current tech. Tools are improving, though. I suspect the next cutting edge will be the unification of functional-language-to-hardware compilers like Clash or Lambda-CCC with fully dependent languages like Agda or Idris, where you can statically verify arbitrary properties though the type system.
      - throwupper247 11y ago
        I've just written an Idris programm that determined "your mothers phone number", hows that for arbitrary facts? (tip of the hat to xkcd.)
- awalton 11y ago
  It's also able to load microcode patches very early on in the machine bring-up process. Microcode can be anywhere as simple as a single hardware enable bit "turn on this hardware unit" or as complex as "this instruction should be decoded as this sequence of microops." It's hard to know exactly what's wrong without more internal, never-to-see-the-light-of-day Intel information, but it's possible the patch is as simple as tweaking the decoding, or disabling an optimization. It's doubtful a fix like this would require disabling an entire instruction set or hardware unit (like the TSX fix required; those are fairly rare bugs, even for Intel). Specifically, given that it happens to be a bug that only happens with FMA3 disabled and hyperthreading enabled, it's probably some kind of hardware scheduling deadlock, and the patch could be as simple as "when the decoder sees this sequence of instructions, add some strategically placed NOPs to avoid the deadlock."
pierrec 11y ago
Better change the link to this: https://communities.intel.com/thread/96157?start=0&tstart=0 https://communities.intel.com/thread/96157?start=0&tstart=0 (which was probably the intended link) Let's hope that the planned BIOS update won't involve disabling an entire feature set like what happened with TSX: http://www.anandtech.com/show/8376/intel-disables-tsx-instructions-erratum-found-in-haswell-haswelleep-broadwelly http://www.anandtech.com/show/8376/intel-disables-tsx-instru...
duncans 11y ago
There was me thinking this would be about some crazy new overclocking technique ...
Rolpa 11y ago
I just had a newly-built Skylake desktop completely lock up twice in the past week while using a resource intensive development tool. I'm curious as to whether this is related - I was freaking out over the possibility of my ~1200$ machine being defective in some way.
conistonwater 11y ago
So how exactly does one debug such a thing? I'm assuming it's not like a normal debugger? They say "simple instructions", but it seems to involve running a specific program for "minutes or hours", so they haven't figured out what exactly causes this, but Intel somehow has.
- nhaehnle 11y ago
  I recommend this CCC presentation by an AMD engineer: https://media.ccc.de/v/32c3-7171-when_hardware_must_just_work https://media.ccc.de/v/32c3-7171-when_hardware_must_just_wor... The bottom line is that as a last resort debugging method, all modern chips can do scan dumps, which means reading out the state of every flip-flop on the chip. Obviously I have no idea whether that was needed by Intel to find this bug. It's possible that a higher-level emulation or some other technique was sufficient. But when all else fails, you can still do scan dumps.
  - wyager 11y ago
    That sounds incredibly expensive to route. Every flip flop?
    - trsohmers 11y ago
      It's a process called scan chain insertion. As a chip designer, I can tell you that DFT (Design for Test, which scan chain falls under) can be a real PITA even though the tools are automated. EDIT: Should have also said that it really isn't that big in terms of area... a register plus two wires is nothing in the scheme of things. https://en.wikipedia.org/wiki/Scan_chain https://en.wikipedia.org/wiki/Scan_chain
    - nhaehnle 11y ago
      Kind of, but it's a chain. It works by shifting bits through all flip-flops until they've all "dropped out" of an I/O pin. So you don't need any expensive muxes or crossbars. It's actually one of the largest and most realistic real-world application of TSP and related problems.
    - 11y ago
kup0 11y ago
I had a Gigabyte board (GA-Z170N-Gaming 5) that was very unstable with Skylake. The newly released beta BIOS update resolved the issue. Not sure that it's related to this issue directly but it did involve random freezes.

86 points by yuhong 11y ago

MichaelBurge 11y ago

That's interesting that a BIOS upgrade can fix this. Maybe the BIOS is able to disable an instruction set like AVX?

ah- 11y ago
This talk from the 32c3 just two weeks ago looks at exactly that problem. They have "chicken bits" (not sure if that's the technical term :)) which can turn off certain features, and they can patch microcode. https://media.ccc.de/v/32c3-7171-when_hardware_must_just_work https://media.ccc.de/v/32c3-7171-when_hardware_must_just_wor... Software design and testing is hard, but what happens when each bug fix can cost months of delay and millions of dollars? In this talk we’ll take a behind-the-scenes look at the challenges in the design of a very complex, yet critical piece of hardware: the modern x86 CPU.
- jeswin 11y ago
  Great video. Very informative. Thanks.
- rincebrain 11y ago
  Killbits is the one I've heard most commonly - as the talk says, most modern processors have a bunch because it's impossible to test every possible edge case a priori. (Good recent examples include the AMD lockup bug that Matt Dillon found, and the nasty counter-example of the Intel Quark segfault bug, which can't be worked around so easily because the Quark doesn't have killbits. "Oops.")
  - wyager 11y ago
    >most modern processors have a bunch because it's impossible to test every possible edge case a priori. Only with the sort of engineering most current CPU manufacturers use. It's possible to have formally verified hardware; it's just expensive with current tech. Tools are improving, though. I suspect the next cutting edge will be the unification of functional-language-to-hardware compilers like Clash or Lambda-CCC with fully dependent languages like Agda or Idris, where you can statically verify arbitrary properties though the type system.
    - throwupper247 11y ago
      I've just written an Idris programm that determined "your mothers phone number", hows that for arbitrary facts? (tip of the hat to xkcd.)
awalton 11y ago
It's also able to load microcode patches very early on in the machine bring-up process. Microcode can be anywhere as simple as a single hardware enable bit "turn on this hardware unit" or as complex as "this instruction should be decoded as this sequence of microops." It's hard to know exactly what's wrong without more internal, never-to-see-the-light-of-day Intel information, but it's possible the patch is as simple as tweaking the decoding, or disabling an optimization. It's doubtful a fix like this would require disabling an entire instruction set or hardware unit (like the TSX fix required; those are fairly rare bugs, even for Intel). Specifically, given that it happens to be a bug that only happens with FMA3 disabled and hyperthreading enabled, it's probably some kind of hardware scheduling deadlock, and the patch could be as simple as "when the decoder sees this sequence of instructions, add some strategically placed NOPs to avoid the deadlock."

pierrec 11y ago

Better change the link to this: https://communities.intel.com/thread/96157?start=0&tstart=0 https://communities.intel.com/thread/96157?start=0&tstart=0 (which was probably the intended link) Let's hope that the planned BIOS update won't involve disabling an entire feature set like what happened with TSX: http://www.anandtech.com/show/8376/intel-disables-tsx-instructions-erratum-found-in-haswell-haswelleep-broadwelly http://www.anandtech.com/show/8376/intel-disables-tsx-instru...

duncans 11y ago

There was me thinking this would be about some crazy new overclocking technique ...

Rolpa 11y ago

I just had a newly-built Skylake desktop completely lock up twice in the past week while using a resource intensive development tool. I'm curious as to whether this is related - I was freaking out over the possibility of my ~1200$ machine being defective in some way.

conistonwater 11y ago

So how exactly does one debug such a thing? I'm assuming it's not like a normal debugger? They say "simple instructions", but it seems to involve running a specific program for "minutes or hours", so they haven't figured out what exactly causes this, but Intel somehow has.

nhaehnle 11y ago
I recommend this CCC presentation by an AMD engineer: https://media.ccc.de/v/32c3-7171-when_hardware_must_just_work https://media.ccc.de/v/32c3-7171-when_hardware_must_just_wor... The bottom line is that as a last resort debugging method, all modern chips can do scan dumps, which means reading out the state of every flip-flop on the chip. Obviously I have no idea whether that was needed by Intel to find this bug. It's possible that a higher-level emulation or some other technique was sufficient. But when all else fails, you can still do scan dumps.
- wyager 11y ago
  That sounds incredibly expensive to route. Every flip flop?
  - trsohmers 11y ago
    It's a process called scan chain insertion. As a chip designer, I can tell you that DFT (Design for Test, which scan chain falls under) can be a real PITA even though the tools are automated. EDIT: Should have also said that it really isn't that big in terms of area... a register plus two wires is nothing in the scheme of things. https://en.wikipedia.org/wiki/Scan_chain https://en.wikipedia.org/wiki/Scan_chain
  - nhaehnle 11y ago
    Kind of, but it's a chain. It works by shifting bits through all flip-flops until they've all "dropped out" of an I/O pin. So you don't need any expensive muxes or crossbars. It's actually one of the largest and most realistic real-world application of TSP and related problems.
  - 11y ago

kup0 11y ago

I had a Gigabyte board (GA-Z170N-Gaming 5) that was very unstable with Skylake. The newly released beta BIOS update resolved the issue. Not sure that it's related to this issue directly but it did involve random freezes.