13 ms·

Intel CPU Bugs of 2015 and Implications for the Future

204 points by tolien 11y ago

joosters 11y ago
Whenever I've glanced through the errata for processors, it makes me think it's a miracle that the CPUs run as well as they do... https://www-ssl.intel.com/content/dam/www/public/us/en/documents/specification-updates/4th-gen-core-family-desktop-specification-update.pdf#21 https://www-ssl.intel.com/content/dam/www/public/us/en/docum...
- ciupicri 11y ago
  I guess that's why Xeon is a bit behind, so that they can fix those bugs.
  - ksec 11y ago
    My thoughts as well, I guess that is Why things are taking a lot longer for Xeon E5.
    - rincebrain 11y ago
      Xeon E5 and E7 are always longer because getting SMP working reliably can reveal all sorts of bugs that never arise when you have One True Processor Die.
      - deelowe 11y ago
        And more PCIE lanes, more cores, more cache, etc...
- nickpsecurity 11y ago
  Take a look at the challenges at 28nm: http://electronicdesign.com/digital-ics/understanding-28-nm-soc-design-arm-based-cores http://electronicdesign.com/digital-ics/understanding-28-nm-... You're right that it's damn-near a miracle that they work. The miracle is the investment of literally billions in tooling and projects to figure out how to get it to work. Then, it's still so expensive that cutting-edge nodes at 28nm and below where Intel often works are exclusive to only the elite engineers and tools if it's anything of significant size.
  - _yosefk 11y ago
    I dunno, Adapteva taped out at 28nm with 5 people. Do you mean that full custom design is particularly hard at 28nm or that it's hard for EDA vendors? 28nm is also a bulk node, I think it ought to have been easier to understand for everyone relatively to FinFET nodes.
    - nickpsecurity 11y ago
      I've given them props before as they're a special case. They wrote a whole blog article on how they kept things cheap. In comments in past 48 hours, they mentioned something like $2-3 million in 3 years to get their relatively simple design done. I also remember they deal with HPC crowd that pays good money for even low-volume. So, they're just unusual across the board. Doesn't tell us much about a large design except to say physics will fight us every step of the way. Various approaches like asynchronous and self-correcting circuits might help us a bit. Look at Achronix FPGA's to see the power of the former.
    - gluggymug 11y ago
      Uh, tape out doesn't mean you've verified anything properly. Looking at the various Intel bugs found, I can't help sniggering a little. Verification mistakes happen all the time though. It's just a matter of when the bugs are found: if found later, it has much greater impact. On the 28nm issue, getting your design to synthesize to 28 nm depends on the architecture and your constraints. Trickier data paths make it harder to get it meeting timing. Where your IOs are placed, where your RAMs are placed and how much you have, each thing can make it harder and harder to synthesize. Adapteva doesn't have much RAM I believe.
      - _yosefk 11y ago
        Well, yeah, I'm just saying it wasn't a picnic at 40nm or 90nm, either. It didn't go from a walk in the park to a nightmare at 28nm. Also I believe all the bugs in TFA are logical bugs where the circuit would have misbehaved in a logical simulation, not the kind where the circuit "flips zeros to ones" or vice versa because of physical implementation issues. In this sense advanced nodes make things worse only indirectly by enabling larger, more complex designs.
- velox_io 11y ago
  It's also amazing that with over 5 billion transistors (on a high E5). It's not transistors the cause the errors but the CPU design.
tgb 11y ago
I'm curious: what would a fix for these look like? Does it mean a new revision to be bought, a recall, a software patch?
- felixfurtak 11y ago
  presumably just a microcode update
- iwwr 11y ago
  You don't fix them, the tools and systems (gcc or the operating systems) have to work around them (i.e. not allow the sequences of instructions that generate the bad behaviour).
  - rincebrain 11y ago
    That's one way, and sometimes the only way on certain platforms (hello Intel Quark), but - those kinds of workarounds can cause problems on other platforms in turn - they don't stop malicious people from making use of such exploits, if they don't require such privileges as to make them pointless once you can use them So you generally prefer to be able to update the CPU microcode or flip a killbit to disable the relevant silicon path, to having to recompile everything on your architecture to include a workaround detection.
- colejohnson66 11y ago
  Usually just a microcode update. Before microcode though, a bug would usually just be worked around. IIRC, Windows would actually trap the FDIV instruction on buggy Pentiums and compute the true result with software. With enough pressure, a recall might be issued, but it took a while for Intel to finally recall the buggy Pentiums.
  - wglb 11y ago
    For the FDIV bug, IBM pulled and replaced all shipped bad chips. Not every vendor did that.
lumpypua 11y ago
Awesome article! From footnote 1: In the time that it takes a sophisticated attacker to find a hole in Azure that will cause an hour of disruption across 1% of VMs, that same attacker could probably completely take down ten unicorns for a much longer period of time. And yet, these attackers are hyper focused on the most hardened targets. Why is that? I can't figure it out, any fellow posters have ideas? Well funded unicorns with iffy infrastructure seem like good candidates to pay out DDoS ransoms, while Azure, AWS, etc will never pay out a ransom. If attackers want credit cards and email addresses, softer targets seem like a better choice as well. It doesn't seem like an attacker can extract any value from hardened targets unless they have state-sponsored or corporate espionage level resources and skill.
- RickS 11y ago
  Some people just really like hard problems.
- Drakim 11y ago
  Might the attacker have the mindset that "Azure's business depends on it's reputation as a reliable and safe host, so they are more likely to give in to my demands." The unicorns, while they still have much to lose by having their service denied, don't necessarily live and breathe on the image of stability and uptime.
- deleted 11y ago
  [deleted]
Animats 11y ago
That doesn't even include any CPU bugs deliberately installed as backdoors.[1] [1] http://www.eteknix.com/expert-says-nsa-have-backdoors-built-into-intel-and-amd-processors/ http://www.eteknix.com/expert-says-nsa-have-backdoors-built-...
- nickpsecurity 11y ago
  It's called AMT/vPro. It's in the brochure. People tell me it's even on when the system is off. All that circuitry is probably in most of the family just to reduce NRE costs. Couldn't be more ideal. And to think some people here mock people for worrying about their random number function while ignoring their official backdoor and its implications. (sighs)
Andys 11y ago
On the bright side: There are worse things than a processor lockup (which is easy to spot when it happens). And the other bug was in the newest architecture (Skylake) and did not affect Xeons. Arguably the memory Row Hammer exploit was far worse, and is a sign of how bad things can get outside the CPU.
nisa 11y ago
There was a talk on 32C3 about CPU bugs and the insane work that goes into preventing them: https://www.youtube.com/watch?v=eDmv0sDB1Ak https://www.youtube.com/watch?v=eDmv0sDB1Ak
nickpsecurity 11y ago
Old news says Kris Kaspersky: http://cs.dartmouth.edu/~sergey/cs258/2010/D2T1%20-%20Kris%20Kaspersky%20-%20Remote%20Code%20Execution%20Through%20Intel%20CPU%20Bugs.pdf http://cs.dartmouth.edu/~sergey/cs258/2010/D2T1%20-%20Kris%2... Will likely continue to get worse in full-custom designs like Intel's since complexity keeps going up but ease of modeling and verification doesn't. On the other end, look up VAMP from Verisoft, SSP from Sandia, or AAMP7G from Rockwell-Collins if you want to see what high-assurance processors look like. They ran error-free during testing IIRC with a ton of validation. Sandia SSP was first-pass. In any case, they're all kind of simple compared to Intel's stuff. That's on purpose given there's an upper limit to how much complexity you can squeeze into a chip without significant errata. One can expand such methods to larger SOC's but that's not what big vendors are doing [out of necessity]. And that has security implications that aren't going away.

204 points by tolien 11y ago

Whenever I've glanced through the errata for processors, it makes me think it's a miracle that the CPUs run as well as they do... https://www-ssl.intel.com/content/dam/www/public/us/en/documents/specification-updates/4th-gen-core-family-desktop-specification-update.pdf#21 https://www-ssl.intel.com/content/dam/www/public/us/en/docum...

ciupicri 11y ago
I guess that's why Xeon is a bit behind, so that they can fix those bugs.
- ksec 11y ago
  My thoughts as well, I guess that is Why things are taking a lot longer for Xeon E5.
  - rincebrain 11y ago
    Xeon E5 and E7 are always longer because getting SMP working reliably can reveal all sorts of bugs that never arise when you have One True Processor Die.
    - deelowe 11y ago
      And more PCIE lanes, more cores, more cache, etc...
nickpsecurity 11y ago
Take a look at the challenges at 28nm: http://electronicdesign.com/digital-ics/understanding-28-nm-soc-design-arm-based-cores http://electronicdesign.com/digital-ics/understanding-28-nm-... You're right that it's damn-near a miracle that they work. The miracle is the investment of literally billions in tooling and projects to figure out how to get it to work. Then, it's still so expensive that cutting-edge nodes at 28nm and below where Intel often works are exclusive to only the elite engineers and tools if it's anything of significant size.
- _yosefk 11y ago
  I dunno, Adapteva taped out at 28nm with 5 people. Do you mean that full custom design is particularly hard at 28nm or that it's hard for EDA vendors? 28nm is also a bulk node, I think it ought to have been easier to understand for everyone relatively to FinFET nodes.
  - nickpsecurity 11y ago
    I've given them props before as they're a special case. They wrote a whole blog article on how they kept things cheap. In comments in past 48 hours, they mentioned something like $2-3 million in 3 years to get their relatively simple design done. I also remember they deal with HPC crowd that pays good money for even low-volume. So, they're just unusual across the board. Doesn't tell us much about a large design except to say physics will fight us every step of the way. Various approaches like asynchronous and self-correcting circuits might help us a bit. Look at Achronix FPGA's to see the power of the former.
  - gluggymug 11y ago
    Uh, tape out doesn't mean you've verified anything properly. Looking at the various Intel bugs found, I can't help sniggering a little. Verification mistakes happen all the time though. It's just a matter of when the bugs are found: if found later, it has much greater impact. On the 28nm issue, getting your design to synthesize to 28 nm depends on the architecture and your constraints. Trickier data paths make it harder to get it meeting timing. Where your IOs are placed, where your RAMs are placed and how much you have, each thing can make it harder and harder to synthesize. Adapteva doesn't have much RAM I believe.
    - _yosefk 11y ago
      Well, yeah, I'm just saying it wasn't a picnic at 40nm or 90nm, either. It didn't go from a walk in the park to a nightmare at 28nm. Also I believe all the bugs in TFA are logical bugs where the circuit would have misbehaved in a logical simulation, not the kind where the circuit "flips zeros to ones" or vice versa because of physical implementation issues. In this sense advanced nodes make things worse only indirectly by enabling larger, more complex designs.
velox_io 11y ago
It's also amazing that with over 5 billion transistors (on a high E5). It's not transistors the cause the errors but the CPU design.

tgb 11y ago

I'm curious: what would a fix for these look like? Does it mean a new revision to be bought, a recall, a software patch?

felixfurtak 11y ago
presumably just a microcode update
iwwr 11y ago
You don't fix them, the tools and systems (gcc or the operating systems) have to work around them (i.e. not allow the sequences of instructions that generate the bad behaviour).
- rincebrain 11y ago
  That's one way, and sometimes the only way on certain platforms (hello Intel Quark), but - those kinds of workarounds can cause problems on other platforms in turn - they don't stop malicious people from making use of such exploits, if they don't require such privileges as to make them pointless once you can use them So you generally prefer to be able to update the CPU microcode or flip a killbit to disable the relevant silicon path, to having to recompile everything on your architecture to include a workaround detection.
colejohnson66 11y ago
Usually just a microcode update. Before microcode though, a bug would usually just be worked around. IIRC, Windows would actually trap the FDIV instruction on buggy Pentiums and compute the true result with software. With enough pressure, a recall might be issued, but it took a while for Intel to finally recall the buggy Pentiums.
- wglb 11y ago
  For the FDIV bug, IBM pulled and replaced all shipped bad chips. Not every vendor did that.

lumpypua 11y ago

Awesome article! From footnote 1: In the time that it takes a sophisticated attacker to find a hole in Azure that will cause an hour of disruption across 1% of VMs, that same attacker could probably completely take down ten unicorns for a much longer period of time. And yet, these attackers are hyper focused on the most hardened targets. Why is that? I can't figure it out, any fellow posters have ideas? Well funded unicorns with iffy infrastructure seem like good candidates to pay out DDoS ransoms, while Azure, AWS, etc will never pay out a ransom. If attackers want credit cards and email addresses, softer targets seem like a better choice as well. It doesn't seem like an attacker can extract any value from hardened targets unless they have state-sponsored or corporate espionage level resources and skill.

RickS 11y ago
Some people just really like hard problems.
Drakim 11y ago
Might the attacker have the mindset that "Azure's business depends on it's reputation as a reliable and safe host, so they are more likely to give in to my demands." The unicorns, while they still have much to lose by having their service denied, don't necessarily live and breathe on the image of stability and uptime.
deleted 11y ago
[deleted]

Animats 11y ago

That doesn't even include any CPU bugs deliberately installed as backdoors.[1] [1] http://www.eteknix.com/expert-says-nsa-have-backdoors-built-into-intel-and-amd-processors/ http://www.eteknix.com/expert-says-nsa-have-backdoors-built-...

nickpsecurity 11y ago
It's called AMT/vPro. It's in the brochure. People tell me it's even on when the system is off. All that circuitry is probably in most of the family just to reduce NRE costs. Couldn't be more ideal. And to think some people here mock people for worrying about their random number function while ignoring their official backdoor and its implications. (sighs)

Andys 11y ago

On the bright side: There are worse things than a processor lockup (which is easy to spot when it happens). And the other bug was in the newest architecture (Skylake) and did not affect Xeons. Arguably the memory Row Hammer exploit was far worse, and is a sign of how bad things can get outside the CPU.

nisa 11y ago

There was a talk on 32C3 about CPU bugs and the insane work that goes into preventing them: https://www.youtube.com/watch?v=eDmv0sDB1Ak https://www.youtube.com/watch?v=eDmv0sDB1Ak

nickpsecurity 11y ago

Old news says Kris Kaspersky: http://cs.dartmouth.edu/~sergey/cs258/2010/D2T1%20-%20Kris%20Kaspersky%20-%20Remote%20Code%20Execution%20Through%20Intel%20CPU%20Bugs.pdf http://cs.dartmouth.edu/~sergey/cs258/2010/D2T1%20-%20Kris%2... Will likely continue to get worse in full-custom designs like Intel's since complexity keeps going up but ease of modeling and verification doesn't. On the other end, look up VAMP from Verisoft, SSP from Sandia, or AAMP7G from Rockwell-Collins if you want to see what high-assurance processors look like. They ran error-free during testing IIRC with a ton of validation. Sandia SSP was first-pass. In any case, they're all kind of simple compared to Intel's stuff. That's on purpose given there's an upper limit to how much complexity you can squeeze into a chip without significant errata. One can expand such methods to larger SOC's but that's not what big vendors are doing [out of necessity]. And that has security implications that aren't going away.