22 ms·

Sandsifter: find undocumented instructions and bugs on x86 CPU

447 points by argorain 9y ago

partycoder 9y ago
Also from the same author https://sites.google.com/site/xxcantorxdustxx/visual-re https://sites.google.com/site/xxcantorxdustxx/visual-re
- chungy 9y ago
  That looks fun, but the site doesn't seem to have any downloads available?
  - partycoder 9y ago
    There was a demo around somewhere. Not hard to find. There is a similar --but less featured-- open source project. https://github.com/wapiflapi/veles https://github.com/wapiflapi/veles
  - cornchips 9y ago
    https://www.reddit.com/r/ReverseEngineering/comments/1izity/cantordust_a_binary_visualization_tool/ https://www.reddit.com/r/ReverseEngineering/comments/1izity/...
SAI_Peregrinus 9y ago
Christopher Domas does some very cool work. His System Management Mode exploit a few years back was quite nice. It will be interesting to see which processor it is that he found the ring 3 hard lockup instruction in...
- rasz 9y ago
  He works for spooks - Battelle Memorial Institute, a long-time NSA/CIA contractor. One of the places that hires officially retired spies.
  - azinman2 9y ago
    and therefore.... ?
    - qb45 9y ago
      Therefore there is suspicion that spooks may have been using that for years and revealed it once they've had enough for some reason. I'm not familiar with any proven case of such situation, but people speculate.
fovc 9y ago
Here's a link to the slides [pdf]: https://github.com/xoreaxeaxeax/sandsifter/raw/master/references/domas_breaking_the_x86_isa.pdf https://github.com/xoreaxeaxeax/sandsifter/raw/master/refere...
_wmd 9y ago
tl'dr of the slides: Found on one processor... instruction Single malformed instruction in ring 3 locks Tested on 2 Windows kernels, 3 Linux kernels Kernel debugging, serial I/O, interrupt analysis seem to confirm Unfortunately, not finished with responsible disclosure No details available [yet] on chip, vendor, or instructions He's found a new f00f bug, winter 2017 is going to be interesting :)
- tempay 9y ago
  For those not aware: https://en.wikipedia.org/wiki/Pentium_F00F_bug https://en.wikipedia.org/wiki/Pentium_F00F_bug Can these kind of bugs possible to exploit to cause anything more than minor annoyance?
  - hexadecimated 9y ago
    If it works inside a VM, an attacker could potentially cause a widespread denial of service on cloud computing platforms like Azure and AWS.
  - viraptor 9y ago
    Use them to exploit the system itself - not likely. (Unless they cause some specific bad behaviour rather than a crash) But you can definitely use a DoS issue for other effects. For example if someone is using an auth revokation system which fails open, you could kill that part to use expired credentials. Or if you're able to sometimes inject data, you can keep killing the caching systems until your response is the saved one. (Like in DNS hijack)
- qb45 9y ago
  Observation: the length of the censored "XXX hardware bug" text on the slides matches neither Intel, AMD nor Transmeta. Unlikely to be VIA too. Either it's deception or perhaps some obscure low-end embedded vendor. edit: for the curious, it's "(redacted) hardware bugs" :)
  - Veedrac 9y ago
dtx1 9y ago
This is highly interesting. I assume a lot of those are going to be debug and instructions to help the binning process. Some of these might even unlock access to parts of the CPUs we aren't supposed to have access too, opening the doors to custom microcode (unlikely that anyone outside the CPU OEM can do that though) but may allow us to disable "security features" such as the Management Engine. This is a really interesting approach and i would love to see the results ported to other hardware/vendors. The same could potentially be done with GPUs, ARM-CPUs, etc.
- abainbridge 9y ago
  I expect Intel burn a fuse bit at the end of the binning process to prevent such features being accessed in the finished product.
- duskwuff 9y ago
  Separate research has been done on microcode. The general consensus is that Intel's microcode binaries are encrypted, and are secured with a RSA2048-SHA256 signature. http://inertiawar.com/microcode/ http://inertiawar.com/microcode/
  - webreac 9y ago
    I am surprised private keys have never leaked.
    - duskwuff 9y ago
      I wouldn't be surprised if those keys are in a HSM, so they can't be leaked. That'd be the safe way of handling it, and it's well within Intel's resources.
d33 9y ago
...isn't the usability of the tool limited because it's running in userspace, which has fewer privileges in terms of what instructions can be ran?
- askvictor 9y ago
  As the slides say, this approach prevents the system from falling over entirely, while still resolving instructions from deeper rings.
  - d33 9y ago
    Makes sense. I was thinking if there could be a bootable fuzzer of this kind, but you're right that it would be very difficult for it to be both usable and not crash very quickly.
    - dtx1 9y ago
      I think once you found all possible instructions it shouldn't be too hard (for someone much smarter then me) to essentially generate a minimal OS that systematically tries out all found instructions from ring 0. That should dramatically reduce the amount of reboots necessary to actually try them all out compared to bruteforcing them from ring 3
      - d33 9y ago
        What about hangs? I think you'd need a watchdog in order to have the process automated.
        dtx1 9y ago
        I'm sure there are ways to read out a CPU Hang and reset if that happens, if necessary via external hardware^(I'm guessing)^(I don't actually know)
        fooker 9y ago
purpleidea 9y ago
wow... anyone have a link to the video of his talk?
m00dy 9y ago
Someone built a fuzzer for cpus
shdon 9y ago
No instructions there to disable the IME?
- pgeorgi 9y ago
  If anything, I'd expect such a flag to hide behind MSRs (http://wiki.osdev.org/Model_Specific_Registers http://wiki.osdev.org/Model_Specific_Registers) That's a mostly unused namespace of 2^32 64bit registers. To hide things even better, it would also be possible to change behavior based on officially unrelated registers (eg. MSR $x only acts as IME-switch if the calling address also ends in $y and esi is $z)
  - cesarb 9y ago
    They could also be multiplexed (MSR $x is address/command, MSR $y is data). Or require a sequence of operations (write this magic sequence of numbers to MSR $z). Or memory-mapped/IO-mapped (with the mapping enabled/disabled by MSR or PCI registers). Or be locked by the BIOS during the boot sequence. But IMO, it probably can't be disabled at all. The "disabling" would be to change the program it runs to a program which does nothing. So there wouldn't be a "disable IME" bit; there would be bits to either make its memory visible to the main CPU cores, or to read/write to its memory, and it's possible that these bits are accessible only from the IME side, or from SMM.
    - Kliment 9y ago
      The easiest ways to access them is to rewrite that section of the BIOS directly, such as https://github.com/corna/me_cleaner/wiki/How-does-it-work%3F https://github.com/corna/me_cleaner/wiki/How-does-it-work%3F which literally overwrites them with nops
- __jal 9y ago
  Given Intel's behavior around the IME, I rather doubt there's an instruction for that. The only verifiable way to do so that I know of, on some chips, is here: https://hardenedlinux.github.io/firmware/2016/11/17/neutralize_ME_firmware_on_sandybridge_and_ivybridge.html https://hardenedlinux.github.io/firmware/2016/11/17/neutrali... YMMV, not responsible for bricked chips, and note the caveats at the end.
partycoder 9y ago
Lot of weird stuff done happening nowadays in CPUs. There's a lot of mystery in microcode (equivalent to the CPU firmware), the "system management mode" aka protection ring -2, and the infamous management engine.
mcculley 9y ago
This is great. That a program can learn about and exploit the CPU on which it is running from unprivileged userspace reminds me of the notion in Charlie Stross' Accelerando of running a timing attack against the universe to learn about the virtual machine in which we are being simulated.
- michaelmior 9y ago
  It would be even better if there was a web service that would collect these logs for different processors so everyone didn't have to invest the time to run the analysis.
  - tux1968 9y ago
    Well they do mention in the description: The results of a scan can sometimes be difficult for the tools to automatically classify, and may require manual analysis. For help analyzing your results, feel free to send the ./data/log file to xoreaxeaxeax@gmail.com. No personal information, other than the processor make, model, and revision (from /proc/cpuinfo) are included in this log.
    - collinmanderson 9y ago
      It would be nice if there were something like Geekbench where it publicly listed the results from different processors.
    - michaelmior 9y ago
      That's great, but doesn't solve the problem of how long it takes to run a scan.
- qubex 9y ago
  I'd never heard of Charlie Stross or his Accelerando book. Thanks for mentioning that, it looks right up my hard-sci-fi alley.
  - vsviridov 9y ago
hellbanner 9y ago
Related: https://www.theregister.co.uk/2013/05/20/intel_chip_customization/ https://www.theregister.co.uk/2013/05/20/intel_chip_customiz... "Everybody hates the golden screwdriver upgrade approach, where a feature is either hidden or activated through software, but the truth of the matter is that chip makers have been doing this sort of thing for decades – and charging extra for it." ""We are moving rapidly in the direction of realizing that people want unique things and they are going to want them in silicon. In some cases, it will be done in software," said Waxman." Also, Github says "several million" undocumented instructions.. is that right? I don't know much about assembly but that number sounds absurdly high.
- bem94 9y ago
  >> "several million" undocumented instructions.. is that right? Bear in mind that doesnt really mean that there are several million operations / opcode mnemonics which are undocumented but each distinct instructions. It is more likely they are "loose" decodings of other instructions, where changing a single bit of the opcode still causes the CPU to decode the same instruction. Toy example: If I encode my (imaginary ISA) 8bit instruction for "ADD EAX EBX" as 0101_X000 where X is "don't care" then regardless of whether the core gets 0101_0000 or 0101_1000 , it will still execute the ADD instruction. Now imagine your instructions can be upto 16 bytes long, and you see how loose decoding can lead to a lot of instructions which are undocumented, but that the processor is perfectly happy to execute.
  - hellbanner 9y ago
    Thanks. The scale is still hard to wrap my head around but I see what you're saying. Could this tool find hardware backdoors?
    - bem94 9y ago
      I guess that if there was a special "Open backdoor" instruction which was undocumented, then yes I guess it could find it. Backdoors tend to be separate systems which pry into something larger though (like the intel managment engine being a small, separate core which probes the main system). This means you normally need other means of access to the system other than the standard instruction sequence. Again, the IME needed network access to be abused I think, rather than instructions running on the main processor itself. Implementing backdoors is stupid. Opening / accessing them with an undocumented instruction is moronic, but distressingly possible.
pmarreck 9y ago
Is this basically a CPU fuzzer?
- deathanatos 9y ago
  The subtitle at the very top of both the page and the README… > The x86 processor fuzzer
ngneer 9y ago
Chip vendors do the same in the course of validation, and technically even before any silicon has been fabricated, using simulators.
egberts1 9y ago
found another that is QEMU-specific. https://github.com/unicorn-engine/unicorn/issues/364 https://github.com/unicorn-engine/unicorn/issues/364
- egberts1 9y ago
  It is more about modifying executable code space and not making it stick. Good enough for fooling AV.
pwdisswordfish 9y ago
The slides mention an 'apicall' opcode 0ffff0; searching the web turns up nothing but these same slides. Does anyone know anything about it?
- wmu 9y ago
  It seems to be a MS antivirus bug: http://securityaffairs.co/wordpress/60434/hacking/microsoft-windows-defender-flaw.html http://securityaffairs.co/wordpress/60434/hacking/microsoft-...
pbsd 9y ago
For what it's worth, the size-prefixed jcc/call binutils bug had already been fixed a couple of years ago: https://sourceware.org/bugzilla/show_bug.cgi?id=18386 https://sourceware.org/bugzilla/show_bug.cgi?id=18386
rurban 9y ago
Regarding the ring 3 hard lockup he didn't disclose yet: isn't that the recent kaby lake/skylake error, released about a month ago?
brawny 9y ago
Out of curiosity, are there any toy compiler projects out there that try and make use of the incedental instructions? Could you possibly expect to see a with while performance boost (I'm thinking it would be unlikely...)
tonyg 9y ago
I wonder what dbe0, dbe1, and df{c0-c7} do? They are present and undocumented in all of Intel, AMD and VIA's variations (see p4-p5 of the paper).

447 points by argorain 9y ago

Also from the same author https://sites.google.com/site/xxcantorxdustxx/visual-re https://sites.google.com/site/xxcantorxdustxx/visual-re

chungy 9y ago
That looks fun, but the site doesn't seem to have any downloads available?
- partycoder 9y ago
  There was a demo around somewhere. Not hard to find. There is a similar --but less featured-- open source project. https://github.com/wapiflapi/veles https://github.com/wapiflapi/veles
- cornchips 9y ago
  https://www.reddit.com/r/ReverseEngineering/comments/1izity/cantordust_a_binary_visualization_tool/ https://www.reddit.com/r/ReverseEngineering/comments/1izity/...

SAI_Peregrinus 9y ago

Christopher Domas does some very cool work. His System Management Mode exploit a few years back was quite nice. It will be interesting to see which processor it is that he found the ring 3 hard lockup instruction in...

rasz 9y ago
He works for spooks - Battelle Memorial Institute, a long-time NSA/CIA contractor. One of the places that hires officially retired spies.
- azinman2 9y ago
  and therefore.... ?
  - qb45 9y ago
    Therefore there is suspicion that spooks may have been using that for years and revealed it once they've had enough for some reason. I'm not familiar with any proven case of such situation, but people speculate.

fovc 9y ago

Here's a link to the slides [pdf]: https://github.com/xoreaxeaxeax/sandsifter/raw/master/references/domas_breaking_the_x86_isa.pdf https://github.com/xoreaxeaxeax/sandsifter/raw/master/refere...

_wmd 9y ago

tl'dr of the slides: Found on one processor... instruction Single malformed instruction in ring 3 locks Tested on 2 Windows kernels, 3 Linux kernels Kernel debugging, serial I/O, interrupt analysis seem to confirm Unfortunately, not finished with responsible disclosure No details available [yet] on chip, vendor, or instructions He's found a new f00f bug, winter 2017 is going to be interesting :)

tempay 9y ago
For those not aware: https://en.wikipedia.org/wiki/Pentium_F00F_bug https://en.wikipedia.org/wiki/Pentium_F00F_bug Can these kind of bugs possible to exploit to cause anything more than minor annoyance?
- hexadecimated 9y ago
  If it works inside a VM, an attacker could potentially cause a widespread denial of service on cloud computing platforms like Azure and AWS.
- viraptor 9y ago
  Use them to exploit the system itself - not likely. (Unless they cause some specific bad behaviour rather than a crash) But you can definitely use a DoS issue for other effects. For example if someone is using an auth revokation system which fails open, you could kill that part to use expired credentials. Or if you're able to sometimes inject data, you can keep killing the caching systems until your response is the saved one. (Like in DNS hijack)
qb45 9y ago
Observation: the length of the censored "XXX hardware bug" text on the slides matches neither Intel, AMD nor Transmeta. Unlikely to be VIA too. Either it's deception or perhaps some obscure low-end embedded vendor. edit: for the curious, it's "(redacted) hardware bugs" :)
- Veedrac 9y ago

dtx1 9y ago

This is highly interesting. I assume a lot of those are going to be debug and instructions to help the binning process. Some of these might even unlock access to parts of the CPUs we aren't supposed to have access too, opening the doors to custom microcode (unlikely that anyone outside the CPU OEM can do that though) but may allow us to disable "security features" such as the Management Engine. This is a really interesting approach and i would love to see the results ported to other hardware/vendors. The same could potentially be done with GPUs, ARM-CPUs, etc.

abainbridge 9y ago
I expect Intel burn a fuse bit at the end of the binning process to prevent such features being accessed in the finished product.
duskwuff 9y ago
Separate research has been done on microcode. The general consensus is that Intel's microcode binaries are encrypted, and are secured with a RSA2048-SHA256 signature. http://inertiawar.com/microcode/ http://inertiawar.com/microcode/
- webreac 9y ago
  I am surprised private keys have never leaked.
  - duskwuff 9y ago
    I wouldn't be surprised if those keys are in a HSM, so they can't be leaked. That'd be the safe way of handling it, and it's well within Intel's resources.

d33 9y ago

...isn't the usability of the tool limited because it's running in userspace, which has fewer privileges in terms of what instructions can be ran?

askvictor 9y ago
As the slides say, this approach prevents the system from falling over entirely, while still resolving instructions from deeper rings.
- d33 9y ago
  Makes sense. I was thinking if there could be a bootable fuzzer of this kind, but you're right that it would be very difficult for it to be both usable and not crash very quickly.
  - dtx1 9y ago
    I think once you found all possible instructions it shouldn't be too hard (for someone much smarter then me) to essentially generate a minimal OS that systematically tries out all found instructions from ring 0. That should dramatically reduce the amount of reboots necessary to actually try them all out compared to bruteforcing them from ring 3
    - d33 9y ago
      What about hangs? I think you'd need a watchdog in order to have the process automated.
      - dtx1 9y ago
        I'm sure there are ways to read out a CPU Hang and reset if that happens, if necessary via external hardware^(I'm guessing)^(I don't actually know)
        fooker 9y ago

purpleidea 9y ago

wow... anyone have a link to the video of his talk?

m00dy 9y ago

Someone built a fuzzer for cpus

shdon 9y ago

No instructions there to disable the IME?

pgeorgi 9y ago
If anything, I'd expect such a flag to hide behind MSRs (http://wiki.osdev.org/Model_Specific_Registers http://wiki.osdev.org/Model_Specific_Registers) That's a mostly unused namespace of 2^32 64bit registers. To hide things even better, it would also be possible to change behavior based on officially unrelated registers (eg. MSR $x only acts as IME-switch if the calling address also ends in $y and esi is $z)
- cesarb 9y ago
  They could also be multiplexed (MSR $x is address/command, MSR $y is data). Or require a sequence of operations (write this magic sequence of numbers to MSR $z). Or memory-mapped/IO-mapped (with the mapping enabled/disabled by MSR or PCI registers). Or be locked by the BIOS during the boot sequence. But IMO, it probably can't be disabled at all. The "disabling" would be to change the program it runs to a program which does nothing. So there wouldn't be a "disable IME" bit; there would be bits to either make its memory visible to the main CPU cores, or to read/write to its memory, and it's possible that these bits are accessible only from the IME side, or from SMM.
  - Kliment 9y ago
    The easiest ways to access them is to rewrite that section of the BIOS directly, such as https://github.com/corna/me_cleaner/wiki/How-does-it-work%3F https://github.com/corna/me_cleaner/wiki/How-does-it-work%3F which literally overwrites them with nops
__jal 9y ago
Given Intel's behavior around the IME, I rather doubt there's an instruction for that. The only verifiable way to do so that I know of, on some chips, is here: https://hardenedlinux.github.io/firmware/2016/11/17/neutralize_ME_firmware_on_sandybridge_and_ivybridge.html https://hardenedlinux.github.io/firmware/2016/11/17/neutrali... YMMV, not responsible for bricked chips, and note the caveats at the end.

partycoder 9y ago

Lot of weird stuff done happening nowadays in CPUs. There's a lot of mystery in microcode (equivalent to the CPU firmware), the "system management mode" aka protection ring -2, and the infamous management engine.

mcculley 9y ago

This is great. That a program can learn about and exploit the CPU on which it is running from unprivileged userspace reminds me of the notion in Charlie Stross' Accelerando of running a timing attack against the universe to learn about the virtual machine in which we are being simulated.

michaelmior 9y ago
It would be even better if there was a web service that would collect these logs for different processors so everyone didn't have to invest the time to run the analysis.
- tux1968 9y ago
  Well they do mention in the description: The results of a scan can sometimes be difficult for the tools to automatically classify, and may require manual analysis. For help analyzing your results, feel free to send the ./data/log file to xoreaxeaxeax@gmail.com. No personal information, other than the processor make, model, and revision (from /proc/cpuinfo) are included in this log.
  - collinmanderson 9y ago
    It would be nice if there were something like Geekbench where it publicly listed the results from different processors.
  - michaelmior 9y ago
    That's great, but doesn't solve the problem of how long it takes to run a scan.
qubex 9y ago
I'd never heard of Charlie Stross or his Accelerando book. Thanks for mentioning that, it looks right up my hard-sci-fi alley.
- vsviridov 9y ago

hellbanner 9y ago

Related: https://www.theregister.co.uk/2013/05/20/intel_chip_customization/ https://www.theregister.co.uk/2013/05/20/intel_chip_customiz... "Everybody hates the golden screwdriver upgrade approach, where a feature is either hidden or activated through software, but the truth of the matter is that chip makers have been doing this sort of thing for decades – and charging extra for it." ""We are moving rapidly in the direction of realizing that people want unique things and they are going to want them in silicon. In some cases, it will be done in software," said Waxman." Also, Github says "several million" undocumented instructions.. is that right? I don't know much about assembly but that number sounds absurdly high.

bem94 9y ago
>> "several million" undocumented instructions.. is that right? Bear in mind that doesnt really mean that there are several million operations / opcode mnemonics which are undocumented but each distinct instructions. It is more likely they are "loose" decodings of other instructions, where changing a single bit of the opcode still causes the CPU to decode the same instruction. Toy example: If I encode my (imaginary ISA) 8bit instruction for "ADD EAX EBX" as 0101_X000 where X is "don't care" then regardless of whether the core gets 0101_0000 or 0101_1000 , it will still execute the ADD instruction. Now imagine your instructions can be upto 16 bytes long, and you see how loose decoding can lead to a lot of instructions which are undocumented, but that the processor is perfectly happy to execute.
- hellbanner 9y ago
  Thanks. The scale is still hard to wrap my head around but I see what you're saying. Could this tool find hardware backdoors?
  - bem94 9y ago
    I guess that if there was a special "Open backdoor" instruction which was undocumented, then yes I guess it could find it. Backdoors tend to be separate systems which pry into something larger though (like the intel managment engine being a small, separate core which probes the main system). This means you normally need other means of access to the system other than the standard instruction sequence. Again, the IME needed network access to be abused I think, rather than instructions running on the main processor itself. Implementing backdoors is stupid. Opening / accessing them with an undocumented instruction is moronic, but distressingly possible.

pmarreck 9y ago

Is this basically a CPU fuzzer?

deathanatos 9y ago
The subtitle at the very top of both the page and the README… > The x86 processor fuzzer

ngneer 9y ago

Chip vendors do the same in the course of validation, and technically even before any silicon has been fabricated, using simulators.

egberts1 9y ago

found another that is QEMU-specific. https://github.com/unicorn-engine/unicorn/issues/364 https://github.com/unicorn-engine/unicorn/issues/364

egberts1 9y ago
It is more about modifying executable code space and not making it stick. Good enough for fooling AV.

pwdisswordfish 9y ago

The slides mention an 'apicall' opcode 0ffff0; searching the web turns up nothing but these same slides. Does anyone know anything about it?

wmu 9y ago
It seems to be a MS antivirus bug: http://securityaffairs.co/wordpress/60434/hacking/microsoft-windows-defender-flaw.html http://securityaffairs.co/wordpress/60434/hacking/microsoft-...

pbsd 9y ago

For what it's worth, the size-prefixed jcc/call binutils bug had already been fixed a couple of years ago: https://sourceware.org/bugzilla/show_bug.cgi?id=18386 https://sourceware.org/bugzilla/show_bug.cgi?id=18386

rurban 9y ago

Regarding the ring 3 hard lockup he didn't disclose yet: isn't that the recent kaby lake/skylake error, released about a month ago?

brawny 9y ago

Out of curiosity, are there any toy compiler projects out there that try and make use of the incedental instructions? Could you possibly expect to see a with while performance boost (I'm thinking it would be unlikely...)

tonyg 9y ago

I wonder what dbe0, dbe1, and df{c0-c7} do? They are present and undocumented in all of Intel, AMD and VIA's variations (see p4-p5 of the paper).