10 ms·

Pwn the ESP32 Forever: Flash Encryption and SEC. Boot Keys Extraction

216 points by wolframio 7y ago

andrewstuart 7y ago
I believe this requires physical access to the MCU.
- monocasa 7y ago
  It does, but major part of flash encryption is to protect your supply chain. Ie. to keep people from cloning your boards and just dumping your software on them. It's also a bit of security through obscurity (which despite the memes can be an important piece of defense in depth) to make the MCUs a bit more difficult to attack if you don't know the code that's running.
  - XorNot 7y ago
    Except an attacker operating at that scale would just start decapping the chips and inspecting them.
    - monocasa 7y ago
      Decapping and dynamically instrumenting a chip with little pins like you'd need to do is a lot harder than a timing/glitch attack.
- mindslight 7y ago
  Indeed. It's odd to see "Pwn" in the title, and then read the details and have to completely reverse the context. This situation is closer to the original sense of "own", as in "home ownership". If some squirrels get into your home and you evict them, you wouldn't say you "pwnt" your house.
jws 7y ago
Synopsis: Secret keys are embedded in the device's e-fuses and are not readable by normal means because of a protection e-fuse. By measuring current draw during power up an interval is determined to be the time when the CPU is reading the e-fuses. At that time the power supplies are "glitched" from 3.3v to 6v using unspecified patterns from a signal generator. This causes errors in the e-fuse reading, one of which is to make a bank of read protected fuses readable. The read values have errors in them, but multiple runs and statistical error correction can retrieve the actual values. Physical access to the device is required. Security compromise is permanent.
- Scoundreller 7y ago
  Sounds like there’s a “branch if equals” step that checks the fuse in the ROM. You can trigger the code to actually perform your read without branching if you glitch the right part of the cycle. Maybe enough current leaks around at 6V triggering a “not broken” value at the fuse, or just skips the instruction pointer change. My guess is that the e-fuse is checked on every bit-read, so sometimes you don’t get the true value because your glitch isn’t precise enough. Possibly there’s some randomization in the timing of each read, but there’s a signature current draw before each read that you can use to trigger your glitch.
- jeremyjh 7y ago
  Just piggy backing on the top comment to point out that the primary concern here is not necessarily for the security of the devices that you own and physically control (although that could be an issue in some cases, if others can access them too), but for the IP of the OEM which can now be extracted and flashed to cloned boards. So this may well be a serious issue for some of Espressif's customers, who are mostly OEMs, even if it is not an issue for the consumers who buy from that OEM.
  - swiley 7y ago
    In other words, it’s good for users, who’s should have access to the source much less the binary anyway.
LeifCarrotson 7y ago
This is an interesting attack, and certainly looks highly successful in terms of allowing a determined hardware hacker to gain root/bootloader access to a device that the manufacturer has attempted to lock them out of. Glitching with a 6V supply on a 3.3V bus is certainly something I'd want to be a little cautious of if the hardware was more expensive than a $10 dev board - I wouldn't buy a $800 IoT fridge and use this to install alternate firmware just for fun, but it's nice to know it's possible in case my fridge stops working because the manufacturer declares it end-of-life. It's just not clear to me if or how this is a bad thing. The author writes: > This FATAL exploit allows an attacker to decrypt an encrypted firmware because he is now in possession of the AES Flash Encryption Key. > Worst case scenario, he is now able to forge his own valid firmware (using the Secure Boot Key) then encrypt it (using the Flash Encryption Key) to replace the original firmware PERMANENTLY. > This last post closes my security investigation on ESP32, which I consider now as a broken platform. Isn't that a good thing for me as a consumer? I like the ability to decrypt and modify my own devices. I like that this is a permanent modification, unlike eg. dd-wrt where you have to prevent the bootloader from overwriting your software with that of the manufacturer. The only thing I can think of that would be really bad is if I had a device with an ESP32 inside physically stolen then reinstalled by an attacker (or a counterfeit sold to me with malicious code from the vendor) and this exploit allowed them to get private data from my network to an Internet location. But they could already just buy or build their own device, ESP32 or not, to do that. This is only bad for draconian IoT manufacturers who want to enforce their terms of service and artificial limitations on hardware they think consumers are leasing but consumers think they are buying.
- jdnenej 7y ago
  Might as well call the PC a broken platform since you can install your own OS. Imo a platform is broken if the user can't control it.
  - vorpalhex 7y ago
    This would be more akin to jailbreaking your nintendo switch and installing linux. An IOT platform that's intended to be secure can be tricked into revealing it's key. Most consumers aren't going to write custom firmware for their lightbulbs. Of course, I think this exploit is impractical for a lot of cases given how the ESP32 is typically used, but, ymmv.
MrBuddyCasino 7y ago
Props for the effort, but who expects a cheap china MCU for consumer products to be resilient against glitching attacks? You don’t use that stuff in high-security settings anyway. For consumers products resilient to advanced hardware attacks, I can only think of the iPhone and some consoles. Anything else?
- tyingq 7y ago
  I assume it's to keep someone from cloning your product and underselling you with low effort. It's not the danger of one device being compromised.
- Havoc 7y ago
  >You don’t use that stuff in high-security settings anyway. Forgive my ignorance, but what would one use? i.e. what's the high-sec equivalent of a esp32?
  - sigmaprimus 7y ago
    How about zigbee products? A few years ago I worked on the development of a wireless product for use in the traffic control industry. Security was of great importance and we opted to use a more costly "low power mcu radio combination" using the TI MSP430 series processors and Anaren radios. At the time we were more concerned with hackers spoofing our radios signals but thinking about it now the physical hacking of our firmware would be just as big of threat.
  - SEJeff 7y ago
    https://www.arm.com/products/silicon-ip-security https://www.arm.com/products/silicon-ip-security https://www.arm.com/products/silicon-ip-cpu/securcore/sc300 https://www.arm.com/products/silicon-ip-cpu/securcore/sc300
    - Havoc 7y ago
      Cool. Thanks
  - 7y ago
wiremine 7y ago
Some additional info is in Espressif's notification (CVE-2019-17391) which is linked to in the write up: https://www.espressif.com/en/news/Security_Advisory_Concerning_Fault_Injection_and_eFuse_Protections https://www.espressif.com/en/news/Security_Advisory_Concerni... The fix is in ESP32-D0WD-V3 and ESP32-WROVER-E, but of course that doesn't do you any good if you've already shipped product.
- londons_explore 7y ago
  I wonder why they weren't using Public-Private crypto in the first place? Perhaps because it would use up more space in the ROM? If so, I wonder what functionality they dropped from the ROM to add it now? I somehow doubt they made the ROM bigger - that would be very expensive at this stage of the chips lifecycle.
  - SlowRobotAhead 7y ago
    They do have ecc for signing. They just use the AES for encryption because it’s way faster.
  - simpss 7y ago
    the original solution was probably just brought over from ESP8266 with little or no modifications. The 8266 had less resources then ESP32.
  - gmiller123456 7y ago
    Mainly because it wouldn't help with this issue. The code signing used in devices like phones and game consoles is to prevent users from running unapproved software on the given hardware. The issue here is users taking the software and running it on unapproved hardware. At some point, the software has to be decrypted on the physical device to run. The best you can ever do is put enough physical hoops in the way to make it impractical to defeat.
kbumsik 7y ago
I'm wondering, how many companies that use ESP32 actually use the firmware security feature?
- jdnenej 7y ago
  It's used on most consumer devices simply to slow down clone devices being created.
  - slumos 7y ago
    Here's one that definitely did not: > Vulnerability n*2: No security settings (at all) https://limitedresults.com/2019/01/pwn-the-lifx-mini-white/ https://limitedresults.com/2019/01/pwn-the-lifx-mini-white/
    - kbumsik 7y ago
      Yeah that's what I thought, like many of them don't really care about it...
- viraptor 7y ago
  Almost everyone who's selling a real device, in my experience. It's pretty standard to disable firmware reads in embedded cpu products.
  - baybal2 7y ago
    And from my experience, it's virtually none. Since I first started work in OEM electronics in 2007, I only saw that being requested 3 times.
crankylinuxuser 7y ago
Permanent ownership of device that was previously under control from another is now enforced. Regaining control of your stuff is essential.
traverseda 7y ago
If you're interested in this problem space you should definitely check out the chip whisperer. They make some great hardware for doing this kind of test. https://newae.com/tools/chipwhisperer/ https://newae.com/tools/chipwhisperer/
gorgoiler 7y ago
Something about e-fuses seems quite mystical. The idea of a computer program deliberately and permanently damaging its own hardware (or hardware it is attached to) using a mechanism so close to regular operation (current flowing in memory) but for a good reason rather than to cause harm, and in such an information rich way. Different to say a robotic tool using its tooltip to maim itself and different to one robot building another, because at the e-fuse level of detail it’s so much more information sense. Perhaps it’s like a tattoo? Perhaps I’m thinking of the ship tattoos in Surface Detail by Iain M Banks?
- oakwhiz 7y ago
  Another way to think of it is, a memory-mapped set of wires that are made too thin, and when read, correspond to ones. If ones are written to that region, they become zeroes every time they are read afterward. This is kind of the reverse of how memory normally works. Of course the actual mechanism used in OTP memory is different...
  - peter_d_sherman 7y ago
    I think it would be interesting to learn the history of e-fuses as applies to CPU architecture... That is, where/how/when were they invented, what was the first CPU to use them, for what purpose, and which CPU's have used them since that point in time... Maybe I'll post to Ask HN or one of the StackOverflow websites about this in the future...
planteen 7y ago
I've heard e-fuses in general are vulnerable to optical inspection under polarized light after deliding a part. So if someone capable really wanted to clone a device, it's very possible they already were able to get the e-fuse key values. I once used the e-fuse feature of another part for bootloader integrity. I wasn't worried about encryption, but the part would validate the bootloader integrity when encrypted. If integrity failed, the part would keep searching for a valid image. It was an easy way for some protection against flash corruption.
- baybal2 7y ago
  Indeed, this is the reason ICs used in credit cards don't use them, but embedded flash can still be mechanically probed, and this is how EMV cards are allegedly being cloned
_def 7y ago
I didn't even know the ESP32 had these security capabilities, but I guess I've not been missing out ;)
throwaway77384 7y ago
Could this get around a locked bootloader on a Sony Xperia Z5 Compact? (As in, the normal sony-website-enabled bootloader unlock NOT allowed when checking in service menu) If so, there might be a bounty out for it...
chli 7y ago
From the article: > I quickly identify a pure HW processing 500us before the beginning of the UART ascii strings ‘ets June 2018’ corresponding to the BootROM process. > This HW activity is probably the eFuses Controller initialisation, and a load of the eFuses values in some dedicated buffer memory, to be used by the Flash controller for further steps). How one would come to this specific conclusion without having any prior knowledge of the boot rom ?
- codebeaker 7y ago
  The efuses are still part of hardware initialization, before one gets into the bootrom, so it's feasible to assume that this 500us is still "hardware" init, during which time it's typical for an MCU to be reading input from pins to know about voltage, clock, mode selector jumpers, etc. And, the way all of those things work is by setting registers so that they're visible in the software either _still_ in a register, or mapped into the address space. Edit: I checked your profile and see that you're an embedded engineer, so I must have missed some nuance in your question, because power glitching the boot sequence to mess with hardware init it a really popular vector for attacking embedded devices. Please feel free to disregard my reply.

216 points by wolframio 7y ago

andrewstuart 7y ago

I believe this requires physical access to the MCU.

monocasa 7y ago
It does, but major part of flash encryption is to protect your supply chain. Ie. to keep people from cloning your boards and just dumping your software on them. It's also a bit of security through obscurity (which despite the memes can be an important piece of defense in depth) to make the MCUs a bit more difficult to attack if you don't know the code that's running.
- XorNot 7y ago
  Except an attacker operating at that scale would just start decapping the chips and inspecting them.
  - monocasa 7y ago
    Decapping and dynamically instrumenting a chip with little pins like you'd need to do is a lot harder than a timing/glitch attack.
mindslight 7y ago
Indeed. It's odd to see "Pwn" in the title, and then read the details and have to completely reverse the context. This situation is closer to the original sense of "own", as in "home ownership". If some squirrels get into your home and you evict them, you wouldn't say you "pwnt" your house.

jws 7y ago

Synopsis: Secret keys are embedded in the device's e-fuses and are not readable by normal means because of a protection e-fuse. By measuring current draw during power up an interval is determined to be the time when the CPU is reading the e-fuses. At that time the power supplies are "glitched" from 3.3v to 6v using unspecified patterns from a signal generator. This causes errors in the e-fuse reading, one of which is to make a bank of read protected fuses readable. The read values have errors in them, but multiple runs and statistical error correction can retrieve the actual values. Physical access to the device is required. Security compromise is permanent.

Scoundreller 7y ago
Sounds like there’s a “branch if equals” step that checks the fuse in the ROM. You can trigger the code to actually perform your read without branching if you glitch the right part of the cycle. Maybe enough current leaks around at 6V triggering a “not broken” value at the fuse, or just skips the instruction pointer change. My guess is that the e-fuse is checked on every bit-read, so sometimes you don’t get the true value because your glitch isn’t precise enough. Possibly there’s some randomization in the timing of each read, but there’s a signature current draw before each read that you can use to trigger your glitch.
jeremyjh 7y ago
Just piggy backing on the top comment to point out that the primary concern here is not necessarily for the security of the devices that you own and physically control (although that could be an issue in some cases, if others can access them too), but for the IP of the OEM which can now be extracted and flashed to cloned boards. So this may well be a serious issue for some of Espressif's customers, who are mostly OEMs, even if it is not an issue for the consumers who buy from that OEM.
- swiley 7y ago
  In other words, it’s good for users, who’s should have access to the source much less the binary anyway.

LeifCarrotson 7y ago

This is an interesting attack, and certainly looks highly successful in terms of allowing a determined hardware hacker to gain root/bootloader access to a device that the manufacturer has attempted to lock them out of. Glitching with a 6V supply on a 3.3V bus is certainly something I'd want to be a little cautious of if the hardware was more expensive than a $10 dev board - I wouldn't buy a $800 IoT fridge and use this to install alternate firmware just for fun, but it's nice to know it's possible in case my fridge stops working because the manufacturer declares it end-of-life. It's just not clear to me if or how this is a bad thing. The author writes: > This FATAL exploit allows an attacker to decrypt an encrypted firmware because he is now in possession of the AES Flash Encryption Key. > Worst case scenario, he is now able to forge his own valid firmware (using the Secure Boot Key) then encrypt it (using the Flash Encryption Key) to replace the original firmware PERMANENTLY. > This last post closes my security investigation on ESP32, which I consider now as a broken platform. Isn't that a good thing for me as a consumer? I like the ability to decrypt and modify my own devices. I like that this is a permanent modification, unlike eg. dd-wrt where you have to prevent the bootloader from overwriting your software with that of the manufacturer. The only thing I can think of that would be really bad is if I had a device with an ESP32 inside physically stolen then reinstalled by an attacker (or a counterfeit sold to me with malicious code from the vendor) and this exploit allowed them to get private data from my network to an Internet location. But they could already just buy or build their own device, ESP32 or not, to do that. This is only bad for draconian IoT manufacturers who want to enforce their terms of service and artificial limitations on hardware they think consumers are leasing but consumers think they are buying.

jdnenej 7y ago
Might as well call the PC a broken platform since you can install your own OS. Imo a platform is broken if the user can't control it.
- vorpalhex 7y ago
  This would be more akin to jailbreaking your nintendo switch and installing linux. An IOT platform that's intended to be secure can be tricked into revealing it's key. Most consumers aren't going to write custom firmware for their lightbulbs. Of course, I think this exploit is impractical for a lot of cases given how the ESP32 is typically used, but, ymmv.

MrBuddyCasino 7y ago

Props for the effort, but who expects a cheap china MCU for consumer products to be resilient against glitching attacks? You don’t use that stuff in high-security settings anyway. For consumers products resilient to advanced hardware attacks, I can only think of the iPhone and some consoles. Anything else?

tyingq 7y ago
I assume it's to keep someone from cloning your product and underselling you with low effort. It's not the danger of one device being compromised.
Havoc 7y ago
>You don’t use that stuff in high-security settings anyway. Forgive my ignorance, but what would one use? i.e. what's the high-sec equivalent of a esp32?
- sigmaprimus 7y ago
  How about zigbee products? A few years ago I worked on the development of a wireless product for use in the traffic control industry. Security was of great importance and we opted to use a more costly "low power mcu radio combination" using the TI MSP430 series processors and Anaren radios. At the time we were more concerned with hackers spoofing our radios signals but thinking about it now the physical hacking of our firmware would be just as big of threat.
- SEJeff 7y ago
  https://www.arm.com/products/silicon-ip-security https://www.arm.com/products/silicon-ip-security https://www.arm.com/products/silicon-ip-cpu/securcore/sc300 https://www.arm.com/products/silicon-ip-cpu/securcore/sc300
  - Havoc 7y ago
    Cool. Thanks
- 7y ago

wiremine 7y ago

Some additional info is in Espressif's notification (CVE-2019-17391) which is linked to in the write up: https://www.espressif.com/en/news/Security_Advisory_Concerning_Fault_Injection_and_eFuse_Protections https://www.espressif.com/en/news/Security_Advisory_Concerni... The fix is in ESP32-D0WD-V3 and ESP32-WROVER-E, but of course that doesn't do you any good if you've already shipped product.

londons_explore 7y ago
I wonder why they weren't using Public-Private crypto in the first place? Perhaps because it would use up more space in the ROM? If so, I wonder what functionality they dropped from the ROM to add it now? I somehow doubt they made the ROM bigger - that would be very expensive at this stage of the chips lifecycle.
- SlowRobotAhead 7y ago
  They do have ecc for signing. They just use the AES for encryption because it’s way faster.
- simpss 7y ago
  the original solution was probably just brought over from ESP8266 with little or no modifications. The 8266 had less resources then ESP32.
- gmiller123456 7y ago
  Mainly because it wouldn't help with this issue. The code signing used in devices like phones and game consoles is to prevent users from running unapproved software on the given hardware. The issue here is users taking the software and running it on unapproved hardware. At some point, the software has to be decrypted on the physical device to run. The best you can ever do is put enough physical hoops in the way to make it impractical to defeat.

kbumsik 7y ago

I'm wondering, how many companies that use ESP32 actually use the firmware security feature?

jdnenej 7y ago
It's used on most consumer devices simply to slow down clone devices being created.
- slumos 7y ago
  Here's one that definitely did not: > Vulnerability n*2: No security settings (at all) https://limitedresults.com/2019/01/pwn-the-lifx-mini-white/ https://limitedresults.com/2019/01/pwn-the-lifx-mini-white/
  - kbumsik 7y ago
    Yeah that's what I thought, like many of them don't really care about it...
viraptor 7y ago
Almost everyone who's selling a real device, in my experience. It's pretty standard to disable firmware reads in embedded cpu products.
- baybal2 7y ago
  And from my experience, it's virtually none. Since I first started work in OEM electronics in 2007, I only saw that being requested 3 times.

crankylinuxuser 7y ago

Permanent ownership of device that was previously under control from another is now enforced. Regaining control of your stuff is essential.

traverseda 7y ago

If you're interested in this problem space you should definitely check out the chip whisperer. They make some great hardware for doing this kind of test. https://newae.com/tools/chipwhisperer/ https://newae.com/tools/chipwhisperer/

gorgoiler 7y ago

Something about e-fuses seems quite mystical. The idea of a computer program deliberately and permanently damaging its own hardware (or hardware it is attached to) using a mechanism so close to regular operation (current flowing in memory) but for a good reason rather than to cause harm, and in such an information rich way. Different to say a robotic tool using its tooltip to maim itself and different to one robot building another, because at the e-fuse level of detail it’s so much more information sense. Perhaps it’s like a tattoo? Perhaps I’m thinking of the ship tattoos in Surface Detail by Iain M Banks?

oakwhiz 7y ago
Another way to think of it is, a memory-mapped set of wires that are made too thin, and when read, correspond to ones. If ones are written to that region, they become zeroes every time they are read afterward. This is kind of the reverse of how memory normally works. Of course the actual mechanism used in OTP memory is different...
- peter_d_sherman 7y ago
  I think it would be interesting to learn the history of e-fuses as applies to CPU architecture... That is, where/how/when were they invented, what was the first CPU to use them, for what purpose, and which CPU's have used them since that point in time... Maybe I'll post to Ask HN or one of the StackOverflow websites about this in the future...

planteen 7y ago

I've heard e-fuses in general are vulnerable to optical inspection under polarized light after deliding a part. So if someone capable really wanted to clone a device, it's very possible they already were able to get the e-fuse key values. I once used the e-fuse feature of another part for bootloader integrity. I wasn't worried about encryption, but the part would validate the bootloader integrity when encrypted. If integrity failed, the part would keep searching for a valid image. It was an easy way for some protection against flash corruption.

baybal2 7y ago
Indeed, this is the reason ICs used in credit cards don't use them, but embedded flash can still be mechanically probed, and this is how EMV cards are allegedly being cloned

_def 7y ago

I didn't even know the ESP32 had these security capabilities, but I guess I've not been missing out ;)

throwaway77384 7y ago

Could this get around a locked bootloader on a Sony Xperia Z5 Compact? (As in, the normal sony-website-enabled bootloader unlock NOT allowed when checking in service menu) If so, there might be a bounty out for it...

chli 7y ago

From the article: > I quickly identify a pure HW processing 500us before the beginning of the UART ascii strings ‘ets June 2018’ corresponding to the BootROM process. > This HW activity is probably the eFuses Controller initialisation, and a load of the eFuses values in some dedicated buffer memory, to be used by the Flash controller for further steps). How one would come to this specific conclusion without having any prior knowledge of the boot rom ?

codebeaker 7y ago
The efuses are still part of hardware initialization, before one gets into the bootrom, so it's feasible to assume that this 500us is still "hardware" init, during which time it's typical for an MCU to be reading input from pins to know about voltage, clock, mode selector jumpers, etc. And, the way all of those things work is by setting registers so that they're visible in the software either _still_ in a register, or mapped into the address space. Edit: I checked your profile and see that you're an embedded engineer, so I must have missed some nuance in your question, because power glitching the boot sequence to mess with hardware init it a really popular vector for attacking embedded devices. Please feel free to disregard my reply.