7 ms·

Supermicro hardware weaknesses let researchers backdoor an IBM cloud server

84 points by bauc 8y ago

Dahoon 8y ago
Why does the sign say back OP door EN?
Twirrim 8y ago
There's a good talk on securing bare metal cloud servers against such attacks, that was given at B-Sides in Portland, by two respected security people who worked on securing bare metal cloud infrastructure for one cloud provider: https://www.youtube.com/watch?v=PEVVRkd-wPM https://www.youtube.com/watch?v=PEVVRkd-wPM Boiled down it comes to: Trust absolutely nothing. When a customer finishes with a server, wipe absolutely everything, re-flash every single bit of firmware on every single device in the machine, and don't use the standard flashing mechanisms to do so. It's worth a listen/watch.
jfindley 8y ago
Is there any chance they have stumbled upon the same weakness that Bloomberg tried to report[0]? The industry at large has been pretty sceptical of Bloomberg's claims, and rightly so, but what if they just got the details wrong and it was this (or similar) vulnerability in the BMC software, rather than a dedicated spy chip, that they meant to write about? 0: https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies https://www.bloomberg.com/news/features/2018-10-04/the-big-h...
- mtgx 8y ago
  I think it's possible Bloomberg's "tiny chip" was a BMC, and that BMC was found with a backdoor in it, but the backdoor may not have necessarily come "from factory," but was later added by someone else. Either way, this looks terrible for SuperMicro: "Yeah, our servers don't come with built-in backdoors - it's just SuperEasy™ for attackers to add one once shipped -- please buy more now."
rbanffy 8y ago
It's a bit shocking that the people who develop the firmware for these embedded computers is not more concerned with their security. I understand you are not supposed to attach them to untrusted networks, but can we really call any network trusted these days? Do people still teach about the Maginot Line in schools?
- jandeboevrie 8y ago
  Cheap Chinese labor. Security costs time and time is money. So why bother if it isn't giving direct profits? Customers still buy them servers so, no problem in the revenue stream (yet)
- dsfyu404ed 8y ago
  >Do people still teach about the Maginot Line in schools? No. There are plenty of other opportunities to teach the value of making your adversary (be that adversary man made or natural) work every step of the way and they are missed too. Nobody (for large values of "nobody") gets taught about the value of layered or redundant systems until college and those that do get taught it in college usually only touch on it in their mandatory ethics elective. FWIW I named the last firewall I configured "Little Maginot".
  - abbracadabbra 8y ago
    Another (perhaps more) common use of the term Maginot Line is as a metaphor for expensive efforts that offer a false sense of security.
- dfox 8y ago
  It is even more shocking considering the fact that exposing the BMC directly to the internet is not only often recommended by vendor but in many cases (eg. collocation) actually only way to use it.
  - bpye 8y ago
    Jumpboxes? You need at least two for it to work though. Update A from B and then update B from A.
wyldfire 8y ago
Does this vindicate Bloomberg or is it a different BMC vulnerability?
- Twirrim 8y ago
  Totally different. This is a compromise via a known firmware issue with the standard components on a motherboard. The Bloomberg article was about a hidden chip being installed on a motherboard that provided a backdoor.
  - wyldfire 8y ago
    Strikes me as odd that they didn't acknowledge the recent history of reporting on Supermicro BMC issues (in order to clearly distinguish them).
    - jlgaddis 8y ago
      If you read about both, it becomes pretty clear (at least, it did to me) that they aren't related. Bloomberg's story was about a (supposed) backdoored chip installed into the server at the factory, this is about flipping a bit in the firmware installed on the existing BMC.
      - datavirtue 8y ago
        A microscopic chip that merely makes it one step easier to exploit these controllers.
- throwawaymath 8y ago
  Nope, nothing to do with the Bloomberg story. While we're at it, Bloomberg still hasn't issued a retraction or provided more evidence to support their claims. This is despite the massive number of refutations from the entire industry.
Zenst 8y ago
This highlights how the sterilization process many cloud providers (IBM in this instance) have is not cleaning out every nook and cranny. They should, audit every bit of firmware (indeed it's odd how the researchers changed one bit in the BMC firmware and no checksum flagged it up on boot) and whilst this is daunting, it isn't that hard as they just have to compare and verify it is the same as the known safe image. Sure they could blindly reflash, but then they would miss any attempted expliotations and equally shorted the life of the hardware by increasing the odds of the flash memory failure. Whilst people see BMC's as one avenue, a server/pc has many components, all with their own firmware and in many cases, own CPU. Be that a network card, graphics card and even keyboards and mice (though the later, not so much a factor in server environments, still a consideration). Security is and always will be a mindset. You need to think like somebody who wants to break into your environment, and then counter those ways. But so many avenues. Imagine your sat at your desk as an administrator and one morning you get a nice shiny, cool top of the range keyboard sent, dressed up as a gift. How many would think, cool, plug it in and feel all fuzzy? How many would audit the firmware on that keyboard? How many would question the random gift at every level? I'm sure IBM are not the only ones who would fall foul of this avenue of BMC exploitation, but I'm disappointed that for me, basic sanity checks in their sanitisation process to decommission and recommission a server are being overlooked. Still, when you hire a car - do they audit the cars management engine firmware? Do they erase previous BT and WIFI connections stored on the radio? Well, from my experience - they don't. Remember - you can pay an expert all the money in the World, but do check their work.
- jacquesm 8y ago
  > Do they erase previous BT and WIFI connections stored on the radio? I've pulled many contact details from cached data on rental vehicles. Always worth checking what the stuff you pair your phone with asks for and keeps.
jlgaddis 8y ago
> This indicated that the servers' BMC firmware was not re-flashed during the server reclamation process. I'm not surprised. Unless you pay for their "enterprisey" datacenter management products (which are still relatively new), it's a PITA to perform BIOS and BMC firmware updates. Additionally, Supermicro specifically recommends that you DO NOT flash the firmware unless you are experiencing issues that a new version is suppose to fix -- unlike pretty much every other vendor (like Dell, who makes it fairly easy to do so).
- greglindahl 8y ago
  Flashing the Supermicro BMC is so easy and reliable that I used to do it frequently. Flashing the Supermicro BIOS, yeah, that's a disaster.
paulfurtado 8y ago
I'm a little curious why the BMC hardware is even exposed to the customer at all. Is it provided as a feature or is this just an oversight? Can't they just hide all of this hardware from the customer's OS?
- pas 8y ago
  It should sit on a physically separated network, with the firmware regularly updated.
  - paulfurtado 8y ago
    Wasn't clear to me from the article whether the firmware was modified via the BMC's network interface or whether it was done locally over some hardware interface exposed to the customer's OS. Was this really done over the network?
    - toast0 8y ago
      Softlayer runs two separate networks, the public network, and the private network (vlans per customer, but also Softlayer tools are accessible from customer vlan, and customer vlan is accessible to Softlayer tools). The BMCs are connected to the private network. My understanding is that access to the BMC via the network is password protected, and customers don't get a password with enough access to flash via the network. Of course, older versions of the Supermicro BMC firmware had network accessible vulnerabilities, and current versions may have vulnerabilities as well. But impitool from the host can configure the BMC, and I suspect either flash the BMC or at least configure the BMC with a user that can flash it (or Supermicro tools can likely be run on the host to flash the BMC)
- toast0 8y ago
  Bare metal customers get access to the whole machine. We get to access the BMC for remote management (console, power management, etc). Apparently, you can configure the BMC via tools running in the host OS, without needing to authenticate to the BMC; I've used this to fix BMC provisioning errors, that would otherwise require a SL tech to fix via physical console in the bios screen.
datavirtue 8y ago
Move along, nothing to see here, folks. Everyone uses these servers and Amazon, Apple and others have already declared them safe and free of exploits for all time. Step over here for a taste of cognitive dissonance.
deleted 8y ago
[deleted]
cmurf 8y ago
Could someone speak to the write endurance of BMC flash, vs whatever flash BIOS/UEFI lives in, vs NVRAM where now certain boot parameters are stored for persistence? Is it a total non-concern? It's not a user replaceable part.
londons_explore 8y ago
The article looks written to hurt supermicro here. It looks like the blame squarely lies with IBM (for not correctly resetting the BMC between users of the machine), or Intel (for a poor design which allows this in the first place)

84 points by bauc 8y ago

Why does the sign say back OP door EN?

There's a good talk on securing bare metal cloud servers against such attacks, that was given at B-Sides in Portland, by two respected security people who worked on securing bare metal cloud infrastructure for one cloud provider: https://www.youtube.com/watch?v=PEVVRkd-wPM https://www.youtube.com/watch?v=PEVVRkd-wPM Boiled down it comes to: Trust absolutely nothing. When a customer finishes with a server, wipe absolutely everything, re-flash every single bit of firmware on every single device in the machine, and don't use the standard flashing mechanisms to do so. It's worth a listen/watch.

jfindley 8y ago

Is there any chance they have stumbled upon the same weakness that Bloomberg tried to report[0]? The industry at large has been pretty sceptical of Bloomberg's claims, and rightly so, but what if they just got the details wrong and it was this (or similar) vulnerability in the BMC software, rather than a dedicated spy chip, that they meant to write about? 0: https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies https://www.bloomberg.com/news/features/2018-10-04/the-big-h...

mtgx 8y ago
I think it's possible Bloomberg's "tiny chip" was a BMC, and that BMC was found with a backdoor in it, but the backdoor may not have necessarily come "from factory," but was later added by someone else. Either way, this looks terrible for SuperMicro: "Yeah, our servers don't come with built-in backdoors - it's just SuperEasy™ for attackers to add one once shipped -- please buy more now."

rbanffy 8y ago

It's a bit shocking that the people who develop the firmware for these embedded computers is not more concerned with their security. I understand you are not supposed to attach them to untrusted networks, but can we really call any network trusted these days? Do people still teach about the Maginot Line in schools?

jandeboevrie 8y ago
Cheap Chinese labor. Security costs time and time is money. So why bother if it isn't giving direct profits? Customers still buy them servers so, no problem in the revenue stream (yet)
dsfyu404ed 8y ago
>Do people still teach about the Maginot Line in schools? No. There are plenty of other opportunities to teach the value of making your adversary (be that adversary man made or natural) work every step of the way and they are missed too. Nobody (for large values of "nobody") gets taught about the value of layered or redundant systems until college and those that do get taught it in college usually only touch on it in their mandatory ethics elective. FWIW I named the last firewall I configured "Little Maginot".
- abbracadabbra 8y ago
  Another (perhaps more) common use of the term Maginot Line is as a metaphor for expensive efforts that offer a false sense of security.
dfox 8y ago
It is even more shocking considering the fact that exposing the BMC directly to the internet is not only often recommended by vendor but in many cases (eg. collocation) actually only way to use it.
- bpye 8y ago
  Jumpboxes? You need at least two for it to work though. Update A from B and then update B from A.

wyldfire 8y ago

Does this vindicate Bloomberg or is it a different BMC vulnerability?

Twirrim 8y ago
Totally different. This is a compromise via a known firmware issue with the standard components on a motherboard. The Bloomberg article was about a hidden chip being installed on a motherboard that provided a backdoor.
- wyldfire 8y ago
  Strikes me as odd that they didn't acknowledge the recent history of reporting on Supermicro BMC issues (in order to clearly distinguish them).
  - jlgaddis 8y ago
    If you read about both, it becomes pretty clear (at least, it did to me) that they aren't related. Bloomberg's story was about a (supposed) backdoored chip installed into the server at the factory, this is about flipping a bit in the firmware installed on the existing BMC.
    - datavirtue 8y ago
      A microscopic chip that merely makes it one step easier to exploit these controllers.
throwawaymath 8y ago
Nope, nothing to do with the Bloomberg story. While we're at it, Bloomberg still hasn't issued a retraction or provided more evidence to support their claims. This is despite the massive number of refutations from the entire industry.

Zenst 8y ago

This highlights how the sterilization process many cloud providers (IBM in this instance) have is not cleaning out every nook and cranny. They should, audit every bit of firmware (indeed it's odd how the researchers changed one bit in the BMC firmware and no checksum flagged it up on boot) and whilst this is daunting, it isn't that hard as they just have to compare and verify it is the same as the known safe image. Sure they could blindly reflash, but then they would miss any attempted expliotations and equally shorted the life of the hardware by increasing the odds of the flash memory failure. Whilst people see BMC's as one avenue, a server/pc has many components, all with their own firmware and in many cases, own CPU. Be that a network card, graphics card and even keyboards and mice (though the later, not so much a factor in server environments, still a consideration). Security is and always will be a mindset. You need to think like somebody who wants to break into your environment, and then counter those ways. But so many avenues. Imagine your sat at your desk as an administrator and one morning you get a nice shiny, cool top of the range keyboard sent, dressed up as a gift. How many would think, cool, plug it in and feel all fuzzy? How many would audit the firmware on that keyboard? How many would question the random gift at every level? I'm sure IBM are not the only ones who would fall foul of this avenue of BMC exploitation, but I'm disappointed that for me, basic sanity checks in their sanitisation process to decommission and recommission a server are being overlooked. Still, when you hire a car - do they audit the cars management engine firmware? Do they erase previous BT and WIFI connections stored on the radio? Well, from my experience - they don't. Remember - you can pay an expert all the money in the World, but do check their work.

jacquesm 8y ago
> Do they erase previous BT and WIFI connections stored on the radio? I've pulled many contact details from cached data on rental vehicles. Always worth checking what the stuff you pair your phone with asks for and keeps.

jlgaddis 8y ago

> This indicated that the servers' BMC firmware was not re-flashed during the server reclamation process. I'm not surprised. Unless you pay for their "enterprisey" datacenter management products (which are still relatively new), it's a PITA to perform BIOS and BMC firmware updates. Additionally, Supermicro specifically recommends that you DO NOT flash the firmware unless you are experiencing issues that a new version is suppose to fix -- unlike pretty much every other vendor (like Dell, who makes it fairly easy to do so).

greglindahl 8y ago
Flashing the Supermicro BMC is so easy and reliable that I used to do it frequently. Flashing the Supermicro BIOS, yeah, that's a disaster.

paulfurtado 8y ago

I'm a little curious why the BMC hardware is even exposed to the customer at all. Is it provided as a feature or is this just an oversight? Can't they just hide all of this hardware from the customer's OS?

pas 8y ago
It should sit on a physically separated network, with the firmware regularly updated.
- paulfurtado 8y ago
  Wasn't clear to me from the article whether the firmware was modified via the BMC's network interface or whether it was done locally over some hardware interface exposed to the customer's OS. Was this really done over the network?
  - toast0 8y ago
    Softlayer runs two separate networks, the public network, and the private network (vlans per customer, but also Softlayer tools are accessible from customer vlan, and customer vlan is accessible to Softlayer tools). The BMCs are connected to the private network. My understanding is that access to the BMC via the network is password protected, and customers don't get a password with enough access to flash via the network. Of course, older versions of the Supermicro BMC firmware had network accessible vulnerabilities, and current versions may have vulnerabilities as well. But impitool from the host can configure the BMC, and I suspect either flash the BMC or at least configure the BMC with a user that can flash it (or Supermicro tools can likely be run on the host to flash the BMC)
toast0 8y ago
Bare metal customers get access to the whole machine. We get to access the BMC for remote management (console, power management, etc). Apparently, you can configure the BMC via tools running in the host OS, without needing to authenticate to the BMC; I've used this to fix BMC provisioning errors, that would otherwise require a SL tech to fix via physical console in the bios screen.

datavirtue 8y ago

Move along, nothing to see here, folks. Everyone uses these servers and Amazon, Apple and others have already declared them safe and free of exploits for all time. Step over here for a taste of cognitive dissonance.

deleted 8y ago

[deleted]

cmurf 8y ago

Could someone speak to the write endurance of BMC flash, vs whatever flash BIOS/UEFI lives in, vs NVRAM where now certain boot parameters are stored for persistence? Is it a total non-concern? It's not a user replaceable part.

londons_explore 8y ago

The article looks written to hurt supermicro here. It looks like the blame squarely lies with IBM (for not correctly resetting the BMC between users of the machine), or Intel (for a poor design which allows this in the first place)