10 ms·

Investigating Implausible Bloomberg Supermicro Stories

112 points by vanburen 8y ago

jackconnor 8y ago
"Bloomberg’s report describes an attack that is not possible at the companies listed in the article."
eiaoa 8y ago
>> Since the implants were small, the amount of code they contained was small as well. But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off. (Source: Bloomberg with emphasis added to highlight key points for discussion) > That first part starting with “telling the device…” is nonsensical. If you are in the industry or read our Basic BMC and IPMI Management Security Practices piece, you would know that this is false. This is not a very well-written article. How does this website's "best practices" document refute Bloomberg's story? The obvious problem is that not all organizations follow best practices, including many that you'd assume would, and those that do don't always follow them consistently. More subtly, if the BMC is subverted, you can't rely on to follow its normal programming or configuration: even if you have a segregated management network with no network access, the subverted BMC isn't required to use it and can use the "shared port" instead. When you're dealing with subverted hardware or software, you have to throw out most of your assumptions about how those things work that were formed in normal cases. It's clear that the authors of this article did not do that.
- zero_intp 8y ago
  exceptional claims require exceptional evidence, Bloomberg has not done so. The article raises a fair number of solid criticisms regarding BCP to eliminate the described vulnerability.
gwern 8y ago
> Baseboard management controllers or BMCs are active on crashed or turned off servers. They allow one to, for example, power cycle servers remotely...This line from the Bloomberg is technically inaccurate because a powered off server’s storage with its sensitive code has no power and cannot be accessed. We have discussed two patently false technical details in the Bloomberg article. Er... couldn't a hypothetical backdoor BMC chip... turn the server back on? And then off again when it's done?
- mjevans 8y ago
  Since I couldn't flash my own OS in to the BMC components, where I work those ports and the 'combo' bmc+normal ports are not used on public facing networks. A BMC really should be just a fully open minicomputer. Ideally there'd be a slot for something like a Raspberry PI to be used as an add-on BMC card.
  - Teknoman117 8y ago
    I'd really like there to be more servers which support OpenBMC.
justtopost 8y ago
The story would not exist if not plausible. Even the many of the experts who dismiss the story say it is plausible. I get that it is extrordinary, and requiring extrordinary proof as a result. However, extraordinary is not at all the same as implausible. There are examples of similar attacks in the wild, no matter how many orgs argue they are not affected.
jmull 8y ago
Frankly, I'm not going to read this article or evaluate its claims. But it's interesting. Apple (mainly) and Amazon and a few others seems to have successfully turned the narrative to the question of whether or not Bloomberg got played. At that point it seems like Bloomberg has to put forward something a lot more concrete than "X anonymous sources told us so" or lose all credibility. (Or they can retract, though at this point, after multiple double-downs, it would would take a wholesale editorial clearout and significant time to return to credibility.)
- GeorgeTirebiter 8y ago
  If they could show us a compromised motherboard; and maybe even the Chinese tiny chip?
EB66 8y ago
> That first part starting with “telling the device…” is nonsensical. If you are in the industry or read our Basic BMC and IPMI Management Security Practices piece, you would know that this is false. Based on my experience in the industry, I'd say this isn't quite accurate. Yes, the author is correct in his assertion that BMCs are typically provisioned on private networks that are only accessible to outside users via a VPN. He is also correct that you cannot "tell" the BMC to do anything without access to that private network. However, the author assumes that the operator has disabled egress connectivity on the private network setup for OOB BMC access. In reality, that happens less often than you'd think. Many firewalls by default do not block egress requests and without egress filtering, the BMC can still make outbound requests to the public world. A compromised BMC could easily "phone home" and receive instructions from a command and control server.
- illumin8 8y ago
  I've worked in a dozen or so Fortune 500 companies IT departments and I've never seen any incompetent enough to connect a BMC/management network to any type of Internet connected firewall, router, or egress gateway. I've seen many that use flat management networks, which is a bad security practice, but connecting it to the Internet would require a new level of stupidity.
  - greglindahl 8y ago
    Sure, I've always segregated BMC dedicated ports to a non-connected network. But the BMC can also talk on the regular ethernet ports whenever it feels like it.
    - angry_octet 8y ago
      It can usually only vampire onto the first ethernet port. Depends on the motherboard, whether both ports are e.g. Intel, or one Intel, the other something else. Also, usually only 1000base-T not 10G.
      - yellowapple 8y ago
pdabbadabba 8y ago
This is an interesting piece, which may be correct on some points. But it gets off to a pretty rocky start, building its case around overly aggressive claims directed at uncharitable (some might say crabbed) readings of the Bloomberg piece. > That first part starting with “telling the device…” is nonsensical. A fairer statement, in light of the article's own explanation, would have been "assumes that the BMC is networked in an insecure way unusual (perhaps even unheard of) at large sophisticated tech companies, or a network compromised in such a way to bypass blocked egress routes." In other words: no. The statement is perfectly sensical, and probably even true of many networks. The author simply doubts whether this could be possible in what he regards as a properly configured (and not otherwise compromised) network. > The next inaccuracy to this paragraph is the line describing BMCs as “giving them access to the most sensitive code even on machines that have crashed or are turned off.” That is not how this technology works. But the author then goes on to explain how BMCs could indeed be used to power on machines that were previously powered off, potentially allowing access to sensitive data. (Presuming that the machine is also compromised in other ways, presumably by downloading malicious code).
galadran 8y ago
> The point is, to combine even less powerful elements of each into a package with the functionality Bloomberg describes, it would require amazingly small lithography techniques. I have yet to find someone able to say that adding processing power, signaling interfaces, networking (even a MAC), and memory/ storage onto a package this size is even possible and I have asked folks in the valley that do this at major companies. Meanwhile at the NSA, in 2008: "Not only can we do that, we'll throw in a high frequency radio as well for beating air gaps, and it will only cost you 10k! Oh, and we'll fit it INSIDE an ethernet port." https://en.wikipedia.org/wiki/NSA_ANT_catalog#/media/File:NSA_FIREWALK.jpg https://en.wikipedia.org/wiki/NSA_ANT_catalog#/media/File:NS... I'm sure the 2018 series is even smaller and even cheaper...
- tbyehl 8y ago
  That sort of thing is what stops me from dismissing Bloomberg's reporting. Intelligence agencies have a long history of creating implants that seem fantastical for their era. Consider The Thing and the Selectric Bug. An implant small enough to go unnoticed on a motherboard somehow compromising the BMC to exfiltrate data or control doesn't strike me as the least bit far-fetched. https://en.wikipedia.org/wiki/The_Thing_(listening_device) https://en.wikipedia.org/wiki/The_Thing_(listening_device) http://www.cryptomuseum.com/covert/bugs/selectric/ http://www.cryptomuseum.com/covert/bugs/selectric/
- nneonneo 8y ago
  OK, but Bloomberg said some Chinese spy chips were the size of tiny signal couplers. That "ethernet port" is actually a dual-stacked Ethernet/USB port measuring over 1 cubic inch in volume - it is hardly "small". 2008 NSA's "TRINITY" microcontroller, which forms the basis for most of their designs, is a bit smaller than a US penny, which isn't exactly "signal coupler" tiny. I can't guess at what 2018 NSA might have, nor can I guess at what 2018 PLA might have, but I think it's hard to jump immediately to believing this.
  - Sephr 8y ago
    With FEL lithography and enormous budgets most of the Bloomberg claims seem quite possible.
chx 8y ago
This whole thing reeks. The only company that could be hurt by this is Supermicro and it's a small company, it reached 2B back in 2015 and has only declined since. This matters because if this is a con fueled by an SMCI short then such a short position would need to be so big it would draw the attention of SEC. Something this big, it's not worth for anything less than hundred(s) of millions and then you are sitting on a short position of like what, ten, twenty percent of the entire market cap of SMCI? This makes no sense.
- superkuh 8y ago
  The US feds are currently trying to persuade the Canadians not to buy Chinese telecom equipment for their 5G cell networks. A story like this gives some nice fear, uncertainty, and doubt.
- myWindoonn 8y ago
  The problem with this idea is that SMCI was delisted for not filing basic paperwork with the SEC, and the delisting occurred before the Bloomberg piece was released. [0] [0] https://www.theregister.co.uk/2018/08/22/supermicro_facing_nasdaq_delisting/ https://www.theregister.co.uk/2018/08/22/supermicro_facing_n...
  - chx 8y ago
    https://www.marketwatch.com/press-release/supermicro-announces-suspension-of-trading-of-common-stock-on-nasdaq-and-its-intention-to-appeal-2018-08-23 https://www.marketwatch.com/press-release/supermicro-announc... Right. So then the SMCI short can be ruled out. What else could fuel this madness?
drtz 8y ago
I gave up when I got to the section on munufacturability. The idea that a package this size isn't big enough to house a large amount of digital logic is just wrong.
phire 8y ago
This response appears to be grasping at straws to try and prove that the described chip doesn't and couldn't exist. It reads like FUD. I find this article to be much more convincing https://www.theregister.co.uk/2018/10/04/supermicro_bloomberg/ https://www.theregister.co.uk/2018/10/04/supermicro_bloomber... It points out that 6 pins is enough to intercept and inject code into the SPI bus between the Baseboard Management Controller and it's bootflash. Once you have code running on the BMC, you can use it's capabilities to call home (via either the BMC's nic or the main nic, depending on how well the network firewalls are locked down) and then modify or simply read the main system's memory. I don't want to claim that Bloomberg is right about the attack happening, just that the attack is surprisingly plausible.
- azinman2 8y ago
  Not to mention if you pulled this off, it’s likely you have many other hacks in place too.. such as defeating firewalls and the local network config. They’re also taking bloomberg verbatim, when they likely have technical details incorrect. That doesn’t mean the gist is wrong (which may or may not be, I have no idea, I just think a lot of this article’s arguments are on weak foundation).
ansible 8y ago
So it seems to me that it is possible the original Bloomberg article may not be true, or at least has gotten details considerably wrong. In any event, I have a couple questions. 1. Is it possible that some kind of small chip could subvert there boot process of the BMC? 2. Can the BMC subvert the host system in some useful way that would be hard to detect? I'm thinking about altered microcode loaded at CPU powerup. 3. Can the altered microcode be used to subvert the host complex remotely? This could be something like a Spectre or Rowhammer attack.
- jsjohnst 8y ago
  1. Yes 2. Highly Unlikely, but can’t be entirely disproven from what I’ve seen. 3. See #2
ebeip90 8y ago
As a former spook with absolutely no knowledge about this incident, the whole thing sounds like: * The Chinese supply chain interdiction / implant installation is real * Those companies ARE aware of such events * Those companies DID report it to USGOV * The investigation is classified and Bloomberg didn't know or doesn't care * The government is doing its traditional "Deny, disavow, do not acknowledge" etc. * The companies were probably threatened with legal action / gag order preventing them from acknowledging the event When I started working for DOD, one of the things in training was that you were always to say "I can neither confirm nor deny X". Eventually they realized that this statement itself is revealing, and suggested you just STFU. The whole thing sounds incredibly plausible, and nooooooooobody was supposed to know about it.
- macintux 8y ago
  The fact that Apple and Amazon are willing to go on the record adamantly denying it is more than just a gag order. That’s a serious breach of both SEC rules and public trust if it turns out they’re lying.
  - Niksko 8y ago
    Does freedom of speech guarantee that they can't be coerced into punlic denial? Does a national security letter / other forms of government oversight override this? (It absolutely shouldn't in a democracy, but there are plenty of things that shouldn't happen in democracies that have been happening lately)
    - toomuchtodo 8y ago
      Freedom of speech doesn't supersede securities fraud.
      - Intermernet 8y ago
        NSL probably does.
mverwijs 8y ago
> Bloomberg’s report describes an attack that is not possible at the companies listed in the article. Assumptions. This article is full of them.

112 points by vanburen 8y ago

"Bloomberg’s report describes an attack that is not possible at the companies listed in the article."

>> Since the implants were small, the amount of code they contained was small as well. But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off. (Source: Bloomberg with emphasis added to highlight key points for discussion) > That first part starting with “telling the device…” is nonsensical. If you are in the industry or read our Basic BMC and IPMI Management Security Practices piece, you would know that this is false. This is not a very well-written article. How does this website's "best practices" document refute Bloomberg's story? The obvious problem is that not all organizations follow best practices, including many that you'd assume would, and those that do don't always follow them consistently. More subtly, if the BMC is subverted, you can't rely on to follow its normal programming or configuration: even if you have a segregated management network with no network access, the subverted BMC isn't required to use it and can use the "shared port" instead. When you're dealing with subverted hardware or software, you have to throw out most of your assumptions about how those things work that were formed in normal cases. It's clear that the authors of this article did not do that.

zero_intp 8y ago
exceptional claims require exceptional evidence, Bloomberg has not done so. The article raises a fair number of solid criticisms regarding BCP to eliminate the described vulnerability.

gwern 8y ago

> Baseboard management controllers or BMCs are active on crashed or turned off servers. They allow one to, for example, power cycle servers remotely...This line from the Bloomberg is technically inaccurate because a powered off server’s storage with its sensitive code has no power and cannot be accessed. We have discussed two patently false technical details in the Bloomberg article. Er... couldn't a hypothetical backdoor BMC chip... turn the server back on? And then off again when it's done?

mjevans 8y ago
Since I couldn't flash my own OS in to the BMC components, where I work those ports and the 'combo' bmc+normal ports are not used on public facing networks. A BMC really should be just a fully open minicomputer. Ideally there'd be a slot for something like a Raspberry PI to be used as an add-on BMC card.
- Teknoman117 8y ago
  I'd really like there to be more servers which support OpenBMC.

justtopost 8y ago

The story would not exist if not plausible. Even the many of the experts who dismiss the story say it is plausible. I get that it is extrordinary, and requiring extrordinary proof as a result. However, extraordinary is not at all the same as implausible. There are examples of similar attacks in the wild, no matter how many orgs argue they are not affected.

jmull 8y ago

Frankly, I'm not going to read this article or evaluate its claims. But it's interesting. Apple (mainly) and Amazon and a few others seems to have successfully turned the narrative to the question of whether or not Bloomberg got played. At that point it seems like Bloomberg has to put forward something a lot more concrete than "X anonymous sources told us so" or lose all credibility. (Or they can retract, though at this point, after multiple double-downs, it would would take a wholesale editorial clearout and significant time to return to credibility.)

GeorgeTirebiter 8y ago
If they could show us a compromised motherboard; and maybe even the Chinese tiny chip?

EB66 8y ago

> That first part starting with “telling the device…” is nonsensical. If you are in the industry or read our Basic BMC and IPMI Management Security Practices piece, you would know that this is false. Based on my experience in the industry, I'd say this isn't quite accurate. Yes, the author is correct in his assertion that BMCs are typically provisioned on private networks that are only accessible to outside users via a VPN. He is also correct that you cannot "tell" the BMC to do anything without access to that private network. However, the author assumes that the operator has disabled egress connectivity on the private network setup for OOB BMC access. In reality, that happens less often than you'd think. Many firewalls by default do not block egress requests and without egress filtering, the BMC can still make outbound requests to the public world. A compromised BMC could easily "phone home" and receive instructions from a command and control server.

illumin8 8y ago
I've worked in a dozen or so Fortune 500 companies IT departments and I've never seen any incompetent enough to connect a BMC/management network to any type of Internet connected firewall, router, or egress gateway. I've seen many that use flat management networks, which is a bad security practice, but connecting it to the Internet would require a new level of stupidity.
- greglindahl 8y ago
  Sure, I've always segregated BMC dedicated ports to a non-connected network. But the BMC can also talk on the regular ethernet ports whenever it feels like it.
  - angry_octet 8y ago
    It can usually only vampire onto the first ethernet port. Depends on the motherboard, whether both ports are e.g. Intel, or one Intel, the other something else. Also, usually only 1000base-T not 10G.
    - yellowapple 8y ago

pdabbadabba 8y ago

This is an interesting piece, which may be correct on some points. But it gets off to a pretty rocky start, building its case around overly aggressive claims directed at uncharitable (some might say crabbed) readings of the Bloomberg piece. > That first part starting with “telling the device…” is nonsensical. A fairer statement, in light of the article's own explanation, would have been "assumes that the BMC is networked in an insecure way unusual (perhaps even unheard of) at large sophisticated tech companies, or a network compromised in such a way to bypass blocked egress routes." In other words: no. The statement is perfectly sensical, and probably even true of many networks. The author simply doubts whether this could be possible in what he regards as a properly configured (and not otherwise compromised) network. > The next inaccuracy to this paragraph is the line describing BMCs as “giving them access to the most sensitive code even on machines that have crashed or are turned off.” That is not how this technology works. But the author then goes on to explain how BMCs could indeed be used to power on machines that were previously powered off, potentially allowing access to sensitive data. (Presuming that the machine is also compromised in other ways, presumably by downloading malicious code).

galadran 8y ago

> The point is, to combine even less powerful elements of each into a package with the functionality Bloomberg describes, it would require amazingly small lithography techniques. I have yet to find someone able to say that adding processing power, signaling interfaces, networking (even a MAC), and memory/ storage onto a package this size is even possible and I have asked folks in the valley that do this at major companies. Meanwhile at the NSA, in 2008: "Not only can we do that, we'll throw in a high frequency radio as well for beating air gaps, and it will only cost you 10k! Oh, and we'll fit it INSIDE an ethernet port." https://en.wikipedia.org/wiki/NSA_ANT_catalog#/media/File:NSA_FIREWALK.jpg https://en.wikipedia.org/wiki/NSA_ANT_catalog#/media/File:NS... I'm sure the 2018 series is even smaller and even cheaper...

tbyehl 8y ago
That sort of thing is what stops me from dismissing Bloomberg's reporting. Intelligence agencies have a long history of creating implants that seem fantastical for their era. Consider The Thing and the Selectric Bug. An implant small enough to go unnoticed on a motherboard somehow compromising the BMC to exfiltrate data or control doesn't strike me as the least bit far-fetched. https://en.wikipedia.org/wiki/The_Thing_(listening_device) https://en.wikipedia.org/wiki/The_Thing_(listening_device) http://www.cryptomuseum.com/covert/bugs/selectric/ http://www.cryptomuseum.com/covert/bugs/selectric/
nneonneo 8y ago
OK, but Bloomberg said some Chinese spy chips were the size of tiny signal couplers. That "ethernet port" is actually a dual-stacked Ethernet/USB port measuring over 1 cubic inch in volume - it is hardly "small". 2008 NSA's "TRINITY" microcontroller, which forms the basis for most of their designs, is a bit smaller than a US penny, which isn't exactly "signal coupler" tiny. I can't guess at what 2018 NSA might have, nor can I guess at what 2018 PLA might have, but I think it's hard to jump immediately to believing this.
- Sephr 8y ago
  With FEL lithography and enormous budgets most of the Bloomberg claims seem quite possible.

chx 8y ago

This whole thing reeks. The only company that could be hurt by this is Supermicro and it's a small company, it reached 2B back in 2015 and has only declined since. This matters because if this is a con fueled by an SMCI short then such a short position would need to be so big it would draw the attention of SEC. Something this big, it's not worth for anything less than hundred(s) of millions and then you are sitting on a short position of like what, ten, twenty percent of the entire market cap of SMCI? This makes no sense.

superkuh 8y ago
The US feds are currently trying to persuade the Canadians not to buy Chinese telecom equipment for their 5G cell networks. A story like this gives some nice fear, uncertainty, and doubt.
myWindoonn 8y ago
The problem with this idea is that SMCI was delisted for not filing basic paperwork with the SEC, and the delisting occurred before the Bloomberg piece was released. [0] [0] https://www.theregister.co.uk/2018/08/22/supermicro_facing_nasdaq_delisting/ https://www.theregister.co.uk/2018/08/22/supermicro_facing_n...
- chx 8y ago
  https://www.marketwatch.com/press-release/supermicro-announces-suspension-of-trading-of-common-stock-on-nasdaq-and-its-intention-to-appeal-2018-08-23 https://www.marketwatch.com/press-release/supermicro-announc... Right. So then the SMCI short can be ruled out. What else could fuel this madness?

drtz 8y ago

I gave up when I got to the section on munufacturability. The idea that a package this size isn't big enough to house a large amount of digital logic is just wrong.

phire 8y ago

This response appears to be grasping at straws to try and prove that the described chip doesn't and couldn't exist. It reads like FUD. I find this article to be much more convincing https://www.theregister.co.uk/2018/10/04/supermicro_bloomberg/ https://www.theregister.co.uk/2018/10/04/supermicro_bloomber... It points out that 6 pins is enough to intercept and inject code into the SPI bus between the Baseboard Management Controller and it's bootflash. Once you have code running on the BMC, you can use it's capabilities to call home (via either the BMC's nic or the main nic, depending on how well the network firewalls are locked down) and then modify or simply read the main system's memory. I don't want to claim that Bloomberg is right about the attack happening, just that the attack is surprisingly plausible.

azinman2 8y ago
Not to mention if you pulled this off, it’s likely you have many other hacks in place too.. such as defeating firewalls and the local network config. They’re also taking bloomberg verbatim, when they likely have technical details incorrect. That doesn’t mean the gist is wrong (which may or may not be, I have no idea, I just think a lot of this article’s arguments are on weak foundation).

ansible 8y ago

So it seems to me that it is possible the original Bloomberg article may not be true, or at least has gotten details considerably wrong. In any event, I have a couple questions. 1. Is it possible that some kind of small chip could subvert there boot process of the BMC? 2. Can the BMC subvert the host system in some useful way that would be hard to detect? I'm thinking about altered microcode loaded at CPU powerup. 3. Can the altered microcode be used to subvert the host complex remotely? This could be something like a Spectre or Rowhammer attack.

jsjohnst 8y ago
1. Yes 2. Highly Unlikely, but can’t be entirely disproven from what I’ve seen. 3. See #2

ebeip90 8y ago

As a former spook with absolutely no knowledge about this incident, the whole thing sounds like: * The Chinese supply chain interdiction / implant installation is real * Those companies ARE aware of such events * Those companies DID report it to USGOV * The investigation is classified and Bloomberg didn't know or doesn't care * The government is doing its traditional "Deny, disavow, do not acknowledge" etc. * The companies were probably threatened with legal action / gag order preventing them from acknowledging the event When I started working for DOD, one of the things in training was that you were always to say "I can neither confirm nor deny X". Eventually they realized that this statement itself is revealing, and suggested you just STFU. The whole thing sounds incredibly plausible, and nooooooooobody was supposed to know about it.

macintux 8y ago
The fact that Apple and Amazon are willing to go on the record adamantly denying it is more than just a gag order. That’s a serious breach of both SEC rules and public trust if it turns out they’re lying.
- Niksko 8y ago
  Does freedom of speech guarantee that they can't be coerced into punlic denial? Does a national security letter / other forms of government oversight override this? (It absolutely shouldn't in a democracy, but there are plenty of things that shouldn't happen in democracies that have been happening lately)
  - toomuchtodo 8y ago
    Freedom of speech doesn't supersede securities fraud.
    - Intermernet 8y ago
      NSL probably does.

mverwijs 8y ago

> Bloomberg’s report describes an attack that is not possible at the companies listed in the article. Assumptions. This article is full of them.