14 ms·

The Big Hack: Statements From Amazon, Apple, Supermicro, Chinese Government

466 points by okket 8y ago

bnjmn 8y ago
This sentence, wow: > Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not.
- ZeroCool2u 8y ago
  I actually laughed when I read that this morning. One of Bloomberg's more eloquent moments.
- uptown 8y ago
  One community blindly following, and the other following blindly.
- fredley 8y ago
  That sentence is not in this article. Perhaps you meant to post it here: https://news.ycombinator.com/item?id=18138328 https://news.ycombinator.com/item?id=18138328
  - bnjmn 8y ago
    Ah you’re right!
andrewstuart2 8y ago
For what it's worth, this exact article is linked directly from within the original article, and is addressed. > Read: Statements from Amazon, Apple, Supermicro and Beijing > The companies’ denials are countered by six current and former senior national security officials, who—in conversations that began during the Obama administration and continued under the Trump administration—detailed the discovery of the chips and the government’s investigation.
crunchlibrarian 8y ago
This is interesting in how vehemently all the companies are denying everything. I am pretty clueless about how the feds work so I'll ask: is it possible they would be violating secrecy laws or leaking classified info if they acknowledge this really happened? Could they already be under NDAs or whatever the equivalent is in the national security world? Or is it simply a matter of their shareholders having lofty expectations about tapping the biggest market in the world (China) and saying anything that angers China is the worst thing you could possibly do from a PR perspective?
- Eridrus 8y ago
  I don't think there needs to be a conspiracy for vehement denials to make sense. Security is a hugely important reputational good for both Apple & AWS.
  - ISL 8y ago
    That reputation would be sullied substantially if the BusinessWeek article is correct, and the companies' first public statements state otherwise.
- SyneRyder 8y ago
  The article actually says that some of the information is classified: "The sources were granted anonymity because of the sensitive, and in some cases classified, nature of the information."
  - ectospheno 8y ago
    That people who made promises not to divulge information continue doing so on a regular basis is the disturbing part for me. If you make an agreement then stick with it. If you no longer can then quit.
    - MaupitiBlue 8y ago
      Yes and no. Often, and I think in this situation, the government wants the story out, but not officially, so certain people are told to leak.
neximo64 8y ago
Well, this explains why all those chipmaker acquisitions failed/were rejected on National Security Grounds. It would be whole lot harder to find these modifications if this was on the silicon itself.
- wyldfire 8y ago
  The AVGO/QCOM LBO didn't happen ostensibly because of the role 5G plays in national security, not trust issues with the design. However, I'll grant that there could be other covert reasons (nationalism, these very trust issues, etc).
- phkahler 8y ago
  >> It would be whole lot harder to find these modifications if this was on the silicon itself. Intel ME is well know to be on the chip itself. When you're really good you hide exploits in plain sight.
uptown 8y ago
"Apple has never found malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server. Apple never had any contact with the FBI or any other agency about such an incident. We are not aware of any investigation by the FBI, nor are our contacts in law enforcement." Bloomberg's article and Apple's statement can't both be right.
- baq 8y ago
  expected given that we're talking about is something that could be the plot of a jason bourne movie.
  - cm2187 8y ago
    Or if Apple is bound by some kind of non disclosure order.
    - uptown 8y ago
      Even if they were bound by a non-disclosure order, this response goes beyond what's necessary to refute the story and conveys material information which would be used against Apple by shareholders if it is later found to be factually inaccurate.
  - kazagistar 8y ago
    Oh, the tech described is very real, and has been a known nation state level attack for years. It's the scale and targeting of the attack that is new.
- MrBingley 8y ago
  "The lady doth protest too much, methinks."
  - draw_down 8y ago
    That’s not how this works.
- 8y ago
tsuru 8y ago
Has this story from 2014 ever been discredited [1] ? If not, I don't see how any supply chain is safe from G8 powers [1] https://arstechnica.com/tech-policy/2014/05/photos-of-an-nsa-upgrade-factory-show-cisco-router-getting-implant/ https://arstechnica.com/tech-policy/2014/05/photos-of-an-nsa...
Illniyar 8y ago
There's denial and there's vehement to the point complete denial from multiple different companies. It's either a giant conspiracy by the FBI and multiple mega-corporations to blatantly lie, on public record, about a matter that if happened would most likely come up in the future again. Furthermore if apple and amazon were both notified for comments, there is good reason to suspect that the FBI would hear of the article and try to censor such an article for national security reasons, especially so if they made apple and amazon lie about it. Or ... Bloomberg didn't do their due diligence and were too eager to be duped by agents who wanted to push an agenda to move manufacturing away from china or something similar.
- toyg 8y ago
  They throw out some massive numbers though. You can kinda discount 6 "apparatus" sources (but usually for stuff like this it's just one or two, so it's already unusual), but they also mention several people at Apple, Amazon and the FBI, plus someone else in the "discouragement" meeting. That's a lot of agents. If it's made up, it's more realistic to believe the publication is complicit in the fabrication.
  - Illniyar 8y ago
    That's a good point. It makes things murky, which I suspect the massive amount of collaboration is the only reason Bloomberg published the article. Could it be a conspiracy to deceive Bloomberg? From all the conspiracies, that seems the most likely to me.
- pbalau 8y ago
  The original article contains exactly 0 real technical information and reads like a spy story beginning.
  - delroth 8y ago
    It does, and all the technical details mostly match up. From the plausible attack vector (targetting the BMC with an implant) to CG animations of a compromised motherboard showing the implant at the exact spot between the BMC and its firmware chip (which makes perfect sense if it's some kind of SPI interposer).
hendzen 8y ago
Apple and Amazon probably have a small set of TS/SCI cleared employees who dealt with this mess. It’s likely 99.99% of the employees at those firms had no idea what was going on. The switching out of thousands of compromised servers was probably made to look like routine maintenance or upgrades and the whole affair was kept secret. That is, until some high level government employees intentionally leaked it to the media, probably under direction of the White House to garner support for a more aggressive stance on China - the trade war in particular. Read between the lines.
- onetimemanytime 8y ago
  Plus, they gain nothing by admitting that China hacked them. Your data is there and all.
- simias 8y ago
  That's a lot of unsubstantiated assumptions. It's also explicitly denied by both Amazon: >It’s untrue that AWS knew about a supply chain compromise and Apple: >we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them. There's no in "between the lines" or ambiguous wording there, they flat out deny it. Unless this small set of "TS/SCI cleared employees" worked completely on their own without reporting to anybody else in the company this means that they are lying in these statements. It's possible that they do just that but it's a bit strange to me that they wouldn't find an easier way to deflect the issue without using such a strong and explicit language. Something vague like "we've been working with the authorities and have no reason to believe that any sensitive information has been leaked etc..." would be easier to spin if it turns out that somebody can prove that these attacks took place. Surely if the scale of the attack was as large as reported by Bloomberg in their article it should be possible to find one of these backdoored boards in the wild? Or at least have testimonies by employees in these company that could testify that batches of motherboards were suddenly replaced for no obvious reasons? And if trade war is the reason why deny it now? What do they have to gain from that, they're the victims in this story as far as I can tell.
  - hendzen 8y ago
Ditiris 8y ago
The oldest tactic in the book: DENY DENY DENY https://youtu.be/yN2gU0XU5FU https://youtu.be/yN2gU0XU5FU
ksec 8y ago
>China is a resolute defender of cybersecurity It is missing the word "offender" somewhere. I think the question simply comes down to this; Can the Chinese Government be trusted?
- superflyguy 8y ago
  I'm not sure how you managed to keep a straight face while typing that.
- ElBarto 8y ago
  > Can the Chinese Government be trusted? The US Government can trust the Chinese Government as much as the Chinese Government can trust the US Government. ;) The spy game has been played for 4,000 years...
- sct202 8y ago
  They probably mean defender of cybersecurity for China.
- est 8y ago
  Can the Chinese government trust itself? I mean Chinese govn't literally buys intel from Chinese hackers. It's encouraged business until the Obama deal.
upofadown 8y ago
Why wouldn't the current US government be shouting this from the rooftops if it was actually true?
- Palptine 8y ago
  Because at least half of the people wouldn't believe them and consider this another conspiracy theory. Even when Bloomberg pushed this there are still so many deniers in this thread.
  - pixl97 8y ago
    Hell, how many conspiracy theory's have come out as true in since 2001. Remember the "The telephone companies are routing all our calls and internet data to the NSA" conspiracy. A bunch of people said that was fake, and there was no way this could happen. Then more evidence came out, and the same people said there was no way that could happen, its too big of conspiracy and it would have leaked way before then. Then the government gave the telco's retroactive immunity for spying on the public.
  - deleted 8y ago
    [deleted]
- eksemplar 8y ago
  I’m not sure if you’ve noticed, but IT isn’t really taken very serious outside of IT. I mean, they want money from it, but you don’t get money by telling the world that your primary tech companies are compromised. Though judging by the replies, I think Bloomberg needs some really solid sources, so there is probably nothing to it.
- IncRnd 8y ago
  The US government argued for years that chips needed to be made inside the US - which they were for certain military applications. Then, President Clinton allowed huge technology transfers from the US to China, and the US was no longer making its own chips for military applications.
deleted 8y ago
[deleted]
bovermyer 8y ago
You know, I would watch a movie with a plot like this.
- RRRA 8y ago
  two hours of chip decaping and circuit reversing and firmware dumping! :P
  - kalleboo 8y ago
    I would watch a YouTube channel of that
nvahalik 8y ago
Regardless about how you feel about the hack, outsourcing the vast majority of our technology to another country just doesn't seem like the smartest idea. Why would we put our most trusted technology into someone else's hands—just because it's going to save a few bucks? Wouldn't it be worth it to just do these things ourselves?
- rajataghi 8y ago
  Its not just 'a few bucks', it is a considerable amount of money in the long term.
  - bashallah 8y ago
    You sure that’s adequate cost evaluation if the bloomberg accusations are true?
    - sct202 8y ago
      I'm sure that companies will start to re-shore production on to friendlier countries or at least require sub-contracting restrictions. From the article it sounds like a sub-sub-sub-contractor (SuperMicro->Main Chinese Contractor->Compromised Chinese Contractor) was the weak link.
- RobertoG 8y ago
  You should talk about "our technology" only if you are a big stock owner or, maybe, a naive employee. They are not called "global corporations" because of their patriotism. It's not called "global capital" because it cares where to reproduce. This is the system working as intended.
- est 8y ago
  The boat has been sailed long time ago. First it was the Japanese, then the Taiwanese and Koreans. In the past decade it was the Chinese. Who's next, Vietnamese? Did you see the pattern here?
  - 8y ago
dannyw 8y ago
I think it's now time for Bloomberg to respond. This almost feels like a repeat "Newsweek Reveals Satoshi".
- supertiger 8y ago
  This response piece was published one hour after the original report. Bloomberg obviously had both ready. It would have responded already if intend to.
setquk 8y ago
Nothing adds up here. Supermicro boards have "designed in the USA" proudly stamped all over them. This means that either: (a) the design process was infiltrated, which would have been done US side thus the nationality of the actors is debatable. (b) the manufacturing process was infiltrated, which SHOULD have been picked up during design validation and production sampling. (c) this whole thing is a load of rubbish. Lots of questions here. This is not a tinfoil hat measure as well; genuine questions from someone who HAS worked in the EE side of things. I wonder if this is a bunch of pre-emptive finger pointing and ass covering for an implant closer to home? I don't trust either side of the fence if I'm honest.
- ceejayoz 8y ago
  The article alleges (b), and give indications of why it wouldn't necessarily be picked up: > In one case, the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached, according to one person who saw pictures of the chips. That generation of chips was smaller than a sharpened pencil tip, the person says. (Amazon denies that AWS knew of servers found in China containing malicious chips.) There's plenty of precedent for missing things hidden away in stuff: https://en.wikipedia.org/wiki/The_Thing_(listening_device) https://en.wikipedia.org/wiki/The_Thing_(listening_device)
  - setquk 8y ago
    That is possible but unlikely. Firstly you'd have to do this at the PCB fab phase as this can't be retrospectively applied, so you'd have to have infiltrated two times as many companies and add complex manufacturing exceptions to their workflow. Secondly you'd have to get it through AXI process which is used on these larger boards (Automated X-ray Inspection) which is used by the larger companies for manufacturing AND design validation. So you'd have to infiltrate another company. Also stuff like this tends to show up on boundary scans. It's not that easy to cock around with signal integrity on these sorts of boards and get away with it. I find the whole thing infeasible from a cost and logic perspective. The SMbus firmware and Aspeed sub-vendor are so much easier to hit and don't leave any corpses around to find after the fact.
DeusExMachina 8y ago
Out of curiosity: why do they use the word "untrue" instead of "false"? Are there some legal nuances I am not aware of?
- brohee 8y ago
  PR rep deliberately using newspeak to convey he's ordered to lie?
nickelcitymario 8y ago
Best line from an otherwise serious and and important piece of reporting: "Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not." ...which did not.
leonroy 8y ago
One thing I don't see much about in the article is the supposed chip which allegedly compromised Super Micro servers. The cover image of the piece shows a surface mount component with three solder pads balanced on a finger tip. Looks like a very simple SMD part to me. On top of that Apple, Amazon and Super Micro are flat out denying this - I suspect Bloomberg messed up here.
tSheoghi2 8y ago
These denials remind me of the vehement denials the big tech companies gave when the Snowden leaks came out. Did they turn out to be false? https://googleblog.blogspot.com/2013/06/what.html https://googleblog.blogspot.com/2013/06/what.html
- mercutio2 8y ago
  No. The denials did not turn out to be false. Conspiracy theorists aside, the main new thing that came out of the Snowden revelations was that Google’s physical security for data-center-to-data-center traffic was compromised by the NSA, which Google never denied, and responded by hardening server to server traffic.
- clubm8 8y ago
  >These denials remind me of the vehement denials the big tech companies gave when the Snowden leaks came out. Did they turn out to be false? At what point do such denials constitute a deceptive trade practive, enabling the Federal Trade Commission to bring action? You can't lie in a privacy policy, or a TV commercial. Where is the line?
  - wstuartcl 8y ago
    ... It almost makes you wonder if there is a process for ensuring companies comply with secret investigations and are forced to act publicly and privately as if they have never happened. https://en.wikipedia.org/wiki/United_States_Foreign_Intelligence_Surveillance_Court https://en.wikipedia.org/wiki/United_States_Foreign_Intellig...
deleted 8y ago
[deleted]
jnbiche 8y ago
A lot of people are unaware of how anonymous sources in a serious news organization work. Here, it means that the multiple high-level intelligence officials described in the article are known to and vetted by Bloomberg. They've looked at their resumes and bona fides, and confirmed their backgrounds. They're just not revealing their names to us. So which is more likely: that multiple intelligence officials are making this up, or that Apple/Amazon/Supermicro feel obligated to lie because this is an ongoing classified counterintel investigation?
- cyphunk 8y ago
  > So which is more likely The language of the Apple refute is so strong, to the point of directly attacking Bloomberg and calling them irresponsible. So yes, in this case there are at least a few lies being peddled by the Bloomberg intelligence contacts
- kazagistar 8y ago
  Honestly, both seem somewhat plausible. I'll let it shake itself out.
- lolc 8y ago
  Plenty of claims by "senior intelligence officials" have proven to be factually incorrect. Same goes for press statements. It's simply too early to tell who's telling the truth, who's mistaken, and who's lying here.
- deleted 8y ago
  [deleted]
cyphunk 8y ago
I have not seen anything that indicates how installing this chip would do anything at all without also modifying the trace design and fabrication of the PCB itself. Also does anyone have information about the "baseboard management controller" mentioned? I would like to understand the complexity required to MiTM a ROM or FLASH memory read by such a controller before concluding the feasibility and number of players in manufacturing chain required for it to work.
- adrian_b 8y ago
  The BMC is an ARM microcontroller that has complete access to everything, exactly like the Intel Management Engine, but the server vendors prefer a separate chip for the same job. What is described in the article is very easy to do by inserting a microcontroller on the SPI link that connects the BMC with the flash memory containing the BMC programs, which are copied from there to a RAM at boot. However, such a microcontroller would need 8 pins for at least 7 signals: ground, power, SPI clock and 4 SPI data, to and from BMC and flash memory. Nonetheless, 8-pin packages can be very small, e.g. 0.8 mm by 1.35 mm, i.e. only slightly larger than 1 square millimeter. So from a technical point of view, all is easily feasible. The problem is that it would require compromised people in several places at the subcontractors, because the design files for the PCB must be replaced wherever the PCB is made and in another place, at the PCB assembly, the pick & place document must be replaced and an extra reel with the backdoor component must be mounted on the equipment and that reel must come from somewhere else than from the normal suppliers of the assembly line without raising suspicions. It can be done, but many accomplices are required. Because most of the time the backdoor component will pass the SPI data signals transparently, it will not be detected at any electrical testing and the usual optical inspections are unlikely to detect such a small change. I am using many Supermicro motherboards, so I am wondering if this story is true. If it were true, it would not be much of a surprise, because they did not do something really novel but they just matched what USA also did, e.g. in the Cisco case.
  - cyphunk 8y ago
    But also if a modification of the SPI bus can result in exploitation then the manufacturer will just patch the memory in place. Cheaper, easier, less detectable. So... the manufacturer is not the source of attack.
ggm 8y ago
I've always wondered if that passive device in the great crest at the US embassy in Moscow had equivalents which got hooked up to consumer devices with high voltage parts (to make people reluctant to play inside) Remember the furore when Zenith was the last domestic manufacturer of TVs in the USA? We've come a long way since then..
- astrange 8y ago
  It's not that dangerous to open electronics ever since they stopped using electron guns. But, consumer devices don't need to secretly spy on you when consumers buy them literally for the purpose of being spied on.
21 8y ago
They allege 30 companies were compromised with implanted Supermicro servers. Surely one of them could show the world/researchers an infected motherboard?
- simplecomplex 8y ago
  This. Where did all the infected motherboards go??
kevlar1818 8y ago
To me, this is perhaps the most worrying part of the story. Did Bloomberg, a widely renowned and distributed news outlet with immense resources, sacrifice hard evidence for sensationalism and clicks? Or are these companies, all widely renowned with immense resources, bound to silence due to any multitude of shady reasons? No matter the facts behind the story and these denials, this whole thing reeks of FUD.
PascLeRasc 8y ago
Could anyone find a list of the server models that had this compromise? I haven't seen much if any technical info in either of the Bloomberg articles.
- baybal2 8y ago
  microblade 6128
  - justtopost 8y ago
    Source?
    - baybal2 8y ago
      Just look at the photo, and the motherboard model. It is only used in 1 server. And given Elemental is mentioned, it must be a blade So that matches nicely.
oropolo 8y ago
Are there facilities anywhere in the US where motherboards can be manufactured?
bogomipz 8y ago
The article states: >"Somewhere in the Linux operating system, which runs in many servers, is code that authorizes a user by verifying a typed password against a stored encrypted one. An implanted chip can alter part of that code so the server won’t check for a password—and presto! A secure machine is open to any and all users." I realize the intended audience for this article is not a technical crowd but can someone walk me through in practical terms how such a chip might subvert the /bin/login binary?
- gluegadget 8y ago
  Quite possibly through IPMI as PAM.
  - bogomipz 8y ago
    Can you elaborate?
    - sounds 8y ago
      Let me elaborate your question. "can someone walk me through in practical terms how such a chip might subvert the /bin/login binary?" Unpacking that: 1. Please, random internet stranger, can you... 2. Make a nice bloggy article with pretty pictures... 3. That includes practical handy advice... 4. How, once I've compromised the system at the BMC... 5. To write an exploit for the linux kernel without getting caught. Preferrably one that silently patches /bin/login. Yeah, if you know how to do that, you're not posting it on the internet. You're on anonymous message boards asking for jobs from one side or the other.
      - bogomipz 8y ago
        Yeah that wasn't my question at all. Maybe you aren't a native English speaker but "walk me through" is an idiom in the English language it is not a literal phrase. The OP's comment was quite terse and so I asked if they wouldn't mind elaborating. As you can see someone else replied with a more concrete example, which is all I was asking. Your response displays a shocking level of immaturity that appears to have been triggered by your own total and complete misunderstanding.
sounds 8y ago
"In the three years since the briefing in McLean, no commercially viable way to detect attacks like the one on Supermicro’s motherboards has emerged—or has looked likely to emerge." I know a commercially viable way to detect hardware attacks. Standardized hardware designs, such as the "x86 standard," ARM IP licenses, and more recently, RISC-V, decentralizes manufacturing and drives the cost to commodity levels. I'm specifically proposing that the U.S. Government appropriate the patents on whichever hardware design and declare them a National Security asset, and then guarantee royalty-free licenses to any company that wants to use them. When it's no longer a big profit center, China is no longer as interested in owning a monopoly on it. And presto: there's no longer a monopoly on the hardware. Thus it's no longer a guarantee that your hardware is being bent to the will of a single nation-state. Hardware attacks can be detected as variations between the hardware made by one nation vs. the hardware made by another nation. The downside is that Apple can't have the same profit margins that come from closed, proprietary hardware. The upside is that manufacturing and process innovation (such as Intel used to do) becomes extremely desirable. It becomes so valuable that we saw Intel reluctant to offshore their best processes. There: economic solution and political points, to boot!
- 21 8y ago
  Have you read the article? Apparently one of the big factors in Supermicro success is that it has over 900 different motherboard designs, and hundreds of hardware specialists which can customize them further to client wishes.
  - sounds 8y ago
    Your post is a non-sequitur. I'm not disagreeing that Supermicro makes customized designs.
    - 21 8y ago
      It is a sequitur in the sense that a few open-hardware motherboard designs will not be commercially successful.
fmajid 8y ago
The cost of manufacture is a very small part of the cost of assembling computers. For the iPhone, for instance, it represents less than $10 out of the $240 or so total cost. Thus shifting production to other another locale with higher costs would not significantly increase the price of the product, and in any case labor costs are going up in China as well. Thus you wonder why more production isn't being shifted from China to, say, Thailand, Indonesia or India. Steve Jobs once said the industrial capacity to do the work simply isn't available outside of China, in terms of skilled people and supply chains, and that may be a big reason why.
DenisM 8y ago
It's interesting that the numerous sources all decided to confine in Bloomberg. How did they all know to go to the same paper? And if some of them went to other papers then we should hear about it tomorrow, I guess.
nasseri 8y ago
Can someone point me to the exact laws or cases with precedent that would make Apple and Amazon's statements illegal if the Bloomberg article were true? I'm not doubting it, but I can't find much about this online. Pump and dump schemes, ponzi schemes, and insider trading are illegal, and those aspects of securities fraud are well documented online, but this more general "lying to the public" that these companies may or may not be doing has proved trickier to find precedence for. I am not a lawyer, and my only resource here is google, but I think there is an assumption at play that this form of lying is illegal, and I'm honestly not sure that it is. I could be wayy of base here, but if its not illegal, than it would be pretty obvious what is going on.
maerF0x0 8y ago
This has been posted several times and there are tons of comments: [1]: https://news.ycombinator.com/item?id=18146438 https://news.ycombinator.com/item?id=18146438 [2]: https://news.ycombinator.com/item?id=18138328 https://news.ycombinator.com/item?id=18138328 [3]: https://news.ycombinator.com/item?id=18145645 https://news.ycombinator.com/item?id=18145645 [4]: https://news.ycombinator.com/item?id=18138990 https://news.ycombinator.com/item?id=18138990 [5]: https://news.ycombinator.com/item?id=18141328 https://news.ycombinator.com/item?id=18141328

466 points by okket 8y ago

This sentence, wow: > Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not.

ZeroCool2u 8y ago
I actually laughed when I read that this morning. One of Bloomberg's more eloquent moments.
uptown 8y ago
One community blindly following, and the other following blindly.
fredley 8y ago
That sentence is not in this article. Perhaps you meant to post it here: https://news.ycombinator.com/item?id=18138328 https://news.ycombinator.com/item?id=18138328
- bnjmn 8y ago
  Ah you’re right!

andrewstuart2 8y ago

For what it's worth, this exact article is linked directly from within the original article, and is addressed. > Read: Statements from Amazon, Apple, Supermicro and Beijing > The companies’ denials are countered by six current and former senior national security officials, who—in conversations that began during the Obama administration and continued under the Trump administration—detailed the discovery of the chips and the government’s investigation.

crunchlibrarian 8y ago

This is interesting in how vehemently all the companies are denying everything. I am pretty clueless about how the feds work so I'll ask: is it possible they would be violating secrecy laws or leaking classified info if they acknowledge this really happened? Could they already be under NDAs or whatever the equivalent is in the national security world? Or is it simply a matter of their shareholders having lofty expectations about tapping the biggest market in the world (China) and saying anything that angers China is the worst thing you could possibly do from a PR perspective?

Eridrus 8y ago
I don't think there needs to be a conspiracy for vehement denials to make sense. Security is a hugely important reputational good for both Apple & AWS.
- ISL 8y ago
  That reputation would be sullied substantially if the BusinessWeek article is correct, and the companies' first public statements state otherwise.
SyneRyder 8y ago
The article actually says that some of the information is classified: "The sources were granted anonymity because of the sensitive, and in some cases classified, nature of the information."
- ectospheno 8y ago
  That people who made promises not to divulge information continue doing so on a regular basis is the disturbing part for me. If you make an agreement then stick with it. If you no longer can then quit.
  - MaupitiBlue 8y ago
    Yes and no. Often, and I think in this situation, the government wants the story out, but not officially, so certain people are told to leak.

neximo64 8y ago

Well, this explains why all those chipmaker acquisitions failed/were rejected on National Security Grounds. It would be whole lot harder to find these modifications if this was on the silicon itself.

wyldfire 8y ago
The AVGO/QCOM LBO didn't happen ostensibly because of the role 5G plays in national security, not trust issues with the design. However, I'll grant that there could be other covert reasons (nationalism, these very trust issues, etc).
phkahler 8y ago
>> It would be whole lot harder to find these modifications if this was on the silicon itself. Intel ME is well know to be on the chip itself. When you're really good you hide exploits in plain sight.

uptown 8y ago

"Apple has never found malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server. Apple never had any contact with the FBI or any other agency about such an incident. We are not aware of any investigation by the FBI, nor are our contacts in law enforcement." Bloomberg's article and Apple's statement can't both be right.

baq 8y ago
expected given that we're talking about is something that could be the plot of a jason bourne movie.
- cm2187 8y ago
  Or if Apple is bound by some kind of non disclosure order.
  - uptown 8y ago
    Even if they were bound by a non-disclosure order, this response goes beyond what's necessary to refute the story and conveys material information which would be used against Apple by shareholders if it is later found to be factually inaccurate.
- kazagistar 8y ago
  Oh, the tech described is very real, and has been a known nation state level attack for years. It's the scale and targeting of the attack that is new.
MrBingley 8y ago
"The lady doth protest too much, methinks."
- draw_down 8y ago
  That’s not how this works.
8y ago

tsuru 8y ago

Has this story from 2014 ever been discredited [1] ? If not, I don't see how any supply chain is safe from G8 powers [1] https://arstechnica.com/tech-policy/2014/05/photos-of-an-nsa-upgrade-factory-show-cisco-router-getting-implant/ https://arstechnica.com/tech-policy/2014/05/photos-of-an-nsa...

Illniyar 8y ago

There's denial and there's vehement to the point complete denial from multiple different companies. It's either a giant conspiracy by the FBI and multiple mega-corporations to blatantly lie, on public record, about a matter that if happened would most likely come up in the future again. Furthermore if apple and amazon were both notified for comments, there is good reason to suspect that the FBI would hear of the article and try to censor such an article for national security reasons, especially so if they made apple and amazon lie about it. Or ... Bloomberg didn't do their due diligence and were too eager to be duped by agents who wanted to push an agenda to move manufacturing away from china or something similar.

toyg 8y ago
They throw out some massive numbers though. You can kinda discount 6 "apparatus" sources (but usually for stuff like this it's just one or two, so it's already unusual), but they also mention several people at Apple, Amazon and the FBI, plus someone else in the "discouragement" meeting. That's a lot of agents. If it's made up, it's more realistic to believe the publication is complicit in the fabrication.
- Illniyar 8y ago
  That's a good point. It makes things murky, which I suspect the massive amount of collaboration is the only reason Bloomberg published the article. Could it be a conspiracy to deceive Bloomberg? From all the conspiracies, that seems the most likely to me.
pbalau 8y ago
The original article contains exactly 0 real technical information and reads like a spy story beginning.
- delroth 8y ago
  It does, and all the technical details mostly match up. From the plausible attack vector (targetting the BMC with an implant) to CG animations of a compromised motherboard showing the implant at the exact spot between the BMC and its firmware chip (which makes perfect sense if it's some kind of SPI interposer).

hendzen 8y ago

Apple and Amazon probably have a small set of TS/SCI cleared employees who dealt with this mess. It’s likely 99.99% of the employees at those firms had no idea what was going on. The switching out of thousands of compromised servers was probably made to look like routine maintenance or upgrades and the whole affair was kept secret. That is, until some high level government employees intentionally leaked it to the media, probably under direction of the White House to garner support for a more aggressive stance on China - the trade war in particular. Read between the lines.

onetimemanytime 8y ago
Plus, they gain nothing by admitting that China hacked them. Your data is there and all.
simias 8y ago
That's a lot of unsubstantiated assumptions. It's also explicitly denied by both Amazon: >It’s untrue that AWS knew about a supply chain compromise and Apple: >we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them. There's no in "between the lines" or ambiguous wording there, they flat out deny it. Unless this small set of "TS/SCI cleared employees" worked completely on their own without reporting to anybody else in the company this means that they are lying in these statements. It's possible that they do just that but it's a bit strange to me that they wouldn't find an easier way to deflect the issue without using such a strong and explicit language. Something vague like "we've been working with the authorities and have no reason to believe that any sensitive information has been leaked etc..." would be easier to spin if it turns out that somebody can prove that these attacks took place. Surely if the scale of the attack was as large as reported by Bloomberg in their article it should be possible to find one of these backdoored boards in the wild? Or at least have testimonies by employees in these company that could testify that batches of motherboards were suddenly replaced for no obvious reasons? And if trade war is the reason why deny it now? What do they have to gain from that, they're the victims in this story as far as I can tell.
- hendzen 8y ago

Ditiris 8y ago

The oldest tactic in the book: DENY DENY DENY https://youtu.be/yN2gU0XU5FU https://youtu.be/yN2gU0XU5FU

ksec 8y ago

>China is a resolute defender of cybersecurity It is missing the word "offender" somewhere. I think the question simply comes down to this; Can the Chinese Government be trusted?

superflyguy 8y ago
I'm not sure how you managed to keep a straight face while typing that.
ElBarto 8y ago
> Can the Chinese Government be trusted? The US Government can trust the Chinese Government as much as the Chinese Government can trust the US Government. ;) The spy game has been played for 4,000 years...
sct202 8y ago
They probably mean defender of cybersecurity for China.
est 8y ago
Can the Chinese government trust itself? I mean Chinese govn't literally buys intel from Chinese hackers. It's encouraged business until the Obama deal.

upofadown 8y ago

Why wouldn't the current US government be shouting this from the rooftops if it was actually true?

Palptine 8y ago
Because at least half of the people wouldn't believe them and consider this another conspiracy theory. Even when Bloomberg pushed this there are still so many deniers in this thread.
- pixl97 8y ago
  Hell, how many conspiracy theory's have come out as true in since 2001. Remember the "The telephone companies are routing all our calls and internet data to the NSA" conspiracy. A bunch of people said that was fake, and there was no way this could happen. Then more evidence came out, and the same people said there was no way that could happen, its too big of conspiracy and it would have leaked way before then. Then the government gave the telco's retroactive immunity for spying on the public.
- deleted 8y ago
  [deleted]
eksemplar 8y ago
I’m not sure if you’ve noticed, but IT isn’t really taken very serious outside of IT. I mean, they want money from it, but you don’t get money by telling the world that your primary tech companies are compromised. Though judging by the replies, I think Bloomberg needs some really solid sources, so there is probably nothing to it.
IncRnd 8y ago
The US government argued for years that chips needed to be made inside the US - which they were for certain military applications. Then, President Clinton allowed huge technology transfers from the US to China, and the US was no longer making its own chips for military applications.

deleted 8y ago

[deleted]

bovermyer 8y ago

You know, I would watch a movie with a plot like this.

RRRA 8y ago
two hours of chip decaping and circuit reversing and firmware dumping! :P
- kalleboo 8y ago
  I would watch a YouTube channel of that

nvahalik 8y ago

Regardless about how you feel about the hack, outsourcing the vast majority of our technology to another country just doesn't seem like the smartest idea. Why would we put our most trusted technology into someone else's hands—just because it's going to save a few bucks? Wouldn't it be worth it to just do these things ourselves?

rajataghi 8y ago
Its not just 'a few bucks', it is a considerable amount of money in the long term.
- bashallah 8y ago
  You sure that’s adequate cost evaluation if the bloomberg accusations are true?
  - sct202 8y ago
    I'm sure that companies will start to re-shore production on to friendlier countries or at least require sub-contracting restrictions. From the article it sounds like a sub-sub-sub-contractor (SuperMicro->Main Chinese Contractor->Compromised Chinese Contractor) was the weak link.
RobertoG 8y ago
You should talk about "our technology" only if you are a big stock owner or, maybe, a naive employee. They are not called "global corporations" because of their patriotism. It's not called "global capital" because it cares where to reproduce. This is the system working as intended.
est 8y ago
The boat has been sailed long time ago. First it was the Japanese, then the Taiwanese and Koreans. In the past decade it was the Chinese. Who's next, Vietnamese? Did you see the pattern here?
- 8y ago

dannyw 8y ago

I think it's now time for Bloomberg to respond. This almost feels like a repeat "Newsweek Reveals Satoshi".

supertiger 8y ago
This response piece was published one hour after the original report. Bloomberg obviously had both ready. It would have responded already if intend to.

setquk 8y ago

Nothing adds up here. Supermicro boards have "designed in the USA" proudly stamped all over them. This means that either: (a) the design process was infiltrated, which would have been done US side thus the nationality of the actors is debatable. (b) the manufacturing process was infiltrated, which SHOULD have been picked up during design validation and production sampling. (c) this whole thing is a load of rubbish. Lots of questions here. This is not a tinfoil hat measure as well; genuine questions from someone who HAS worked in the EE side of things. I wonder if this is a bunch of pre-emptive finger pointing and ass covering for an implant closer to home? I don't trust either side of the fence if I'm honest.

ceejayoz 8y ago
The article alleges (b), and give indications of why it wouldn't necessarily be picked up: > In one case, the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached, according to one person who saw pictures of the chips. That generation of chips was smaller than a sharpened pencil tip, the person says. (Amazon denies that AWS knew of servers found in China containing malicious chips.) There's plenty of precedent for missing things hidden away in stuff: https://en.wikipedia.org/wiki/The_Thing_(listening_device) https://en.wikipedia.org/wiki/The_Thing_(listening_device)
- setquk 8y ago
  That is possible but unlikely. Firstly you'd have to do this at the PCB fab phase as this can't be retrospectively applied, so you'd have to have infiltrated two times as many companies and add complex manufacturing exceptions to their workflow. Secondly you'd have to get it through AXI process which is used on these larger boards (Automated X-ray Inspection) which is used by the larger companies for manufacturing AND design validation. So you'd have to infiltrate another company. Also stuff like this tends to show up on boundary scans. It's not that easy to cock around with signal integrity on these sorts of boards and get away with it. I find the whole thing infeasible from a cost and logic perspective. The SMbus firmware and Aspeed sub-vendor are so much easier to hit and don't leave any corpses around to find after the fact.

DeusExMachina 8y ago

Out of curiosity: why do they use the word "untrue" instead of "false"? Are there some legal nuances I am not aware of?

brohee 8y ago
PR rep deliberately using newspeak to convey he's ordered to lie?

nickelcitymario 8y ago

Best line from an otherwise serious and and important piece of reporting: "Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not." ...which did not.

leonroy 8y ago

One thing I don't see much about in the article is the supposed chip which allegedly compromised Super Micro servers. The cover image of the piece shows a surface mount component with three solder pads balanced on a finger tip. Looks like a very simple SMD part to me. On top of that Apple, Amazon and Super Micro are flat out denying this - I suspect Bloomberg messed up here.

tSheoghi2 8y ago

These denials remind me of the vehement denials the big tech companies gave when the Snowden leaks came out. Did they turn out to be false? https://googleblog.blogspot.com/2013/06/what.html https://googleblog.blogspot.com/2013/06/what.html

mercutio2 8y ago
No. The denials did not turn out to be false. Conspiracy theorists aside, the main new thing that came out of the Snowden revelations was that Google’s physical security for data-center-to-data-center traffic was compromised by the NSA, which Google never denied, and responded by hardening server to server traffic.
clubm8 8y ago
>These denials remind me of the vehement denials the big tech companies gave when the Snowden leaks came out. Did they turn out to be false? At what point do such denials constitute a deceptive trade practive, enabling the Federal Trade Commission to bring action? You can't lie in a privacy policy, or a TV commercial. Where is the line?
- wstuartcl 8y ago
  ... It almost makes you wonder if there is a process for ensuring companies comply with secret investigations and are forced to act publicly and privately as if they have never happened. https://en.wikipedia.org/wiki/United_States_Foreign_Intelligence_Surveillance_Court https://en.wikipedia.org/wiki/United_States_Foreign_Intellig...

deleted 8y ago

[deleted]

jnbiche 8y ago

A lot of people are unaware of how anonymous sources in a serious news organization work. Here, it means that the multiple high-level intelligence officials described in the article are known to and vetted by Bloomberg. They've looked at their resumes and bona fides, and confirmed their backgrounds. They're just not revealing their names to us. So which is more likely: that multiple intelligence officials are making this up, or that Apple/Amazon/Supermicro feel obligated to lie because this is an ongoing classified counterintel investigation?

cyphunk 8y ago
> So which is more likely The language of the Apple refute is so strong, to the point of directly attacking Bloomberg and calling them irresponsible. So yes, in this case there are at least a few lies being peddled by the Bloomberg intelligence contacts
kazagistar 8y ago
Honestly, both seem somewhat plausible. I'll let it shake itself out.
lolc 8y ago
Plenty of claims by "senior intelligence officials" have proven to be factually incorrect. Same goes for press statements. It's simply too early to tell who's telling the truth, who's mistaken, and who's lying here.
deleted 8y ago
[deleted]

cyphunk 8y ago

I have not seen anything that indicates how installing this chip would do anything at all without also modifying the trace design and fabrication of the PCB itself. Also does anyone have information about the "baseboard management controller" mentioned? I would like to understand the complexity required to MiTM a ROM or FLASH memory read by such a controller before concluding the feasibility and number of players in manufacturing chain required for it to work.

adrian_b 8y ago
The BMC is an ARM microcontroller that has complete access to everything, exactly like the Intel Management Engine, but the server vendors prefer a separate chip for the same job. What is described in the article is very easy to do by inserting a microcontroller on the SPI link that connects the BMC with the flash memory containing the BMC programs, which are copied from there to a RAM at boot. However, such a microcontroller would need 8 pins for at least 7 signals: ground, power, SPI clock and 4 SPI data, to and from BMC and flash memory. Nonetheless, 8-pin packages can be very small, e.g. 0.8 mm by 1.35 mm, i.e. only slightly larger than 1 square millimeter. So from a technical point of view, all is easily feasible. The problem is that it would require compromised people in several places at the subcontractors, because the design files for the PCB must be replaced wherever the PCB is made and in another place, at the PCB assembly, the pick & place document must be replaced and an extra reel with the backdoor component must be mounted on the equipment and that reel must come from somewhere else than from the normal suppliers of the assembly line without raising suspicions. It can be done, but many accomplices are required. Because most of the time the backdoor component will pass the SPI data signals transparently, it will not be detected at any electrical testing and the usual optical inspections are unlikely to detect such a small change. I am using many Supermicro motherboards, so I am wondering if this story is true. If it were true, it would not be much of a surprise, because they did not do something really novel but they just matched what USA also did, e.g. in the Cisco case.
- cyphunk 8y ago
  But also if a modification of the SPI bus can result in exploitation then the manufacturer will just patch the memory in place. Cheaper, easier, less detectable. So... the manufacturer is not the source of attack.

ggm 8y ago

I've always wondered if that passive device in the great crest at the US embassy in Moscow had equivalents which got hooked up to consumer devices with high voltage parts (to make people reluctant to play inside) Remember the furore when Zenith was the last domestic manufacturer of TVs in the USA? We've come a long way since then..

astrange 8y ago
It's not that dangerous to open electronics ever since they stopped using electron guns. But, consumer devices don't need to secretly spy on you when consumers buy them literally for the purpose of being spied on.

21 8y ago

They allege 30 companies were compromised with implanted Supermicro servers. Surely one of them could show the world/researchers an infected motherboard?

simplecomplex 8y ago
This. Where did all the infected motherboards go??

kevlar1818 8y ago

To me, this is perhaps the most worrying part of the story. Did Bloomberg, a widely renowned and distributed news outlet with immense resources, sacrifice hard evidence for sensationalism and clicks? Or are these companies, all widely renowned with immense resources, bound to silence due to any multitude of shady reasons? No matter the facts behind the story and these denials, this whole thing reeks of FUD.

PascLeRasc 8y ago

Could anyone find a list of the server models that had this compromise? I haven't seen much if any technical info in either of the Bloomberg articles.

baybal2 8y ago
microblade 6128
- justtopost 8y ago
  Source?
  - baybal2 8y ago
    Just look at the photo, and the motherboard model. It is only used in 1 server. And given Elemental is mentioned, it must be a blade So that matches nicely.

oropolo 8y ago

Are there facilities anywhere in the US where motherboards can be manufactured?

bogomipz 8y ago

The article states: >"Somewhere in the Linux operating system, which runs in many servers, is code that authorizes a user by verifying a typed password against a stored encrypted one. An implanted chip can alter part of that code so the server won’t check for a password—and presto! A secure machine is open to any and all users." I realize the intended audience for this article is not a technical crowd but can someone walk me through in practical terms how such a chip might subvert the /bin/login binary?

gluegadget 8y ago
Quite possibly through IPMI as PAM.
- bogomipz 8y ago
  Can you elaborate?
  - sounds 8y ago
    Let me elaborate your question. "can someone walk me through in practical terms how such a chip might subvert the /bin/login binary?" Unpacking that: 1. Please, random internet stranger, can you... 2. Make a nice bloggy article with pretty pictures... 3. That includes practical handy advice... 4. How, once I've compromised the system at the BMC... 5. To write an exploit for the linux kernel without getting caught. Preferrably one that silently patches /bin/login. Yeah, if you know how to do that, you're not posting it on the internet. You're on anonymous message boards asking for jobs from one side or the other.
    - bogomipz 8y ago
      Yeah that wasn't my question at all. Maybe you aren't a native English speaker but "walk me through" is an idiom in the English language it is not a literal phrase. The OP's comment was quite terse and so I asked if they wouldn't mind elaborating. As you can see someone else replied with a more concrete example, which is all I was asking. Your response displays a shocking level of immaturity that appears to have been triggered by your own total and complete misunderstanding.

sounds 8y ago

"In the three years since the briefing in McLean, no commercially viable way to detect attacks like the one on Supermicro’s motherboards has emerged—or has looked likely to emerge." I know a commercially viable way to detect hardware attacks. Standardized hardware designs, such as the "x86 standard," ARM IP licenses, and more recently, RISC-V, decentralizes manufacturing and drives the cost to commodity levels. I'm specifically proposing that the U.S. Government appropriate the patents on whichever hardware design and declare them a National Security asset, and then guarantee royalty-free licenses to any company that wants to use them. When it's no longer a big profit center, China is no longer as interested in owning a monopoly on it. And presto: there's no longer a monopoly on the hardware. Thus it's no longer a guarantee that your hardware is being bent to the will of a single nation-state. Hardware attacks can be detected as variations between the hardware made by one nation vs. the hardware made by another nation. The downside is that Apple can't have the same profit margins that come from closed, proprietary hardware. The upside is that manufacturing and process innovation (such as Intel used to do) becomes extremely desirable. It becomes so valuable that we saw Intel reluctant to offshore their best processes. There: economic solution and political points, to boot!

21 8y ago
Have you read the article? Apparently one of the big factors in Supermicro success is that it has over 900 different motherboard designs, and hundreds of hardware specialists which can customize them further to client wishes.
- sounds 8y ago
  Your post is a non-sequitur. I'm not disagreeing that Supermicro makes customized designs.
  - 21 8y ago
    It is a sequitur in the sense that a few open-hardware motherboard designs will not be commercially successful.

fmajid 8y ago

The cost of manufacture is a very small part of the cost of assembling computers. For the iPhone, for instance, it represents less than $10 out of the $240 or so total cost. Thus shifting production to other another locale with higher costs would not significantly increase the price of the product, and in any case labor costs are going up in China as well. Thus you wonder why more production isn't being shifted from China to, say, Thailand, Indonesia or India. Steve Jobs once said the industrial capacity to do the work simply isn't available outside of China, in terms of skilled people and supply chains, and that may be a big reason why.

DenisM 8y ago

It's interesting that the numerous sources all decided to confine in Bloomberg. How did they all know to go to the same paper? And if some of them went to other papers then we should hear about it tomorrow, I guess.

nasseri 8y ago

Can someone point me to the exact laws or cases with precedent that would make Apple and Amazon's statements illegal if the Bloomberg article were true? I'm not doubting it, but I can't find much about this online. Pump and dump schemes, ponzi schemes, and insider trading are illegal, and those aspects of securities fraud are well documented online, but this more general "lying to the public" that these companies may or may not be doing has proved trickier to find precedence for. I am not a lawyer, and my only resource here is google, but I think there is an assumption at play that this form of lying is illegal, and I'm honestly not sure that it is. I could be wayy of base here, but if its not illegal, than it would be pretty obvious what is going on.

maerF0x0 8y ago

This has been posted several times and there are tons of comments: [1]: https://news.ycombinator.com/item?id=18146438 https://news.ycombinator.com/item?id=18146438 [2]: https://news.ycombinator.com/item?id=18138328 https://news.ycombinator.com/item?id=18138328 [3]: https://news.ycombinator.com/item?id=18145645 https://news.ycombinator.com/item?id=18145645 [4]: https://news.ycombinator.com/item?id=18138990 https://news.ycombinator.com/item?id=18138990 [5]: https://news.ycombinator.com/item?id=18141328 https://news.ycombinator.com/item?id=18141328