13 ms·

What Businessweek got wrong about Apple

267 points by rakkhi 8y ago

calimac 8y ago
Bloomberg never said you bought supermicro servers they said you bought supermicro motherboard. Is Apple’s legal team creatively writing to misinterpret the fax
deft 8y ago
My personal theory is this is a ploy to get people to believe Trump's anti-china narrative meant to distract from the Russia narrative.
- _up 8y ago
  The Russia narrative makes no sense Russia and Saudi Arabia would surely have preferred Hillary who promised to outlaw fracking. Now the US is becoming the biggest Oil exporter. Russian GDP is like 40% from Oil, Saudi Arabia even more.
- chvid 8y ago
  As an outside observer of us politics it seems to me that the democrats want Russia as an external enemy whereas the republicans want China. Perhaps now it seems like the republicans have the upper hand but it is not obvious how it is going to end.
baxtr 8y ago
In an appearance this morning on Bloomberg Television, reporter Jordan Robertson made further claims about the supposed discovery of malicious chips, saying, “In Apple’s case, our understanding is it was a random spot check of some problematic servers that led to this detection.” As we have previously informed Bloomberg, this is completely untrue. Apple has never found malicious chips in our servers. Finally, in response to questions we have received from other news organisations since Businessweek published its story, we are not under any kind of gag order or other confidentiality obligations. It can’t get much clearer than that. The whole story is quite weird. I don’t know who should be believed but I’ve never read any such vehement denial. If Apple is lying they risk quite a bit of credibility here
- _Codemonkeyism 8y ago
  Free get out of jail card: Three letter agency force me.
ElBarto 8y ago
They complain too much... Apple apparently entirely dropped Supermicro as a supplier over a few weeks when they were planning a large order(source: theregister.co.uk). The ones who should strongly deny such a story, if it is indeed incorrect, are Supermicro. Is there a statement from them? Edit: yes, there is. They are "not aware of any investigation". That tells me all I need to know...
- Macuyiko 8y ago
  They did: https://www.bloomberg.com/news/articles/2018-10-04/the-big-hack-amazon-apple-supermicro-and-beijing-respond https://www.bloomberg.com/news/articles/2018-10-04/the-big-h...
  - ElBarto 8y ago
    Yes saw that and edited.
deleted 8y ago
[deleted]
onetimemanytime 8y ago
Very strong denial. Frankly, if true and Apple is saying this kind of a "no" shareholders will sue. Two possibilities: Left hand doesn't know (or can't know) what the right hand is doing at Apple. Top secret? Bloomberg was a victim of a hoax, some nation state (huh huhm!) wants to target China for something so they need a story. Based on what I've read here these past days, I'm leaning towards the second one. Apple can hire the best or all Infosec companies in the world if security was compromised. In other words, they'd know by now, even if they missed it originally. Cat and mouse and all...
- amf12 8y ago
  > Two possibilities: Left hand doesn't know (or can't know) what the right hand is doing at Apple. Top secret? I am sure the CEO must know both the hands, if the hands don't know each other. Also that Apple is going to release a strong denial must also have been known to Tim. I don't think that's the case here. Your second possibility looks more plausible.
  - dannyw 8y ago
    If you talk to anyone with a TS/SCI clearance, you'll know that it is very common to be unable to discuss or divulge what you're working on to your superiors. There are MANY material things that CEOs do not know about TS/SCI information.
    - onetimemanytime 8y ago
      Can they say that "our servers /info" has been compromised? I mean, what's the point,Apple is not FBI /NSA owned. They have the right to know material things, even if specifics might be lacking. Yes, looks like Apple dropped SuperMicro as their supplier in 2017. Why is that $64k question.
    - brorfred 8y ago
      But that's when your company has a contract with the government, right? Why would there be anyone on Apple's side with TS/SCI clearance? Didn't Bloomberg say that Apple found the chips? How would you be able to replace + change suppliers of 7000 servers without explaining why to senior leadership?
jdorfman 8y ago
Stupid legal question, could this end up becoming a defamation lawsuit?
- CalChris 8y ago
  Similar question, but should BW turn out to be correct and Apple was for lack of a better word, lying, aren't they on the hook as a public company?
  - Tomte 8y ago
    Matt Levine likes to say that "everything is securities fraud".
    - hyperrail 8y ago
      Funnily enough, he actually covered this angle yesterday! Regular readers know that a major theme of this newsletter is Everything Is Securities Fraud, so in that vein, let us consider a hypothetical. What if: 1. Everything in the Businessweek story is true, Chinese spies planted hardware backdoors in computers built and used by major American companies, and the FBI investigated along with those companies and discovered the backdoors. 2. It is a national-security secret and the companies were instructed by the FBI never to acknowledge it. 3. The companies are patriotically but falsely denying the hack. If that is true—and I have no particular reason to think it is, it’s just the sort of hypothetical we like around here—then, obvious question: Is it securities fraud? (Assuming that the hack is potentially material to the companies’ business?) I do not think that the securities laws explicitly allow companies to make false statements of material fact if required for national security, but you could see giving them a pass here. https://www.bloomberg.com/view/articles/2018-10-04/computer-spies-hacked-reality https://www.bloomberg.com/view/articles/2018-10-04/computer-...
      - hef19898 8y ago
        It will definetly be bad for the business of these companies, especially for Amazon, as it undermines customer trust. In the case of Amazon also the most profitable division, AWS, which provided the majority of profits in 2017.
Jerry2 8y ago
Both Apple and Amazon have released VERY STRONG denial statements that bring the whole Bloomberg narrative into question. It's also convenient that no one has yet been able to verify or find any of these mysterious Chinese chips on any of the Supermicro servers in the wild. So what is the real story here? Did Bloomberg reporters deliberately deceive everyone or were they deceived by the US IC ("intelligence community") as a way to scare technology companies from doing business in China? Someone at the SEC should scrutinize SMCI shorts at the very least.
- peterlk 8y ago
  To be clear, there is a very real threat of China embedding chips that do "bad things" inside chips that are sold to the US (source: I once worked on a government contract and saw such chips - they were, to my knowledge, unrelated to this specific case). I'm not sure where that points, but there are strong motivations at play.
  - cm2187 8y ago
    Not the least because since Snowden we know the NSA does it.
  - xiphias2 8y ago
    The key is ,,inside chips''. There's no reason to do those bad things on the die using an extra chip, as it's too easy to prove.
- AdequateLobster 8y ago
  Well, did you expect any of those companies to acknowledge their own incompetence? And do you seriously think Businessweek can make all this up out of thin air?
- notatoad 8y ago
  It's certainly beginning to look like Bloomberg got played. I don't know what the motivation would be for Bloomberg to deliberately deceive readers, but I can't help notice that the article came out at roughly the same time as Mike Pence was giving a speech whose central premise was that China is meddling in american politics, and all the attention currently focused on Russian hacking should be focused on Chinese hacking. If this story actually is false the US government has plenty of incentive to have planted it.
GordonS 8y ago
The cynic in me wonders about the plausibility of all this. Firstly, why would you add a new chip to a board, rather than alter an existing one? That would be essentially undetectable. Secondly, why Bloomberg? It's an odd organisation to get a scoop on something like this. Thirdly, they talk of the PLA approaching plant owners and such; to do all this, a lot of people would need to know about it, from the top to the bottom. I imagine that would be very difficult to keep secret. Finally, the timing is very suspicious - it comes with midterms approaching, and Trump and China arguing over trade tarrifs; it would serve the political narrative well for China to be painted as the 'bugbear de jour', and this also plays to the MAGA crowd.
- akavel 8y ago
  Can you delete one of your 2 nearly identical comments?
  - GordonS 8y ago
    Done, and thanks for pointing this out. I wrote this on mobile, and thought the app (Materialistic) had lost what I'd written, hence I wrote it again. Not targeting you with this, but it seems a bit harsh to get downvoted for what was obviously a mistake.
    - mercer 8y ago
      I think many people, myself included, will downvote duplicate comments without too much thought. It might seem harsh, and perhaps it shouldn't be done so thoughtlessly, but at least based on my personal experience it's not meant to be harsh.
- robbiep 8y ago
  >Thirdly, they talk of the PLA approaching plant owners and such; to do all this, a lot of people would need to know about it, from the top to the bottom. I imagine that would be very difficult to keep secret. Well the bloomberg article said that intelligence sources knew that those boards were going to Apple and Amazon, so presumably (if the story is true) someone did speak, maybe
chillax 8y ago
The Norwegian National Security Authority (https://nsm.stat.no/english/ https://nsm.stat.no/english/) is quoted in a norwegian paper today saying they knew about problems with Super Micro since at least june. https://www.vg.no/nyheter/i/xRkLep/storavis-hevder-kina-installerte-spionverktoey-i-maskinvare https://www.vg.no/nyheter/i/xRkLep/storavis-hevder-kina-inst...
- Jerry2 8y ago
  What exactly did they know? Did they check anything themselves or did they hear the same story from the US IC that Bloomberg also heard? Supermicro servers are extremely popular in data centers. Yet no one has noticed anything and no one has found any malicious chips in them. Unless the Chinese secret services also hacked all of the firewalls, someone would have picked up some outgoing packets that are going to Chinese C&C servers. And those would have been scrutinized and chased down. A company I interned at this past summer was scrutinizing every client on their internal network. They would have easily detected a machine that was connecting to an IP outside the subnet, for example. This whole story is just fishy and every company mentioned in that Bloomberg piece is not only denying it, but strongly denying it ever happened. And Bloomberg reporters have zero evidence... except for some stories from "anonymous sources".
  - cm2187 8y ago
    The article suggested the chips were hidden under components. You would have to tear apart the motherboard and know what to expect exactly. The number of companies who do this kind of audit must be in the single digit if any. Now the article may or may not be a hoax, but a few years ago the NSA was exposed doing exactly that. So at the very least it is credible. As for detection, I’d rather expect this devices to be activated on demand, like creating a backdoor, rather than sending streams of data from all servers, most of which (per the article) will be owned by an adult website or a mormon church which aren’t exactly strategic activities.
    - rickycook 8y ago
      well, the article said that amazon noticed pings to c&c servers in its beijing datacentre. it did also say that these were more sophisticated hardware attacks but didn’t make it clear if these chips were also found
RepAgent 8y ago
> Despite numerous discussions across multiple teams and organisations, no one at Apple has ever heard of this investigation. If this is some kind of ongoing national security issue with nondisclosure requirement authorized by the Director of the FBI, like this big breach could be, people involved are not allowed to talk about it even inside their company. Of course it would be advisable to inform higher ups in the Apple so that they would not issue a denial.
writepub 8y ago
Bloomberg needs to make a statement about all of this, either doubling down or issuing an apology. Either ways, we need a follow up and conclusion. Can we HN-ers tweet-request them (politely) to follow up?
whatever1 8y ago
I mean what was the alternative? To admit that your supply chain is compromised and blame directly the government of the country where you produce (and sell to a level) all of your hardware? That would be a huge blow in the credibility of the company and would raise serious questions on why they did not move the manufacturing elsewhere.
- IshKebab 8y ago
  If this were true I would have expected some corporate waffle like "Apple takes supply chain security very seriously and regularly audits suppliers. We cannot comment on internal security matters but customers should be assured that blah blah blah." I wouldn't have expected this extremely strong denial which is one step short of outright calling Bloomberg liars.
  - whatever1 8y ago
    No offense, but I think that your suggested statement would only lead to serious escalation for media enquiries, especially since this story came 2 days after the interview of the Apple CEO [1], where he underlined that privacy (and the relevant security measures) constitutes the core value of the company. [1] https://www.youtube.com/watch?v=VD1cP8SK3Q0 https://www.youtube.com/watch?v=VD1cP8SK3Q0
    - Arn_Thor 8y ago
      If Apple or Amazon are found to have lied in their statements I think they could be in serious legal trouble with both their customers but also (sadly, more importantly) their stock holders.
      - _Tev 8y ago
        More trouble than if they pissed China off enough to threaten Apple's manufacturing in China?
deleted 8y ago
[deleted]
kerng 8y ago
This will be interesting to follow, it's very unlikely there is not some truth to this. The fact that Apple and others are pushing so strongly against the story (very defensive) which makes me believe they are hiding something for sure.
charlysl 8y ago
Regardless of whether this particular case is true or not, given the crucial role of computer systems in so many key institutions, it seems to me extremely risky to trust Chinese suppliers not to try to compromise critical infrastructure. Then again, I understand that it could be argued that, if this is confirmed, to me it would seem quite rash from the Chinese, given that they would have known all along that such a scheme would be discovered sooner or later. It is one thing to plant a device as part of a spy operation, quite another to consistently compromise a whole supply chain. Whichever is the case, the national interest and commercial interests seem to be seriously incompatible with one another when it comes to outsourcing such critical infrastructure to China, this seems obvious to me, regardless of the China policy of who is in government in US.
aylmao 8y ago
This article sounded a bit weird to me from the technical level, but I just assumed it could be lack of clear understanding on the nitty-gritty from the journalist, or just me not knowing about hardware enough to know what's possible and how. Given this is all getting a little fishy I'll share what had me thinking: 1. The article mentions "they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet..." Servers tend to run on VPNs. This being a dormant backdoor is believable, but then the article mentions: > "American investigators eventually figured out who else had been hit. Since the implanted chips were designed to ping anonymous computers on the internet for further instructions, operatives could hack those computers to identify others who’d been affected." Which makes me believe the devices were active and somehow circumvented corporate VPNs. I'm unsure how undetectable this could be using the system's network stack (or if it would be possible at all)-- would the claim then be that this tiny device shipped with a whole TCP/IP layer and some sort of very powerful wireless capability? 2. It continues with: "and preparing the device’s operating system to accept this new code" Is this possible? Where would a device like this need to be wired to be able to write to memory with some arbitrary payload to do this? From the pictures it looks like it has 6 pins maximum-- could this do? If so, wouldn't this mean this device would need to do some next-level signal processing that would probably require advanced computation? Could said computation be done by a processing unit that fits the size of this chip? Moreover, assuming it takes control of the OS independently would imply there's some decent amounts of memory in here, to hold the payload, etc. no? But if it's just a backdoor that doesn't take control of the OS, then how is it communicating over the internet with other machines like the article claims? Again, I might be wrong and things that I don't think possible might. I'm mostly just curious to know if my intuition is too naive. Please comment below if you know more about these things than I do. EDIT: I was really disappointed that the article itself didn't go into these technicalities, because IMO this would be an impressive feat and newsworthy by itself. The lack of alternative coverage in sources more close to technical expertise was weird to me.
- solarkraft 8y ago
  If this is actually true (and I have doubts), the reporting will explode in the next few days/weeks with all the detail you could wish for. A few pins can be enough to write to memory, if there's an interface for it (SPI, I2C, but I'd be surprised to see those unprotectedly lying around on a server board ...). It's be very slow, but I think networking could work as well. With efficient software the resources needed on the chip itself could be held very low and eventually offloaded to the main processor. I don't think it has any wireless or DSP stuff. I don't know enough about processors, memory, networking and low level software security to know how difficult it is to compromise a computer with such a chip on the main board, but the chip itself certainly seems feasible. But: It must be really expensive to manufacture the compromised devices. It just seems stupid to manufacture every device with such a chip. 1/1000 seems reasonable. The larger companies have (had?) lots of them anyway.
xmly 8y ago
The rice is indeed small, but it is not small on an IC chip. When people check the chip, they usually use a tool called microscope, like this https://goo.gl/1XK4YK https://goo.gl/1XK4YK.
- xmly 8y ago
  And there is xray to detect what is inside a chip like this: https://www.youtube.com/watch?v=XXDsM3mUv3Y https://www.youtube.com/watch?v=XXDsM3mUv3Y These are quite mature and popular technique. Rice is too big to be unnoticed....
  - rasz 8y ago
    No, you cant inspect inside chips with this Xray machine, its used for inspecting solder joints under the package.
- eps 8y ago
  Link de-googled: https://www.amscope.com/digital-zoom-inspection-microscope-with-3d-lighting-and-dual-focus.html https://www.amscope.com/digital-zoom-inspection-microscope-w...
IOT_Apprentice 8y ago
From 2016: Report: Apple designing its own servers to avoid snooping Apple suspects that servers are intercepted and modified during shipping. "Apple has long suspected that servers it ordered from the traditional supply chain were intercepted during shipping, with additional chips and firmware added to them by unknown third parties in order to make them vulnerable to infiltration, according to a person familiar with the matter," the report said. "At one point, Apple even assigned people to take photographs of motherboards and annotate the function of each chip, explaining why it was supposed to be there. Building its own servers with motherboards it designed would be the most surefire way for Apple to prevent unauthorized snooping via extra chips." https://arstechnica.com/information-technology/2016/03/report-apple-designing-its-own-servers-to-avoid-snooping/ https://arstechnica.com/information-technology/2016/03/repor...
- solarkraft 8y ago
  Eh? Instead of verifying an existing design they'll just make one themselves? That could be compromised too?
  - princekolt 8y ago
    Because using a 3rd party design you have to ask or guess what each chip does, and they might not want to tell you everything for IP reasons. When you create your own board, you know what every little chip and trace does exactly, and it's a lot easier to detect a rogue chip.
hacknat 8y ago
Companies don’t give vehement denials like this unless they’re telling the truth. People claiming gag orders are crazy, mostly for thinking that Apple, or anyone else for that matter, would ever sign a document forcing them to lie to their customers (I’m not saying they wouldn’t lie, just that they wouldn’t sign anything that would force them to do so).
- ItsMe000001 8y ago
  That makes no sense. What do you think is the difference in "denial levels"? Kind of like Dragonball-Z power levels? Does a "vehement denial" cost the one making it any more than a "meek denial"? If making "vehement denials", coming at exactly the same cost as less strong denials, are more effective, you would just have made all PR companies/people extremely happy - they get a stringer weapon for free. All the have to do is issue "vehement denials" instead of just "denials" and a larger share of the population believes them (for no valid reason). Especially when the public has no way to get the "real truth" but can only watch a Dragonball-Z style "strong statement" vs "incredibly strong statement" showdown. > People claiming gag orders are crazy, mostly for thinking that Apple, or anyone else for that matter, would ever sign a document forcing them to lie to their customers Why do you think a "gag order" is something that has to be signed off on by the one getting it? That too does not make any sense. An order is issued by someone who has more power, here, the government. You don't have to sign anything for the order to exist. "Gag order" has "order" right in the name.
  - miracle2k 8y ago
    Maybe vehemence isn't the right measure. It would be better to say specificity. "The allegations are false" leaves more wiggle room then to say "we don't have a business relationship with Suoermicro and a Company called Apple doesn't even exist".
  - brorfred 8y ago
    I saw figures that Bloomberg had 17 sources for this story. For that to in nay way be realistic, this must be a deliberate leak by the US government. Why would they put such a strong gag order on Apple/Amazon and then leak the information themselves? It makes no sense.
mcqueenjordan 8y ago
AWS Reply: https://aws.amazon.com/blogs/security/setting-the-record-straight-on-bloomberg-businessweeks-erroneous-article/ https://aws.amazon.com/blogs/security/setting-the-record-str...
Bryan_Tiernan 8y ago
It's hard to say what the truth is here, but what I will say is if that Bloomberg reporter doesn't have substantial evidence to prove that claim he could be in serious trouble. SuperMicro's stock was down 50% straight after that articles release, and it's not looking so hot right now either. He could be looking down the barrel of an SEC investigation very soon.
yAnonymous 8y ago
Consider that Apple also stated this a few years back: >"We have never heard of PRISM. We do not provide any government agency direct access to our servers, and any government agency requesting customer data must get a court order." Their whole business is built around lying to customers.
_pmf_ 8y ago
> "As a matter of practice, before servers are put into production at Apple they are inspected for security vulnerabilities and we update all firmware and software with the latest protections." They do not have equipment to detect this kind of attack, period. It's not viable for each device, and it's not even viable for sampling a subset of devices from a given production batch. Some components are physically inaccessible and would require desoldering of other components to even access them in any way. These kinds of attacks cannot be generically detected in any economically feasible way; it must be prevented by drastically clamping down the supply chain and the logistics chain.
doe88 8y ago
I would say one specific detail (I haven't looked at it though) would challenge the truth of the rebuttal of both Amazon and Apple is that if it is confirmed that both have severed ties with Supermicro around the same time, the coincidence would really seem odd then.
avryhof 8y ago
Political mind games. Right now, most of the tech industry, and a good portion of the news media are at odds with the executive branch of the government. This article puts at least one popular news outlet against several tech industry giants. Divide. What comes after divide? ...and who has the most to gain? I doubt it's actually our executive branch. I think they could be getting played just as much as Bloomberg and the Tech industry.
partiallypro 8y ago
DoD contracts for the military require the hardware to be sourced and made in the US to prevent compromise. I wonder if one day we will see the DoD require any Cloud contractor that has DoD datacenters to source from the US or NAFTA countries...and what impact that would have. I've heard ramblings about a lot of companies moving their manufacturing and sourcing from China to Vietnam already.
chasd00 8y ago
Incredibly interesting story and discussion, this is why i come to this site. Isn't this practice known, if not common, in the infosec/intelligence communities at the nation level? There's lots of stories of hardware exploits in copy machines, faxes, etc that took place during the Cold War.
- SpicyLemonZest 8y ago
  It's a known practice, but security-sensitive organizations have supply chain management that's typically understood to prevent it from happening on a large scale.
okket 8y ago
FYI: "Britain’s national cyber security agency said on Friday it had no reason to doubt the assessments made by Apple and Amazon that refuted a Bloomberg story that their systems contained malicious computer chips inserted by Chinese intelligence. [...]" https://www.reuters.com/article/us-china-cyber-britain/uk-cyber-security-agency-backs-apple-amazon-china-hack-denials-idUSKCN1MF1DN https://www.reuters.com/article/us-china-cyber-britain/uk-cy...

267 points by rakkhi 8y ago

Bloomberg never said you bought supermicro servers they said you bought supermicro motherboard. Is Apple’s legal team creatively writing to misinterpret the fax

deft 8y ago

My personal theory is this is a ploy to get people to believe Trump's anti-china narrative meant to distract from the Russia narrative.

_up 8y ago
The Russia narrative makes no sense Russia and Saudi Arabia would surely have preferred Hillary who promised to outlaw fracking. Now the US is becoming the biggest Oil exporter. Russian GDP is like 40% from Oil, Saudi Arabia even more.
chvid 8y ago
As an outside observer of us politics it seems to me that the democrats want Russia as an external enemy whereas the republicans want China. Perhaps now it seems like the republicans have the upper hand but it is not obvious how it is going to end.

baxtr 8y ago

In an appearance this morning on Bloomberg Television, reporter Jordan Robertson made further claims about the supposed discovery of malicious chips, saying, “In Apple’s case, our understanding is it was a random spot check of some problematic servers that led to this detection.” As we have previously informed Bloomberg, this is completely untrue. Apple has never found malicious chips in our servers. Finally, in response to questions we have received from other news organisations since Businessweek published its story, we are not under any kind of gag order or other confidentiality obligations. It can’t get much clearer than that. The whole story is quite weird. I don’t know who should be believed but I’ve never read any such vehement denial. If Apple is lying they risk quite a bit of credibility here

_Codemonkeyism 8y ago
Free get out of jail card: Three letter agency force me.

ElBarto 8y ago

They complain too much... Apple apparently entirely dropped Supermicro as a supplier over a few weeks when they were planning a large order(source: theregister.co.uk). The ones who should strongly deny such a story, if it is indeed incorrect, are Supermicro. Is there a statement from them? Edit: yes, there is. They are "not aware of any investigation". That tells me all I need to know...

Macuyiko 8y ago
They did: https://www.bloomberg.com/news/articles/2018-10-04/the-big-hack-amazon-apple-supermicro-and-beijing-respond https://www.bloomberg.com/news/articles/2018-10-04/the-big-h...
- ElBarto 8y ago
  Yes saw that and edited.

deleted 8y ago

[deleted]

onetimemanytime 8y ago

Very strong denial. Frankly, if true and Apple is saying this kind of a "no" shareholders will sue. Two possibilities: Left hand doesn't know (or can't know) what the right hand is doing at Apple. Top secret? Bloomberg was a victim of a hoax, some nation state (huh huhm!) wants to target China for something so they need a story. Based on what I've read here these past days, I'm leaning towards the second one. Apple can hire the best or all Infosec companies in the world if security was compromised. In other words, they'd know by now, even if they missed it originally. Cat and mouse and all...

amf12 8y ago
> Two possibilities: Left hand doesn't know (or can't know) what the right hand is doing at Apple. Top secret? I am sure the CEO must know both the hands, if the hands don't know each other. Also that Apple is going to release a strong denial must also have been known to Tim. I don't think that's the case here. Your second possibility looks more plausible.
- dannyw 8y ago
  If you talk to anyone with a TS/SCI clearance, you'll know that it is very common to be unable to discuss or divulge what you're working on to your superiors. There are MANY material things that CEOs do not know about TS/SCI information.
  - onetimemanytime 8y ago
    Can they say that "our servers /info" has been compromised? I mean, what's the point,Apple is not FBI /NSA owned. They have the right to know material things, even if specifics might be lacking. Yes, looks like Apple dropped SuperMicro as their supplier in 2017. Why is that $64k question.
  - brorfred 8y ago
    But that's when your company has a contract with the government, right? Why would there be anyone on Apple's side with TS/SCI clearance? Didn't Bloomberg say that Apple found the chips? How would you be able to replace + change suppliers of 7000 servers without explaining why to senior leadership?

jdorfman 8y ago

Stupid legal question, could this end up becoming a defamation lawsuit?

CalChris 8y ago
Similar question, but should BW turn out to be correct and Apple was for lack of a better word, lying, aren't they on the hook as a public company?
- Tomte 8y ago
  Matt Levine likes to say that "everything is securities fraud".
  - hyperrail 8y ago
    Funnily enough, he actually covered this angle yesterday! Regular readers know that a major theme of this newsletter is Everything Is Securities Fraud, so in that vein, let us consider a hypothetical. What if: 1. Everything in the Businessweek story is true, Chinese spies planted hardware backdoors in computers built and used by major American companies, and the FBI investigated along with those companies and discovered the backdoors. 2. It is a national-security secret and the companies were instructed by the FBI never to acknowledge it. 3. The companies are patriotically but falsely denying the hack. If that is true—and I have no particular reason to think it is, it’s just the sort of hypothetical we like around here—then, obvious question: Is it securities fraud? (Assuming that the hack is potentially material to the companies’ business?) I do not think that the securities laws explicitly allow companies to make false statements of material fact if required for national security, but you could see giving them a pass here. https://www.bloomberg.com/view/articles/2018-10-04/computer-spies-hacked-reality https://www.bloomberg.com/view/articles/2018-10-04/computer-...
    - hef19898 8y ago
      It will definetly be bad for the business of these companies, especially for Amazon, as it undermines customer trust. In the case of Amazon also the most profitable division, AWS, which provided the majority of profits in 2017.

Jerry2 8y ago

Both Apple and Amazon have released VERY STRONG denial statements that bring the whole Bloomberg narrative into question. It's also convenient that no one has yet been able to verify or find any of these mysterious Chinese chips on any of the Supermicro servers in the wild. So what is the real story here? Did Bloomberg reporters deliberately deceive everyone or were they deceived by the US IC ("intelligence community") as a way to scare technology companies from doing business in China? Someone at the SEC should scrutinize SMCI shorts at the very least.

peterlk 8y ago
To be clear, there is a very real threat of China embedding chips that do "bad things" inside chips that are sold to the US (source: I once worked on a government contract and saw such chips - they were, to my knowledge, unrelated to this specific case). I'm not sure where that points, but there are strong motivations at play.
- cm2187 8y ago
  Not the least because since Snowden we know the NSA does it.
- xiphias2 8y ago
  The key is ,,inside chips''. There's no reason to do those bad things on the die using an extra chip, as it's too easy to prove.
AdequateLobster 8y ago
Well, did you expect any of those companies to acknowledge their own incompetence? And do you seriously think Businessweek can make all this up out of thin air?
notatoad 8y ago
It's certainly beginning to look like Bloomberg got played. I don't know what the motivation would be for Bloomberg to deliberately deceive readers, but I can't help notice that the article came out at roughly the same time as Mike Pence was giving a speech whose central premise was that China is meddling in american politics, and all the attention currently focused on Russian hacking should be focused on Chinese hacking. If this story actually is false the US government has plenty of incentive to have planted it.

GordonS 8y ago

The cynic in me wonders about the plausibility of all this. Firstly, why would you add a new chip to a board, rather than alter an existing one? That would be essentially undetectable. Secondly, why Bloomberg? It's an odd organisation to get a scoop on something like this. Thirdly, they talk of the PLA approaching plant owners and such; to do all this, a lot of people would need to know about it, from the top to the bottom. I imagine that would be very difficult to keep secret. Finally, the timing is very suspicious - it comes with midterms approaching, and Trump and China arguing over trade tarrifs; it would serve the political narrative well for China to be painted as the 'bugbear de jour', and this also plays to the MAGA crowd.

akavel 8y ago
Can you delete one of your 2 nearly identical comments?
- GordonS 8y ago
  Done, and thanks for pointing this out. I wrote this on mobile, and thought the app (Materialistic) had lost what I'd written, hence I wrote it again. Not targeting you with this, but it seems a bit harsh to get downvoted for what was obviously a mistake.
  - mercer 8y ago
    I think many people, myself included, will downvote duplicate comments without too much thought. It might seem harsh, and perhaps it shouldn't be done so thoughtlessly, but at least based on my personal experience it's not meant to be harsh.
robbiep 8y ago
>Thirdly, they talk of the PLA approaching plant owners and such; to do all this, a lot of people would need to know about it, from the top to the bottom. I imagine that would be very difficult to keep secret. Well the bloomberg article said that intelligence sources knew that those boards were going to Apple and Amazon, so presumably (if the story is true) someone did speak, maybe

chillax 8y ago

The Norwegian National Security Authority (https://nsm.stat.no/english/ https://nsm.stat.no/english/) is quoted in a norwegian paper today saying they knew about problems with Super Micro since at least june. https://www.vg.no/nyheter/i/xRkLep/storavis-hevder-kina-installerte-spionverktoey-i-maskinvare https://www.vg.no/nyheter/i/xRkLep/storavis-hevder-kina-inst...

Jerry2 8y ago
What exactly did they know? Did they check anything themselves or did they hear the same story from the US IC that Bloomberg also heard? Supermicro servers are extremely popular in data centers. Yet no one has noticed anything and no one has found any malicious chips in them. Unless the Chinese secret services also hacked all of the firewalls, someone would have picked up some outgoing packets that are going to Chinese C&C servers. And those would have been scrutinized and chased down. A company I interned at this past summer was scrutinizing every client on their internal network. They would have easily detected a machine that was connecting to an IP outside the subnet, for example. This whole story is just fishy and every company mentioned in that Bloomberg piece is not only denying it, but strongly denying it ever happened. And Bloomberg reporters have zero evidence... except for some stories from "anonymous sources".
- cm2187 8y ago
  The article suggested the chips were hidden under components. You would have to tear apart the motherboard and know what to expect exactly. The number of companies who do this kind of audit must be in the single digit if any. Now the article may or may not be a hoax, but a few years ago the NSA was exposed doing exactly that. So at the very least it is credible. As for detection, I’d rather expect this devices to be activated on demand, like creating a backdoor, rather than sending streams of data from all servers, most of which (per the article) will be owned by an adult website or a mormon church which aren’t exactly strategic activities.
  - rickycook 8y ago
    well, the article said that amazon noticed pings to c&c servers in its beijing datacentre. it did also say that these were more sophisticated hardware attacks but didn’t make it clear if these chips were also found

RepAgent 8y ago

> Despite numerous discussions across multiple teams and organisations, no one at Apple has ever heard of this investigation. If this is some kind of ongoing national security issue with nondisclosure requirement authorized by the Director of the FBI, like this big breach could be, people involved are not allowed to talk about it even inside their company. Of course it would be advisable to inform higher ups in the Apple so that they would not issue a denial.

writepub 8y ago

Bloomberg needs to make a statement about all of this, either doubling down or issuing an apology. Either ways, we need a follow up and conclusion. Can we HN-ers tweet-request them (politely) to follow up?

whatever1 8y ago

I mean what was the alternative? To admit that your supply chain is compromised and blame directly the government of the country where you produce (and sell to a level) all of your hardware? That would be a huge blow in the credibility of the company and would raise serious questions on why they did not move the manufacturing elsewhere.

IshKebab 8y ago
If this were true I would have expected some corporate waffle like "Apple takes supply chain security very seriously and regularly audits suppliers. We cannot comment on internal security matters but customers should be assured that blah blah blah." I wouldn't have expected this extremely strong denial which is one step short of outright calling Bloomberg liars.
- whatever1 8y ago
  No offense, but I think that your suggested statement would only lead to serious escalation for media enquiries, especially since this story came 2 days after the interview of the Apple CEO [1], where he underlined that privacy (and the relevant security measures) constitutes the core value of the company. [1] https://www.youtube.com/watch?v=VD1cP8SK3Q0 https://www.youtube.com/watch?v=VD1cP8SK3Q0
  - Arn_Thor 8y ago
    If Apple or Amazon are found to have lied in their statements I think they could be in serious legal trouble with both their customers but also (sadly, more importantly) their stock holders.
    - _Tev 8y ago
      More trouble than if they pissed China off enough to threaten Apple's manufacturing in China?

deleted 8y ago

[deleted]

kerng 8y ago

This will be interesting to follow, it's very unlikely there is not some truth to this. The fact that Apple and others are pushing so strongly against the story (very defensive) which makes me believe they are hiding something for sure.

charlysl 8y ago

Regardless of whether this particular case is true or not, given the crucial role of computer systems in so many key institutions, it seems to me extremely risky to trust Chinese suppliers not to try to compromise critical infrastructure. Then again, I understand that it could be argued that, if this is confirmed, to me it would seem quite rash from the Chinese, given that they would have known all along that such a scheme would be discovered sooner or later. It is one thing to plant a device as part of a spy operation, quite another to consistently compromise a whole supply chain. Whichever is the case, the national interest and commercial interests seem to be seriously incompatible with one another when it comes to outsourcing such critical infrastructure to China, this seems obvious to me, regardless of the China policy of who is in government in US.

aylmao 8y ago

This article sounded a bit weird to me from the technical level, but I just assumed it could be lack of clear understanding on the nitty-gritty from the journalist, or just me not knowing about hardware enough to know what's possible and how. Given this is all getting a little fishy I'll share what had me thinking: 1. The article mentions "they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet..." Servers tend to run on VPNs. This being a dormant backdoor is believable, but then the article mentions: > "American investigators eventually figured out who else had been hit. Since the implanted chips were designed to ping anonymous computers on the internet for further instructions, operatives could hack those computers to identify others who’d been affected." Which makes me believe the devices were active and somehow circumvented corporate VPNs. I'm unsure how undetectable this could be using the system's network stack (or if it would be possible at all)-- would the claim then be that this tiny device shipped with a whole TCP/IP layer and some sort of very powerful wireless capability? 2. It continues with: "and preparing the device’s operating system to accept this new code" Is this possible? Where would a device like this need to be wired to be able to write to memory with some arbitrary payload to do this? From the pictures it looks like it has 6 pins maximum-- could this do? If so, wouldn't this mean this device would need to do some next-level signal processing that would probably require advanced computation? Could said computation be done by a processing unit that fits the size of this chip? Moreover, assuming it takes control of the OS independently would imply there's some decent amounts of memory in here, to hold the payload, etc. no? But if it's just a backdoor that doesn't take control of the OS, then how is it communicating over the internet with other machines like the article claims? Again, I might be wrong and things that I don't think possible might. I'm mostly just curious to know if my intuition is too naive. Please comment below if you know more about these things than I do. EDIT: I was really disappointed that the article itself didn't go into these technicalities, because IMO this would be an impressive feat and newsworthy by itself. The lack of alternative coverage in sources more close to technical expertise was weird to me.

solarkraft 8y ago
If this is actually true (and I have doubts), the reporting will explode in the next few days/weeks with all the detail you could wish for. A few pins can be enough to write to memory, if there's an interface for it (SPI, I2C, but I'd be surprised to see those unprotectedly lying around on a server board ...). It's be very slow, but I think networking could work as well. With efficient software the resources needed on the chip itself could be held very low and eventually offloaded to the main processor. I don't think it has any wireless or DSP stuff. I don't know enough about processors, memory, networking and low level software security to know how difficult it is to compromise a computer with such a chip on the main board, but the chip itself certainly seems feasible. But: It must be really expensive to manufacture the compromised devices. It just seems stupid to manufacture every device with such a chip. 1/1000 seems reasonable. The larger companies have (had?) lots of them anyway.

xmly 8y ago

The rice is indeed small, but it is not small on an IC chip. When people check the chip, they usually use a tool called microscope, like this https://goo.gl/1XK4YK https://goo.gl/1XK4YK.

xmly 8y ago
And there is xray to detect what is inside a chip like this: https://www.youtube.com/watch?v=XXDsM3mUv3Y https://www.youtube.com/watch?v=XXDsM3mUv3Y These are quite mature and popular technique. Rice is too big to be unnoticed....
- rasz 8y ago
  No, you cant inspect inside chips with this Xray machine, its used for inspecting solder joints under the package.
eps 8y ago
Link de-googled: https://www.amscope.com/digital-zoom-inspection-microscope-with-3d-lighting-and-dual-focus.html https://www.amscope.com/digital-zoom-inspection-microscope-w...

IOT_Apprentice 8y ago

From 2016: Report: Apple designing its own servers to avoid snooping Apple suspects that servers are intercepted and modified during shipping. "Apple has long suspected that servers it ordered from the traditional supply chain were intercepted during shipping, with additional chips and firmware added to them by unknown third parties in order to make them vulnerable to infiltration, according to a person familiar with the matter," the report said. "At one point, Apple even assigned people to take photographs of motherboards and annotate the function of each chip, explaining why it was supposed to be there. Building its own servers with motherboards it designed would be the most surefire way for Apple to prevent unauthorized snooping via extra chips." https://arstechnica.com/information-technology/2016/03/report-apple-designing-its-own-servers-to-avoid-snooping/ https://arstechnica.com/information-technology/2016/03/repor...

solarkraft 8y ago
Eh? Instead of verifying an existing design they'll just make one themselves? That could be compromised too?
- princekolt 8y ago
  Because using a 3rd party design you have to ask or guess what each chip does, and they might not want to tell you everything for IP reasons. When you create your own board, you know what every little chip and trace does exactly, and it's a lot easier to detect a rogue chip.

hacknat 8y ago

Companies don’t give vehement denials like this unless they’re telling the truth. People claiming gag orders are crazy, mostly for thinking that Apple, or anyone else for that matter, would ever sign a document forcing them to lie to their customers (I’m not saying they wouldn’t lie, just that they wouldn’t sign anything that would force them to do so).

ItsMe000001 8y ago
That makes no sense. What do you think is the difference in "denial levels"? Kind of like Dragonball-Z power levels? Does a "vehement denial" cost the one making it any more than a "meek denial"? If making "vehement denials", coming at exactly the same cost as less strong denials, are more effective, you would just have made all PR companies/people extremely happy - they get a stringer weapon for free. All the have to do is issue "vehement denials" instead of just "denials" and a larger share of the population believes them (for no valid reason). Especially when the public has no way to get the "real truth" but can only watch a Dragonball-Z style "strong statement" vs "incredibly strong statement" showdown. > People claiming gag orders are crazy, mostly for thinking that Apple, or anyone else for that matter, would ever sign a document forcing them to lie to their customers Why do you think a "gag order" is something that has to be signed off on by the one getting it? That too does not make any sense. An order is issued by someone who has more power, here, the government. You don't have to sign anything for the order to exist. "Gag order" has "order" right in the name.
- miracle2k 8y ago
  Maybe vehemence isn't the right measure. It would be better to say specificity. "The allegations are false" leaves more wiggle room then to say "we don't have a business relationship with Suoermicro and a Company called Apple doesn't even exist".
- brorfred 8y ago
  I saw figures that Bloomberg had 17 sources for this story. For that to in nay way be realistic, this must be a deliberate leak by the US government. Why would they put such a strong gag order on Apple/Amazon and then leak the information themselves? It makes no sense.

mcqueenjordan 8y ago

AWS Reply: https://aws.amazon.com/blogs/security/setting-the-record-straight-on-bloomberg-businessweeks-erroneous-article/ https://aws.amazon.com/blogs/security/setting-the-record-str...

Bryan_Tiernan 8y ago

It's hard to say what the truth is here, but what I will say is if that Bloomberg reporter doesn't have substantial evidence to prove that claim he could be in serious trouble. SuperMicro's stock was down 50% straight after that articles release, and it's not looking so hot right now either. He could be looking down the barrel of an SEC investigation very soon.

yAnonymous 8y ago

Consider that Apple also stated this a few years back: >"We have never heard of PRISM. We do not provide any government agency direct access to our servers, and any government agency requesting customer data must get a court order." Their whole business is built around lying to customers.

_pmf_ 8y ago

> "As a matter of practice, before servers are put into production at Apple they are inspected for security vulnerabilities and we update all firmware and software with the latest protections." They do not have equipment to detect this kind of attack, period. It's not viable for each device, and it's not even viable for sampling a subset of devices from a given production batch. Some components are physically inaccessible and would require desoldering of other components to even access them in any way. These kinds of attacks cannot be generically detected in any economically feasible way; it must be prevented by drastically clamping down the supply chain and the logistics chain.

doe88 8y ago

I would say one specific detail (I haven't looked at it though) would challenge the truth of the rebuttal of both Amazon and Apple is that if it is confirmed that both have severed ties with Supermicro around the same time, the coincidence would really seem odd then.

avryhof 8y ago

Political mind games. Right now, most of the tech industry, and a good portion of the news media are at odds with the executive branch of the government. This article puts at least one popular news outlet against several tech industry giants. Divide. What comes after divide? ...and who has the most to gain? I doubt it's actually our executive branch. I think they could be getting played just as much as Bloomberg and the Tech industry.

partiallypro 8y ago

DoD contracts for the military require the hardware to be sourced and made in the US to prevent compromise. I wonder if one day we will see the DoD require any Cloud contractor that has DoD datacenters to source from the US or NAFTA countries...and what impact that would have. I've heard ramblings about a lot of companies moving their manufacturing and sourcing from China to Vietnam already.

chasd00 8y ago

Incredibly interesting story and discussion, this is why i come to this site. Isn't this practice known, if not common, in the infosec/intelligence communities at the nation level? There's lots of stories of hardware exploits in copy machines, faxes, etc that took place during the Cold War.

SpicyLemonZest 8y ago
It's a known practice, but security-sensitive organizations have supply chain management that's typically understood to prevent it from happening on a large scale.

okket 8y ago

FYI: "Britain’s national cyber security agency said on Friday it had no reason to doubt the assessments made by Apple and Amazon that refuted a Bloomberg story that their systems contained malicious computer chips inserted by Chinese intelligence. [...]" https://www.reuters.com/article/us-china-cyber-britain/uk-cyber-security-agency-backs-apple-amazon-china-hack-denials-idUSKCN1MF1DN https://www.reuters.com/article/us-china-cyber-britain/uk-cy...