20 ms·

New Evidence of Hacked Supermicro Hardware Found in U.S. Telecom

875 points by gorbachev 8y ago

jackconnor 8y ago
They don’t say what the hack was and definitely do not say it was one of their pins. Probably some truth here, but as hard as they try, does not seem supportive of their “chinese pin” theory. Very suspicious that this is related, I’m guessing they’re trying to do anything to cover their asses.
- annerajb 8y ago
  What is the chinese pin theory? Is this the name being used for the supposedly embedded on the PCB layers chip? Or the one that attaches on unpopulated pins/pads near the BMC memory area?
- trevyn 8y ago
  Sounds like the Ethernet connector module was not from the, ahem, correct manufacturer: “Appleboum said one key sign of the implant is that the manipulated Ethernet connector has metal sides instead of the usual plastic ones. The metal is necessary to diffuse heat from the chip hidden inside, which acts like a mini computer. "The module looks really innocent, high quality and 'original' but it was added as part of a supply chain attack," he said.”
  - RL_Quine 8y ago
    I'm not sure I believe this one as much, just based on the part you quoted. I can see a chip manipulating the BMC/IPMI flash to make it do things it shouldn't. I don't see how an ethernet port could be modified to be interesting. They're typically after the magnetics, or contain the magnetics themselves, so the only source of power would be the activity LEDs, or something, or maybe we assume a custom PCB as well. You've then also got to have it doing gigabit ethernet, or otherwise tampering with data it got from that interface, which feels unlikely. Maybe it's just the same as the last implant story, hidden in a less easy to find place? Hard to know without something even approaching technical information.
    - simias 8y ago
      You could easily DoS obviously, but beyond that I agree that it seems tricky to do anything worthwhile.
      - moftz 8y ago
        It could just be a sort of beacon to help identify where hardware went after the manufacturing process. If the same company is building the same hardware, the agent can slip in something more nefarious to make sure they target the right company. Servers are commodity products but they aren't manufactured in mass quantities like phones are. If a company orders thousands of them, that's likely thousands that will need to be made. A chinese manufacturing plant gets contracted to spin up production and an implant is slipped into some of the first boards just to see where they go. You don't want an expensive hardware trojan to end up in a Fortnite server; you want to hit Apple, Google, Lockheed Martin, Spacex, anyone with valuable IP or information. The more beacon implants you throw out there, the more likely someone will find one and you don't want to get caught too early in the game. Once those implants come online and phone home, you have a better idea where the remaining boards are going and slip in the real deal implants, the ones that will actually get you a backdoor.
        simias 8y ago
        How would such a beacon work though? As RL_Quine points out there's only so much you can do at this point, especially if you want to be super stealthy. If you wanted to send a ping to an external server you'd have to craft an ethernet frame with the right target MAC address containing an IP datagram with the right IP address to be routed correctly in the datacenter and through the public firewall. You better make sure that your packet looks legit otherwise you're sure to trip anything looking for suspicious activity. "Hey look, our servers send weird packets to this suspicious IP, what gives?" And you have to do all that with a very low power device running from within the port itself. Seems like a very high bar to me, especially when there seems to be so many easier ways to backdoor a motherboard. But maybe the component is only hosted in the ethernet port but is actually connected to other signals on the motherboard.
  - makomk 8y ago
    I've never seen an onboard Ethernet jack that doesn't have metal sides. The only places I've seen all-plastic Ethernet jacks are consumer networking gear and really ancient add-on cards. That makes me wonder if their source actually knows what he's talking about, especially given the lack of technical details about how this works.
- maxden 8y ago
  This seems different than the extra chip attack according to the article. "subsequent physical inspection revealed an implant built into the server’s Ethernet connector"
  - close04 8y ago
    This feels like piling some more to cover up the weak premise of the previous article. It doesn't mean it's false, it just means Bloomberg already cried wolf and couldn't show it. At this point it feels more like taking shots at Supermicro than professional reporting.
    - late2part 8y ago
      totally seems like this is the case to me.
- simias 8y ago
  That's my take as well. They quote many "experts" and report many incidents but they seem hard pressed to come with details and pictures. I can believe that these attacks occur but are they really at the scale implied by these stories? Why can they only report hearsay instead of showcasing even a single backdoored motherboard found in the wild? Haven't these experts taken pictures? Can we see them? Can we have some details on the components used? This story is incredibly weird, nothing makes sense to me. I can't believe that Bloomberg would willingly report fake news. The fact that they decided to go with it even though Apple and others told them several time using unambiguous language that they were not aware of such an attack means that they must have really trusted their sources since they're effectively saying "Apple is blatantly lying about such a critical issue". But then why would Apple be lying like that? Seems like it can only backfire for them when the truth is eventually exposed. Nothing makes sense.
- djrogers 8y ago
  What context are you using the word ‘pin’ here? I’m having some trouble follow your comment as ‘pin’ doenst line up with anything I’d contextually assume you to be referring to.
chollida1 8y ago
Ironically, China might have done more to bring back manufacturing to the US than Trump and Obama combined. Regardless of if this is actually true, and so far that's debatable, clearly the rest of the world needs to retrieve ownership of their supply chains for mission critical computer systems. It's hard to see how this doesn't embolden Trump to take more action against China. and to be fair at this point, its hard to disagree with him that China is taking advantage of the rest of the world. Poor Super Micro. There time as a public company is probably measured in months now:( > Appleboum said that he's consulted with intelligence agencies outside the U.S. that have told him they've been tracking the manipulation of Supermicro hardware, and the hardware of other companies, for some time. I'd imagine at this point everyone who has the ability to do deep inspection of their hardware is working as fast as possible to resweep their servers.
- jackconnor 8y ago
  If Super Micro goes under and we discover this story was massively incorrect, can they sue for the lost market value? If so, this may cost bloomberg billions.
  - java-man 8y ago
    and if it is correct?
    - imglorp 8y ago
      If it's correct, it's highly likely that most cloud vendors are in the same boat. Imagine Google or AWS, who each have multiple millions of servers: even if they build their own motherboards, there are so many 3rd party components there's no way to vet all the boards. Their IDS will catch some, but not all. One might imagine a cloud vendor is constantly the target from multiple state actors, foreign and domestic, all vying for universal access.
      - cm2187 8y ago
        I can't imagine the cost of x-ray-ing all motherboards on an AWS scale.
annerajb 8y ago
This saga is fascinating I really have no doubt of the hardware existing. Thought the original picture from the article and description made it hard for me to imagine the connectivity. is it connected to the SPI of the BMC flash/OS storage? Why would software integrity checks like making sure the image is signed and not tampered wouldn't capture it? (Answer to this one sounds easy bad security practices regarding firmware process)
- RL_Quine 8y ago
  This article talks about something elsewhere entirely than the original one, and unrelated to the BMC.
  - annerajb 8y ago
    Yes I read it. seems they added another device to the network/ethernet interface which they detected sending network packets. Curious how this one affected the server or did compare to the original article. This one seems more benign considering it won't be able to mess around like the BMC has access to things like secure boot and other system busses like the PCI.
berbec 8y ago
OK so this is a different hack than Bloomberg reported before: ethernet jack piggyback instead of bmc. I'm not sure this adds credibility to the allegations in the other story. The details that Bloomberg related previously are so different that this couldnt be what they originally were reporting on. This adds to the China hacking server board narrative, but it does nothing to prove the Bloomberg reporting actually true. It does cast doubt on the denials made previously. It seems this story isn't totally smoke and mirrors as Apple, Amazon and Super Micro seem to want us to believe.
- resters 8y ago
  Read it more carefully. The ethernet jack is a tactic used by US intelligence years ago. That was mentioned in the story to explain the history of supply chain attacks.
  - cm2187 8y ago
    Do you think the NSA manufactured their implants in China?
    - Gokenstein 8y ago
      Plausible deniability. Wouldn't surprise me if China is getting the blame for a LOT of US hacking.
      - resters 8y ago
        I think it's totally plausible. See my other comment in this thread for my conspiracy theory :)
      - setquk 8y ago
        With the current political climate, that might be the intention. If you undermine international trade though marketing you don't have to fight a tariff war.
        8y ago
ulkesh 8y ago
So which company is the "major U.S. telecommunications company"?
deleted 8y ago
[deleted]
pjc50 8y ago
Finally a named source, but still no photos and the alleged hacked board is still not in the hands of a public security researcher. The "trojan ethernet connector" paragraph mentions similarity to an NSA implant, which appears to be this: https://en.wikipedia.org/wiki/NSA_ANT_catalog#/media/File:NSA_FIREWALK.jpg https://en.wikipedia.org/wiki/NSA_ANT_catalog#/media/File:NS... I'm now wondering if someone found an NSA implant and misreported it as Chinese. We're going to end up in the stupid situation where people are afraid to report foreign intelligence attacks because it's illegal to report an attack by US intelligence agencies, aren't we?
- xxpor 8y ago
  >it's illegal to report an attack by US intelligence agencies Could you expand more on that?
  - kibwen 8y ago
    IANAL, but New York Times Co. v. United States is a famous precedent for the first amendment protecting the press' right to publish classified government documents.
    - xxpor 8y ago
      That's what I was kind of getting at, the press is free to publish pretty much anything they'd like AFAIK.
- bybyme 8y ago
  ~
  - throwawaymath 8y ago
    Just about anyone on Google's Project Zero team, off the top of my head. They would probably be both competent and enthusiastic. Beyond that Apple, Microsoft, Tesla and probably Amazon have sufficiently capable public (or past public) researchers working for them. But I doubt many would want the publicity one way or the other.
krn 8y ago
I find it hard to believe, that Bloomberg would publish an extremely detailed story involving some of the largest public companies in the world, knowing that it is entirely false. The cost of reputation is just too high.
- dkonofalski 8y ago
  Isn't that the question, though. I don't think anyone is accusing them of publishing information that they know to be false. It seems more likely that their sources are just not as good as believed and possibly have reinforced the details and information through an echo chamber. If these sources are within the same subsection of the industry and regularly exchange information, it may just be that they've been regurgitating information between themselves that somehow coalesced into a "real" story.
  - pishpash 8y ago
    Not to be tin-foily, but it is also possible that Bloomberg has been deliberately fed or seeded with "bad" information -- it doesn't have to be an organic echo chamber. But no, I don't think Bloomberg itself set out to deliberately make a false story.
    - dkonofalski 8y ago
      Of course it's possible. Foreign states have fed much less believable information to work against US interests and a good chunk of the population believes it wholesale.
- late2part 8y ago
  They don't have to know it is false. They just have to find their sources credible.
- WillPostForFood 8y ago
  You can read this story as a CYA because they overstepped in the first story. The have good details here. But in the Apple rebuttal, they note Bloomberg only had a single source for the Apple claims.
rectang 8y ago
It may soon be that the only companies who can sell hardware outside of their own country are those who sell Open Source Hardware which can be 100% verified as true to its published design.
- 394549 8y ago
  > It may soon be that the only companies who can sell hardware outside of their own country are those who sell Open Source Hardware which can be 100% verified as true to its published design. That would be a very, very good outcome, IMHO. Use espionage fears to push forward other objectives, like open source.
- ars 8y ago
  How would that help? SuperMicro was not able to verify that what they designed and ordered is what was shipped. What makes you think that just because it's open source, that that would change? i.e. the design is "Open Source" to SuperMicro - but it didn't help them.
buserror 8y ago
That story is a bit odd, still -- normally behind the connector there is optionally magnetics, and at least a PHY... being able to integrate the magnetics in the connector exists allright, but adding the phy /as well/ must make it a marvel of integration regular manufacturers would dream of... especially at Gb speed! Also, you can't really 'piggyback' ethernet easily, for the same reasons; you would need TWO phy in there to decode/reencode... Even if you'd want to 'piggyback' on the link itself, it would be very, very difficult to say the least -- Gb ethernet is definitely not a gimme to synthesise, let alone piggyback. So, color me dubious -- the SPI 'chip' of last week was a but dubious but doable (given not just a custom chip, but a custom PCB) but this ethernet story makes even less sense!
- jstanley 8y ago
  Elsewhere in this thread there is an actual link to an actual NSA device that does exactly this. I don't think it's in the realm of science fiction.
  - buserror 8y ago
    If you look at that illustration, you see that it's not just one ethernet connector, it's one of these massive connector stack with one ethernet and 2 USB, also, it adds quite a bit of depth to the connector; it must have been made with one particular brand/type of motherboard in mind. Still, if these are in the wild, then perhaps our chinese friends might have reduced the footprint even more to the size of one connector. I know the connectors with integrated magnetics are quite a bit 'longer' and 'beefier' than the passive ones.
    - monocasa 8y ago
      That's also at least a five year old implant since Snowden leaked it back in 2013.
      - kulahan 8y ago
        The wikipedia article on the device shows a date of 2008, so yeah, it's probably pretty outdated as far as spy tech is concerned.
okket 8y ago
So it is a compromised ethernet adapter. Nobody question the ability of Chinese spies to plant such a thing, but the "Big Hack" story implies that this is used as a mass infiltration tool, which I still find very improbable and lacks any evidence.
- cm2187 8y ago
  Did they say "mass"? I thought it was a handful or large companies.
- gargravarr 8y ago
  The way I interpret it is: since it's being introduced at the manufacturing plant, those installing the devices have no idea where the finished product will end up. Thus, these are not targetted attacks; they could wind up in the servers of a Fortune 500 company, or just as easily in some hobbyist's home lab. You'd need to compromise a large percentage of the products to increase the chances of landing a juicy target. The scattergun manner is what they're playing up here.
  - saudioger 8y ago
    Considering that large scale internet companies like Amazon and Apple buy enormous amounts of hardware, it's not an insane strategy. If you compromise 1 of every 1,000 pieces of hardware (or even one of every 10,000) odds are you'll end up in a major datacenter pretty quickly.
carapace 8y ago
So does it really matter if it's true or not? Isn't the sane response to assume it's all true and verify your system integrity? Even if you find nothing you would then have a pile of strong evidence for the null case, eh?
- RL_Quine 8y ago
  You've got to be kidding. There's no way of validating anything about modern hardware, it's packed with independent systems running various firmware, software that's decades old, parts nobody can even identify unless you're the OEM. You can get to "that probably does this" level easily, but that doesn't tell you anything about its actual security or authenticity.
  - andai 8y ago
    Is this a desirable state of affairs?
    - RL_Quine 8y ago
      No, but it seems to be inescapable.
  - carapace 8y ago
    Funny you should say that... Guess what I'm working on, go on, guess. :-D And don't call me kidding. ;-P
- platinumrad 8y ago
  >So does it really matter if it's true or not? Is this Bloomberg's new slogan?
mmastrac 8y ago
It seems like there are two possibilities to me: 1) Bloomberg has a number of sources that are mistaken/misinformed, but this is not necessarily a made-up story, or 2) Bloomberg is nearly correct (minus some technical details) but the US government is forcing these companies to respond as if the story is wrong - possibly because of diplomatic reasons. What is the likelihood that #2 is correct? (there are other alternatives, but I believe that the likelihood that this is 100% or at least majorly fabricated by Bloomberg is near enough to zero)
- roblabla 8y ago
  The US cannot force those companies to lie. They can force them to stay silent, in which case they'd just say "No comments". If those companies are lying, they are committing security fraud.
  - 394549 8y ago
    > The US cannot force those companies to lie. They can force them to stay silent, in which case they'd just say "No comments". If those companies are lying, they are committing security fraud. Would the US government have to force these companies to lie? It's quite possible that the denials were the result of voluntary cooperation.
    - macintux 8y ago
      A public company issuing such strongly-worded denials that turn out to be untrue would be leaving themselves at risk of an investigation by the SEC and/or a shareholder lawsuit.
      - ggggtez 8y ago
        It would be pretty extraordinary for the government to sue a company for cooperating with the government. A company has to follow court orders. The government would have to sue itself.
dwighttk 8y ago
Bloomberg really has it out for Supermicro.
- printA 8y ago
  Or an article author wanted to move the market. https://news.ycombinator.com/item?id=18162440 https://news.ycombinator.com/item?id=18162440
- makomk 8y ago
  Apparently, even their sole source for this article isn't happy with how they have it out for Supermicro: https://www.servethehome.com/yossi-appleboum-disagrees-bloomberg-is-positioning-his-research-against-supermicro/ https://www.servethehome.com/yossi-appleboum-disagrees-bloom...
amatecha 8y ago
I wonder what evidence caused them to claim the embedded "bug" is planted by China or its operatives -- how can they prove the source of the unauthorized modification? Are they just assuming?
- kickopotomus 8y ago
  This is my question as well. > Based on his inspection of the device, Appleboum determined that the telecom company's server was modified at the factory where it was manufactured. Any further evidence to support that claim? How can they be so sure that the boards weren't tampered with after they were manufactured and shipped to the states? Anyone with a little soldering experience could easily replace an ethernet port.
ipunchghosts 8y ago
Why should I believe any of this? No proof!
dawhizkid 8y ago
If Bloomberg's reporting is/was, in fact, entirely false then why wouldn't Amazon/Apple/Supermicro immediately sue them for libel/defamation?
- educationdata 8y ago
  In the U.S., there is a very high bar to accuse defamation. You not only need to prove the report is false, you also need to prove the reporter knew it was false and had bad intention to cause damage.
  - Karunamon 8y ago
    That's defamation, but not libel. For clearing libel, all you have to prove is that the statement was presented as fact, was false, and caused harm. From my lay understanding, Supermicro has a slam dunk case given the impact on their stock price.
    - nkurz 8y ago
      For clearing libel, all you have to prove is that the statement was presented as fact, was false, and caused harm. At least in the US, I don't think this is true. Defamation is a broader class that includes libel, and always requires proving that the publisher was at least "negligent" in publishing the false statement: https://www.law.cornell.edu/wex/defamation https://www.law.cornell.edu/wex/defamation
    - roywiggins 8y ago
      You either have to know the fact is false, or act with reckless disregard as to whether it's false or not. 17 sources suggests that they did care whether it was true; if they didn't care, they would not have gone to all the effort.
jaclaz 8y ago
Set aside whether this report is reliable or not, I don't see how something like the Digi Connect ME: https://www.digi.com/products/embedded-systems/system-on-modules/digiconnectme https://www.digi.com/products/embedded-systems/system-on-mod... or the 9210 version: https://www.digi.com/products/embedded-systems/system-on-modules/digiconnectme9210 https://www.digi.com/products/embedded-systems/system-on-mod... cannot exist in Gb speed, if I recall correctly the original Digi Connect came out in 2005 or so.
deleted 8y ago
[deleted]
holstvoogd 8y ago
i call bs. again. one source, to many technical inaccuracies imo
detaro 8y ago
Could be due to losses in "translation", but this paragraph seems odd: > Three security experts who have analyzed foreign hardware implants for the U.S. Department of Defense confirmed that the way Sepio's software detected the implant is sound. One of the few ways to identify suspicious hardware is by looking at the lowest levels of network traffic. Those include not only normal network transmissions, but also analog signals -- such as power consumption -- that can indicate the presence of a covert piece of hardware. Ideas what they mean? I'd be surprised if mainboards etc exposed power consumption of random periphery parts. Maybe sleep/link states of the actual NIC that get influenced?
- wyldfire 8y ago
  If you look at the activity in the frequency domain, you might find a clock that's present in adulterated hardware that's missing in the nominal hardware. > exposed power consumption of random periphery parts Sampling these analog signals is done outside of the computer itself. Even if it were exposed via i2c, you couldn't trust anything that it would tell you.
  - detaro 8y ago
    To me the article reads like it's a purely software solution, thus my confusion, but that might be misinterpretation.
    - makomk 8y ago
      I think it's probably also a software solution that the company whose CEO was the sole source for this article is trying to promote, judging from this marketing brochure for them: https://twitter.com/securelyfitz/status/1049725014075830272 https://twitter.com/securelyfitz/status/1049725014075830272 The claims in that certainly look every part as improbable.
- gargravarr 8y ago
  If both are going over the same physical interface, there could be tiny voltage drops to power the chip itself. The vague-as-ever article gives no indication as to how the chip would be powered, but such a small implant could conceivably leech off the legitimate ethernet interface.
smaili 8y ago
Supermicro shareholders just can't catch a break. Down yet another 25% today.
platinumrad 8y ago
Seeing as they screwed up a bunch of the details yet again[1], I'm still skeptical. 1. https://mobile.twitter.com/marcan42/status/1049690480806584325 https://mobile.twitter.com/marcan42/status/10496904808065843...
creeble 8y ago
Picture or it didn't happen. What is up with Bloomberg?
- platinumrad 8y ago
  And please no pictures of generic couplers marked in small text as "illustrations" like last time.
gargravarr 8y ago
Some real doublethink going on here, with Bloomberg continuing to insist that supply chain attacks are real, yet seemingly accepting the denials of each company involved. I am really not sure what to make of this.
- passwordreset 8y ago
  This is just a suggestion, but it might be useful to stop thinking about news reports as "the news report says X is real and true", because that's not what most news reports actually say. The text used usually reads like: "An unnamed source says blah blah", and "Joe Smith, a retied auto-worker, says blah blah blah", or "In response, a spokesman for Large Company, Inc. says blah blah blah blah", and so forth. Bloomberg (probably) isn't insisting that the story or the denial is true or false. They repeat the facts that other people have said. Often those facts are true; often those facts are false. Thinking about it this way helps me make sense of the topic. Maybe I'm weird.
  - kickopotomus 8y ago
    It is not that cut and dry though. By publishing this story, they are asserting that the hack happened. They are not saying the attack is possible. They are saying it happened and that halved SMCI's market value. If they are wrong, Bloomberg is going to pay through the teeth for this one.
na85 8y ago
What's interesting to me in all this is not the immediate response but what the medium- to long-term impact will be. Does anyone doubt that semiconductor/electronics manufacturing will become a strategic industry in the same realm as uranium refining? I think we'll see electronics supply chain integrity play a growing role in major project delivery in the years to come.
tomc1985 8y ago
Are there any other sources with these allegations or is it just Bloomberg?
smurfysmurf 8y ago
I've seen several comments regarding whether or not Apple, Amazon etc. would deny the hacking if its true and if that is fraud or not. I work at Amazon now and previously was in the Navy, holding a TS/SCI. My firm belief is if such a hack happened, it would not be disclosed to anyone without a clearance, and the organizations that are denying it have no knowledge that it occurred. Furthermore if there truly was a compromise by a foreign nation it would be classified as a national security threat and subsequently classified and kept from public knowledge. Anyone who disclosed the truth would be at risk of loosing their clearance, job, and could end up getting the snowden treatment.
- wil421 8y ago
  I can see where the Navy/Military/Government could compartmentalize a hack like this. How could a company like Apple or Amazon keep this under wraps? How could they keep the knowledge of such a hack within the TS/SCI employees?
  - smurfysmurf 8y ago
    The cleared department is handled the same way as in the military in terms of security. Amazon has SCIF's etc. So unless a disgruntled employee steps forward who doesn't care about there life, I imagine its easily contained (and symptoms of an employee being disgruntled are highly monitored when they hold a clearance)
    - wil421 8y ago
      I’m thinking about the non cleared data center folk, the sys admins and developers who use the servers for their applications. How do a bunch of Supermicro servers vanish wintout anyone noticing? I’d expect quite a few people would be involved that do not have any clearances. Apple is known for their secrecy but a few other companies named are not.
      - darawk 8y ago
        Maybe they didn't remove them.
noisy_boy 8y ago
This gets me thinking - what if this issue (assuming this is genuine and I'm starting to believe it is), is not limited to server hardware and more pervasive? What if this affects mobile phones as well? I'm looking suspiciously at my Nexus 6p made by Huawei.
kccqzy 8y ago
Might be off topic, but can Bloomberg even consistently use punctuation? I see mixes of dumb quotes and smart quotes in the same paragraph and the same sentence. And the backtick as well. It is those details that reveal to me the article is rushed, possibly very little care has been afforded to the article.
dig1 8y ago
Accidentally, all of this starts happening when trade war between USA and China is raging and when some countries decided to not follow USA orders to reduce business with China...
gepeto42 8y ago
Define "evidence".
thro_a_way 8y ago
why is YCombinator China a thing? I hope the firm gets brought before the Senate to answer for this.
virtualadmin 8y ago
This is not the first nor the last time we see reports about hardware manufactured in China being suspect of tampering like this. It will continue to happen. China steals technology and then manufacturers it for itself.
echevil 8y ago
Oh, it's Bloomberg again...
norseman 8y ago
This smells like a Snowden/Wikileaks strategy. Publish the first article expecting the major players to deny and discredit, but then publish more info after the fact to not only reveal the spying but trap all the big players in their lies.
kyrieeschaton 8y ago
Is no one going to mention that the named source is ex-Israeli Unit 8200, whose alumni themselves have a long record of espionage in the US telecom sector?
- contingencies 8y ago
  Questions to ask when making up your own mind on this issue: AMDOCS billing customer list includes how many major US telcos? How much customer data is required to generate a bill? How much billing is executed in real or near-real time? How much is hosted off-site on non telco infrastructure? How many non-billing services? (Someone elsewhere here already mentioned 100% of voicemail is now outsourced to AMDOCS at one telco). Incidentally, the source for this article is also conveniently based in Maryland.
deleted 8y ago
[deleted]
ectospheno 8y ago
Wake me up when a news organization other than bloomberg will go on record as saying the story has merit.
jackpirate 8y ago
What if the bloomberg article was a sort of false flag? The Trump administration has been consistently escalating retoric against the Chinese, and it's not hard to imagine the CIA/NSA/etc intentionally leaking facts to bloomberg that would make China look like a national security threat. This could even be done in a way where the security agencies don't leak anything actually false, but let the non-phds at bloomberg run wild with speculation to create a sensationalist story that's not really true. A recent WSJ article [1] has called particular attention to the Trump administration's escalating anti-Chinese rhetoric, calling it the start of a "second cold war". We know for a fact that these sorts of operations happened during the first cold war [2], so it's not at all hard to imagine they would happen now. A false flag attack fits with all the information we have so far about the event: There's no direct evidence of the attack, and if the bloomberg article is ever proven to be false, then only a small number of security researchers (and HNers) will ever learn about the retraction. The vast majority of Americans will only remember reading about how "China hacked major US companies" and create an anti-Chinese atmosphere that will help fuel future anti-Chinese policies. [1] https://www.wsj.com/articles/mike-pence-announces-cold-war-ii-1539039480 https://www.wsj.com/articles/mike-pence-announces-cold-war-i... [2] https://fas.org/sgp/news/2002/02/re022502.html https://fas.org/sgp/news/2002/02/re022502.html
4684499 8y ago
> The security expert, Yossi Appleboum, provided documents, analysis and other evidence > ... said Appleboum, who accompanied them for a visual inspection of the machine. So can we at least get some photos of tampered hardware from these "evidence"?
apo 8y ago
Appleboum said one key sign of the implant is that the manipulated Ethernet connector has metal sides instead of the usual plastic ones. The metal is necessary to diffuse heat from the chip hidden inside, which acts like a mini computer. "The module looks really innocent, high quality and 'original' but it was added as part of a supply chain attack," he said. How uncommon are metal vs plastic ethernet connector sleeves? This seems like a clue that even non-experts could use to track down an implant.
- marshray 8y ago
  I don't recall seeing a board-mounted Ethernet jack that didn't have metal sides. Reasons for an Ethernet jack to include a small metal enclosure: 1. The jack is subject to physical strain as we insert, remove, and tug on the cables. Most components are affixed to the board using only solder on the signal leads, but these do not provide significant mechanical strength. If you've ever had a bad headphone jack on personal electronics you are familiar with this phenomenon. So the little metal box has additional metal tabs that fit snugly in holes or slots on the board, and provide a stronger mechanical fit and a much larger surface area for solder adhesion. 2. Electromagnetic interference compliance. While Ethernet by definition involves pumping gigahertz signals out a long wire, these signals are carefully shaped and the cables pairs twisted to reduce leakage. But the designer of the jack doesn't know how much EM is flying around inside the case into which the port will be fitted. A metal box around the jack minimizes the size of the unshielded opening in the case. If you've installed a PC motherboard, you know the springy metal fingers on the backplate that seal against the block of external ports. Is it possible the article is talking about an Ethernet cable plug? I have occasionally seen those with metal sides. But they are not normally supplied as part of a motherboard or server.
Balvarez 8y ago
I'm not sure it matters to me whether the original article is 100% correct. What I think the original article points out correctly is that the Chinese supply chain is possibly a pretty easy vector for hardware based hacks. I would suspect most nation states have the pull to bribe/blackmail contractors to make malicious modifications. Though the chinese gov. would be the most likely culprit. It also seems plausible the NSA would prefer techies not to look too closely at their hardware *removes tinfoil hat
narrator 8y ago
And to think a couple of weeks ago I was called a lunatic for saying that backdoors are purposefully put into hardware by government actors: https://news.ycombinator.com/item?id=17736721 https://news.ycombinator.com/item?id=17736721
sunstone 8y ago
This story is so weird as Bloomberg doubles down with their reputation. What if the boards were compromised, but by the USA rather than China?
aranw 8y ago
I've been wondering if Google and Google Cloud is affected by this? Does anyone know who Google uses to build the custom motherboards that they use? Also how does something like the Titan Chip [0] help protect against such attack? I'm curious to understand if theres anything that can protect against this? [0] https://cloud.google.com/blog/products/gcp/titan-in-depth-security-in-plaintext https://cloud.google.com/blog/products/gcp/titan-in-depth-se...
fipple 8y ago
If anything, the fog of war here shows the costs that have been incurred alongside the massive efficiency gains from outsourcing/subcontracting in the global economy.
jhallenworld 8y ago
So I have bought Supermicro motherboards in quantity in the past. We went through a distributor and shipping was not direct from China. So I'm having doubts that Supermicro's Chinese CM has any idea who the end customer of any of their boards is. Maybe it's different for really huge customer, but I still have doubts- it implies they are building to order and have no inventory. This implies that the CM installs the Chinese Spy Chip on every board, or a random sampling of boards (so they should show up in the wild) or the tampering happens later in the supply chain- in the US. Also I have further doubts about embedding a chip in a middle layer of a PCB. I doubt the CM is going to add that much extra expense when it's easier to add a chip to the surface. You don't really buy much secrecy from embedding a chip given that it's easy to x-ray a board, so why bother? Anyway, there are devices called optical comparators. Supermicro could buy some security by providing comparison images that allow customers to perform an incoming inspection. I'm thinking they should do this now to add assurance / help their stock price. Here is an optical one: https://www.visionxinc.com/digital-optical-comparators/pcb https://www.visionxinc.com/digital-optical-comparators/pcb Here is an X-Ray one: http://www.glenbrooktech.com/multi-layer-pcb.php http://www.glenbrooktech.com/multi-layer-pcb.php Maybe the X-Ray inspection could pick up the Ethernet connector attack.
jakobegger 8y ago
This story is getting more incredible every day. Bloomberg only has second hand sources, and all the exploit details are based on speculation from security researchers -- not from insiders. It looks like Bloomberg heard several rumors of supply chain manipulations, mixed that up with plausible scenarios thought up by security researchers, added a few photos from random electronic parts, and voila you have a compelling story to tell. This "new evidence" talks about a completely different type of attack than the original article. It corroborates nothing. It just shows how misleading the original story was. I think the most damning part was the use of so many misleading photos and illustrations. All photos were pure speculation (this is what this chip might look like, this is where it would make sense to put the chip). But neither the captions nor the text made that clear. The only thing I believe about the story is that they have a couple of sources who have vague, second hand rumors about supply chain manipulation.
- Havoc 8y ago
  Yup. Which is why I'm holding on to my discount bin supermicro shares. The bloomberg story stinks to high heavens even if there are live examples of infiltrated supply chains
- Symmetry 8y ago
  The reporters in question apparently have a reputation for credulously repeating things they hear about cyber-attacks. https://twitter.com/RobertMLee/status/1049617855396933632 https://twitter.com/RobertMLee/status/1049617855396933632
mmaunder 8y ago
The credibility of the journalists is being called into question by influential people in the infosec community: https://twitter.com/RobertMLee/status/1049617855396933632?s=19 https://twitter.com/RobertMLee/status/1049617855396933632?s=... https://risky.biz/RB517_feature/ https://risky.biz/RB517_feature/ The podcast is an interview with Joe Fitzpatrick, one of the named sources. It's an impartial and mature discussion that left me with the same feeling of unease that Joe says he had when he first read the story. Basically, Joe would describe a theoretical hardware exploit to the Bloomberg journalist and the journo's sources would then confirm exactly that as a real world exploit.
jgamman 8y ago
tl;dr US shocked (shocked i say!) that other countries doing the same sorts of things they do.
mrb 8y ago
It's interesting to note that the organization supposedly at the center of the investigation of this tampering of Supermicro servers—the FBI—has never issued a statement on the veracity of the story. Strange silence...
directusss 8y ago
Damn... sounds like Bloomberg is really hating China, even it's driving directly into the law-suit hell.
browsercoin 8y ago
maybe its just me but this seems all too familiar. we've seen it happen with iraq, reputable media outputs sensational hostile state actions, immediately faces criticism, but more importantly the timing after announcing China tried meddling with elections. i dont know sometimes i like reading into the big picture, what is the purpose of this bloomberg article for those outside HN? it won't invoke feelings of calm but rather moral panic. now half the americans are riled up to think china is attacking usa, and we remember they need just teh right amount of support. also as the noose is tightening around trump, a war or a limited military conflict would be the perfect distraction. /end conspiracy
a3n 8y ago
Maybe this was an NSA operation, but they outsourced the hardware to China, because they are, you know, cheap.
beamatronic 8y ago
Who profits from this story?
mohammedbin 8y ago
I want to mention a. political side of the argument- M Bloomberg himself is very pro open trade and has strongly hinted at running in 2020. Also, newspapers don't usually hurt your owners' candidacy, even for explosive stories. Bloomberg isn't just putting it's reputation on stake here, it's legitimized Trump presidency further. I would say there is definitely something behind this story.
kerng 8y ago
Even if Bloomberg would be incorrect, the good thing will be that everyone starts looking now and in the end it might become a self fulfilling prophecy. It's unlikely that something like this wasn't already done. Scale and targets could differ though
OliverJones 8y ago
Looks like the infosec cold war is in full swing. All secrets eventually leak. All secrets. No exceptions. Not even nation-state players with unlimited budgets can prevent leaks. This is reality. What's a hacker (in the original sense: programmer, not cybercreep) to do? Our task as custodians of secrets for our end-users is to reduce the attack surfaces on our systems, slow down those leaks and mitigate the effects of leaks when they happen. We must do these things to the best of our ability. We must do them whether we rig systems for large organizations or for our grandmothers. Who's trustworthy? Apple? Probably. They're pushing security as a major component of their brand. AWS? Possibly. They have a lot to lose if they're compromised. Microsoft? Possibly. They too have a lot to lose. Seventeen well-placed but unnamed sources in the US security apparatus, babbling to journalists? Possibly. Journalists? Their trustworthiness is eroding. Cryptographers Whit Diffie, Martin Hellman, Ralph Merkle, Bruce Scheier, Ron Rivest and colleagues? Likely. Motherboard vendors? Probably not. Router / switch / firewall vendors? Probably not. Nation-states? No. (They could change this by abolishing "security by obscurity" in their work, but that would require major changes in mindset.)
HillaryBriss 8y ago
my guess is that the NSA, spy agencies from China, Bloomberg, SuperMicro, Apple, Amazon and anyone else involved are carefully watching HN comments to see who interprets the story correctly so they can seek that person out and hire
prirun 8y ago
The article says: "The manipulation of the Ethernet connector appeared to be similar to a method also used by the U.S. National Security Agency, details of which were leaked in 2013." So the US doesn't exactly have clean hands. Here's a theory: 1. The US wants to have more ability to spy 2. With most manufacturing in China, this is hard 3. Implicating Supermicro might cause their customers to switch suppliers. It has already caused Supermicro stock to go down 41% then 27% again. 4. The US / Sepio Systems / former Mossad / former CIA officials come to the rescue with a company manufacturing "secure servers" not made in China. Or more likely, still made in China, only under different control. 5. Result: profit, increase US ability to spy, "bring manufacturing back to US companies" IMO, governments - all governments - have no ethics and will do whatever they want to further their agenda.

875 points by gorbachev 8y ago

They don’t say what the hack was and definitely do not say it was one of their pins. Probably some truth here, but as hard as they try, does not seem supportive of their “chinese pin” theory. Very suspicious that this is related, I’m guessing they’re trying to do anything to cover their asses.

annerajb 8y ago
What is the chinese pin theory? Is this the name being used for the supposedly embedded on the PCB layers chip? Or the one that attaches on unpopulated pins/pads near the BMC memory area?
trevyn 8y ago
Sounds like the Ethernet connector module was not from the, ahem, correct manufacturer: “Appleboum said one key sign of the implant is that the manipulated Ethernet connector has metal sides instead of the usual plastic ones. The metal is necessary to diffuse heat from the chip hidden inside, which acts like a mini computer. "The module looks really innocent, high quality and 'original' but it was added as part of a supply chain attack," he said.”
- RL_Quine 8y ago
  I'm not sure I believe this one as much, just based on the part you quoted. I can see a chip manipulating the BMC/IPMI flash to make it do things it shouldn't. I don't see how an ethernet port could be modified to be interesting. They're typically after the magnetics, or contain the magnetics themselves, so the only source of power would be the activity LEDs, or something, or maybe we assume a custom PCB as well. You've then also got to have it doing gigabit ethernet, or otherwise tampering with data it got from that interface, which feels unlikely. Maybe it's just the same as the last implant story, hidden in a less easy to find place? Hard to know without something even approaching technical information.
  - simias 8y ago
    You could easily DoS obviously, but beyond that I agree that it seems tricky to do anything worthwhile.
    - moftz 8y ago
      It could just be a sort of beacon to help identify where hardware went after the manufacturing process. If the same company is building the same hardware, the agent can slip in something more nefarious to make sure they target the right company. Servers are commodity products but they aren't manufactured in mass quantities like phones are. If a company orders thousands of them, that's likely thousands that will need to be made. A chinese manufacturing plant gets contracted to spin up production and an implant is slipped into some of the first boards just to see where they go. You don't want an expensive hardware trojan to end up in a Fortnite server; you want to hit Apple, Google, Lockheed Martin, Spacex, anyone with valuable IP or information. The more beacon implants you throw out there, the more likely someone will find one and you don't want to get caught too early in the game. Once those implants come online and phone home, you have a better idea where the remaining boards are going and slip in the real deal implants, the ones that will actually get you a backdoor.
      - simias 8y ago
        How would such a beacon work though? As RL_Quine points out there's only so much you can do at this point, especially if you want to be super stealthy. If you wanted to send a ping to an external server you'd have to craft an ethernet frame with the right target MAC address containing an IP datagram with the right IP address to be routed correctly in the datacenter and through the public firewall. You better make sure that your packet looks legit otherwise you're sure to trip anything looking for suspicious activity. "Hey look, our servers send weird packets to this suspicious IP, what gives?" And you have to do all that with a very low power device running from within the port itself. Seems like a very high bar to me, especially when there seems to be so many easier ways to backdoor a motherboard. But maybe the component is only hosted in the ethernet port but is actually connected to other signals on the motherboard.
- makomk 8y ago
  I've never seen an onboard Ethernet jack that doesn't have metal sides. The only places I've seen all-plastic Ethernet jacks are consumer networking gear and really ancient add-on cards. That makes me wonder if their source actually knows what he's talking about, especially given the lack of technical details about how this works.
maxden 8y ago
This seems different than the extra chip attack according to the article. "subsequent physical inspection revealed an implant built into the server’s Ethernet connector"
- close04 8y ago
  This feels like piling some more to cover up the weak premise of the previous article. It doesn't mean it's false, it just means Bloomberg already cried wolf and couldn't show it. At this point it feels more like taking shots at Supermicro than professional reporting.
  - late2part 8y ago
    totally seems like this is the case to me.
simias 8y ago
That's my take as well. They quote many "experts" and report many incidents but they seem hard pressed to come with details and pictures. I can believe that these attacks occur but are they really at the scale implied by these stories? Why can they only report hearsay instead of showcasing even a single backdoored motherboard found in the wild? Haven't these experts taken pictures? Can we see them? Can we have some details on the components used? This story is incredibly weird, nothing makes sense to me. I can't believe that Bloomberg would willingly report fake news. The fact that they decided to go with it even though Apple and others told them several time using unambiguous language that they were not aware of such an attack means that they must have really trusted their sources since they're effectively saying "Apple is blatantly lying about such a critical issue". But then why would Apple be lying like that? Seems like it can only backfire for them when the truth is eventually exposed. Nothing makes sense.
djrogers 8y ago
What context are you using the word ‘pin’ here? I’m having some trouble follow your comment as ‘pin’ doenst line up with anything I’d contextually assume you to be referring to.

chollida1 8y ago

Ironically, China might have done more to bring back manufacturing to the US than Trump and Obama combined. Regardless of if this is actually true, and so far that's debatable, clearly the rest of the world needs to retrieve ownership of their supply chains for mission critical computer systems. It's hard to see how this doesn't embolden Trump to take more action against China. and to be fair at this point, its hard to disagree with him that China is taking advantage of the rest of the world. Poor Super Micro. There time as a public company is probably measured in months now:( > Appleboum said that he's consulted with intelligence agencies outside the U.S. that have told him they've been tracking the manipulation of Supermicro hardware, and the hardware of other companies, for some time. I'd imagine at this point everyone who has the ability to do deep inspection of their hardware is working as fast as possible to resweep their servers.

jackconnor 8y ago
If Super Micro goes under and we discover this story was massively incorrect, can they sue for the lost market value? If so, this may cost bloomberg billions.
- java-man 8y ago
  and if it is correct?
  - imglorp 8y ago
    If it's correct, it's highly likely that most cloud vendors are in the same boat. Imagine Google or AWS, who each have multiple millions of servers: even if they build their own motherboards, there are so many 3rd party components there's no way to vet all the boards. Their IDS will catch some, but not all. One might imagine a cloud vendor is constantly the target from multiple state actors, foreign and domestic, all vying for universal access.
    - cm2187 8y ago
      I can't imagine the cost of x-ray-ing all motherboards on an AWS scale.

annerajb 8y ago

This saga is fascinating I really have no doubt of the hardware existing. Thought the original picture from the article and description made it hard for me to imagine the connectivity. is it connected to the SPI of the BMC flash/OS storage? Why would software integrity checks like making sure the image is signed and not tampered wouldn't capture it? (Answer to this one sounds easy bad security practices regarding firmware process)

RL_Quine 8y ago
This article talks about something elsewhere entirely than the original one, and unrelated to the BMC.
- annerajb 8y ago
  Yes I read it. seems they added another device to the network/ethernet interface which they detected sending network packets. Curious how this one affected the server or did compare to the original article. This one seems more benign considering it won't be able to mess around like the BMC has access to things like secure boot and other system busses like the PCI.

berbec 8y ago

OK so this is a different hack than Bloomberg reported before: ethernet jack piggyback instead of bmc. I'm not sure this adds credibility to the allegations in the other story. The details that Bloomberg related previously are so different that this couldnt be what they originally were reporting on. This adds to the China hacking server board narrative, but it does nothing to prove the Bloomberg reporting actually true. It does cast doubt on the denials made previously. It seems this story isn't totally smoke and mirrors as Apple, Amazon and Super Micro seem to want us to believe.

resters 8y ago
Read it more carefully. The ethernet jack is a tactic used by US intelligence years ago. That was mentioned in the story to explain the history of supply chain attacks.
- cm2187 8y ago
  Do you think the NSA manufactured their implants in China?
  - Gokenstein 8y ago
    Plausible deniability. Wouldn't surprise me if China is getting the blame for a LOT of US hacking.
    - resters 8y ago
      I think it's totally plausible. See my other comment in this thread for my conspiracy theory :)
    - setquk 8y ago
      With the current political climate, that might be the intention. If you undermine international trade though marketing you don't have to fight a tariff war.
      - 8y ago

ulkesh 8y ago

So which company is the "major U.S. telecommunications company"?

deleted 8y ago

[deleted]

pjc50 8y ago

Finally a named source, but still no photos and the alleged hacked board is still not in the hands of a public security researcher. The "trojan ethernet connector" paragraph mentions similarity to an NSA implant, which appears to be this: https://en.wikipedia.org/wiki/NSA_ANT_catalog#/media/File:NSA_FIREWALK.jpg https://en.wikipedia.org/wiki/NSA_ANT_catalog#/media/File:NS... I'm now wondering if someone found an NSA implant and misreported it as Chinese. We're going to end up in the stupid situation where people are afraid to report foreign intelligence attacks because it's illegal to report an attack by US intelligence agencies, aren't we?

xxpor 8y ago
>it's illegal to report an attack by US intelligence agencies Could you expand more on that?
- kibwen 8y ago
  IANAL, but New York Times Co. v. United States is a famous precedent for the first amendment protecting the press' right to publish classified government documents.
  - xxpor 8y ago
    That's what I was kind of getting at, the press is free to publish pretty much anything they'd like AFAIK.
bybyme 8y ago
~
- throwawaymath 8y ago
  Just about anyone on Google's Project Zero team, off the top of my head. They would probably be both competent and enthusiastic. Beyond that Apple, Microsoft, Tesla and probably Amazon have sufficiently capable public (or past public) researchers working for them. But I doubt many would want the publicity one way or the other.

krn 8y ago

I find it hard to believe, that Bloomberg would publish an extremely detailed story involving some of the largest public companies in the world, knowing that it is entirely false. The cost of reputation is just too high.

dkonofalski 8y ago
Isn't that the question, though. I don't think anyone is accusing them of publishing information that they know to be false. It seems more likely that their sources are just not as good as believed and possibly have reinforced the details and information through an echo chamber. If these sources are within the same subsection of the industry and regularly exchange information, it may just be that they've been regurgitating information between themselves that somehow coalesced into a "real" story.
- pishpash 8y ago
  Not to be tin-foily, but it is also possible that Bloomberg has been deliberately fed or seeded with "bad" information -- it doesn't have to be an organic echo chamber. But no, I don't think Bloomberg itself set out to deliberately make a false story.
  - dkonofalski 8y ago
    Of course it's possible. Foreign states have fed much less believable information to work against US interests and a good chunk of the population believes it wholesale.
late2part 8y ago
They don't have to know it is false. They just have to find their sources credible.
WillPostForFood 8y ago
You can read this story as a CYA because they overstepped in the first story. The have good details here. But in the Apple rebuttal, they note Bloomberg only had a single source for the Apple claims.

rectang 8y ago

It may soon be that the only companies who can sell hardware outside of their own country are those who sell Open Source Hardware which can be 100% verified as true to its published design.

394549 8y ago
> It may soon be that the only companies who can sell hardware outside of their own country are those who sell Open Source Hardware which can be 100% verified as true to its published design. That would be a very, very good outcome, IMHO. Use espionage fears to push forward other objectives, like open source.
ars 8y ago
How would that help? SuperMicro was not able to verify that what they designed and ordered is what was shipped. What makes you think that just because it's open source, that that would change? i.e. the design is "Open Source" to SuperMicro - but it didn't help them.

buserror 8y ago

That story is a bit odd, still -- normally behind the connector there is optionally magnetics, and at least a PHY... being able to integrate the magnetics in the connector exists allright, but adding the phy /as well/ must make it a marvel of integration regular manufacturers would dream of... especially at Gb speed! Also, you can't really 'piggyback' ethernet easily, for the same reasons; you would need TWO phy in there to decode/reencode... Even if you'd want to 'piggyback' on the link itself, it would be very, very difficult to say the least -- Gb ethernet is definitely not a gimme to synthesise, let alone piggyback. So, color me dubious -- the SPI 'chip' of last week was a but dubious but doable (given not just a custom chip, but a custom PCB) but this ethernet story makes even less sense!

jstanley 8y ago
Elsewhere in this thread there is an actual link to an actual NSA device that does exactly this. I don't think it's in the realm of science fiction.
- buserror 8y ago
  If you look at that illustration, you see that it's not just one ethernet connector, it's one of these massive connector stack with one ethernet and 2 USB, also, it adds quite a bit of depth to the connector; it must have been made with one particular brand/type of motherboard in mind. Still, if these are in the wild, then perhaps our chinese friends might have reduced the footprint even more to the size of one connector. I know the connectors with integrated magnetics are quite a bit 'longer' and 'beefier' than the passive ones.
  - monocasa 8y ago
    That's also at least a five year old implant since Snowden leaked it back in 2013.
    - kulahan 8y ago
      The wikipedia article on the device shows a date of 2008, so yeah, it's probably pretty outdated as far as spy tech is concerned.

okket 8y ago

So it is a compromised ethernet adapter. Nobody question the ability of Chinese spies to plant such a thing, but the "Big Hack" story implies that this is used as a mass infiltration tool, which I still find very improbable and lacks any evidence.

cm2187 8y ago
Did they say "mass"? I thought it was a handful or large companies.
gargravarr 8y ago
The way I interpret it is: since it's being introduced at the manufacturing plant, those installing the devices have no idea where the finished product will end up. Thus, these are not targetted attacks; they could wind up in the servers of a Fortune 500 company, or just as easily in some hobbyist's home lab. You'd need to compromise a large percentage of the products to increase the chances of landing a juicy target. The scattergun manner is what they're playing up here.
- saudioger 8y ago
  Considering that large scale internet companies like Amazon and Apple buy enormous amounts of hardware, it's not an insane strategy. If you compromise 1 of every 1,000 pieces of hardware (or even one of every 10,000) odds are you'll end up in a major datacenter pretty quickly.

carapace 8y ago

So does it really matter if it's true or not? Isn't the sane response to assume it's all true and verify your system integrity? Even if you find nothing you would then have a pile of strong evidence for the null case, eh?

RL_Quine 8y ago
You've got to be kidding. There's no way of validating anything about modern hardware, it's packed with independent systems running various firmware, software that's decades old, parts nobody can even identify unless you're the OEM. You can get to "that probably does this" level easily, but that doesn't tell you anything about its actual security or authenticity.
- andai 8y ago
  Is this a desirable state of affairs?
  - RL_Quine 8y ago
    No, but it seems to be inescapable.
- carapace 8y ago
  Funny you should say that... Guess what I'm working on, go on, guess. :-D And don't call me kidding. ;-P
platinumrad 8y ago
>So does it really matter if it's true or not? Is this Bloomberg's new slogan?

mmastrac 8y ago

It seems like there are two possibilities to me: 1) Bloomberg has a number of sources that are mistaken/misinformed, but this is not necessarily a made-up story, or 2) Bloomberg is nearly correct (minus some technical details) but the US government is forcing these companies to respond as if the story is wrong - possibly because of diplomatic reasons. What is the likelihood that #2 is correct? (there are other alternatives, but I believe that the likelihood that this is 100% or at least majorly fabricated by Bloomberg is near enough to zero)

roblabla 8y ago
The US cannot force those companies to lie. They can force them to stay silent, in which case they'd just say "No comments". If those companies are lying, they are committing security fraud.
- 394549 8y ago
  > The US cannot force those companies to lie. They can force them to stay silent, in which case they'd just say "No comments". If those companies are lying, they are committing security fraud. Would the US government have to force these companies to lie? It's quite possible that the denials were the result of voluntary cooperation.
  - macintux 8y ago
    A public company issuing such strongly-worded denials that turn out to be untrue would be leaving themselves at risk of an investigation by the SEC and/or a shareholder lawsuit.
    - ggggtez 8y ago
      It would be pretty extraordinary for the government to sue a company for cooperating with the government. A company has to follow court orders. The government would have to sue itself.

dwighttk 8y ago

Bloomberg really has it out for Supermicro.

printA 8y ago
Or an article author wanted to move the market. https://news.ycombinator.com/item?id=18162440 https://news.ycombinator.com/item?id=18162440
makomk 8y ago
Apparently, even their sole source for this article isn't happy with how they have it out for Supermicro: https://www.servethehome.com/yossi-appleboum-disagrees-bloomberg-is-positioning-his-research-against-supermicro/ https://www.servethehome.com/yossi-appleboum-disagrees-bloom...

amatecha 8y ago

I wonder what evidence caused them to claim the embedded "bug" is planted by China or its operatives -- how can they prove the source of the unauthorized modification? Are they just assuming?

kickopotomus 8y ago
This is my question as well. > Based on his inspection of the device, Appleboum determined that the telecom company's server was modified at the factory where it was manufactured. Any further evidence to support that claim? How can they be so sure that the boards weren't tampered with after they were manufactured and shipped to the states? Anyone with a little soldering experience could easily replace an ethernet port.

ipunchghosts 8y ago

Why should I believe any of this? No proof!

dawhizkid 8y ago

If Bloomberg's reporting is/was, in fact, entirely false then why wouldn't Amazon/Apple/Supermicro immediately sue them for libel/defamation?

educationdata 8y ago
In the U.S., there is a very high bar to accuse defamation. You not only need to prove the report is false, you also need to prove the reporter knew it was false and had bad intention to cause damage.
- Karunamon 8y ago
  That's defamation, but not libel. For clearing libel, all you have to prove is that the statement was presented as fact, was false, and caused harm. From my lay understanding, Supermicro has a slam dunk case given the impact on their stock price.
  - nkurz 8y ago
    For clearing libel, all you have to prove is that the statement was presented as fact, was false, and caused harm. At least in the US, I don't think this is true. Defamation is a broader class that includes libel, and always requires proving that the publisher was at least "negligent" in publishing the false statement: https://www.law.cornell.edu/wex/defamation https://www.law.cornell.edu/wex/defamation
  - roywiggins 8y ago
    You either have to know the fact is false, or act with reckless disregard as to whether it's false or not. 17 sources suggests that they did care whether it was true; if they didn't care, they would not have gone to all the effort.

jaclaz 8y ago

Set aside whether this report is reliable or not, I don't see how something like the Digi Connect ME: https://www.digi.com/products/embedded-systems/system-on-modules/digiconnectme https://www.digi.com/products/embedded-systems/system-on-mod... or the 9210 version: https://www.digi.com/products/embedded-systems/system-on-modules/digiconnectme9210 https://www.digi.com/products/embedded-systems/system-on-mod... cannot exist in Gb speed, if I recall correctly the original Digi Connect came out in 2005 or so.

deleted 8y ago

[deleted]

holstvoogd 8y ago

i call bs. again. one source, to many technical inaccuracies imo

detaro 8y ago

Could be due to losses in "translation", but this paragraph seems odd: > Three security experts who have analyzed foreign hardware implants for the U.S. Department of Defense confirmed that the way Sepio's software detected the implant is sound. One of the few ways to identify suspicious hardware is by looking at the lowest levels of network traffic. Those include not only normal network transmissions, but also analog signals -- such as power consumption -- that can indicate the presence of a covert piece of hardware. Ideas what they mean? I'd be surprised if mainboards etc exposed power consumption of random periphery parts. Maybe sleep/link states of the actual NIC that get influenced?

wyldfire 8y ago
If you look at the activity in the frequency domain, you might find a clock that's present in adulterated hardware that's missing in the nominal hardware. > exposed power consumption of random periphery parts Sampling these analog signals is done outside of the computer itself. Even if it were exposed via i2c, you couldn't trust anything that it would tell you.
- detaro 8y ago
  To me the article reads like it's a purely software solution, thus my confusion, but that might be misinterpretation.
  - makomk 8y ago
    I think it's probably also a software solution that the company whose CEO was the sole source for this article is trying to promote, judging from this marketing brochure for them: https://twitter.com/securelyfitz/status/1049725014075830272 https://twitter.com/securelyfitz/status/1049725014075830272 The claims in that certainly look every part as improbable.
gargravarr 8y ago
If both are going over the same physical interface, there could be tiny voltage drops to power the chip itself. The vague-as-ever article gives no indication as to how the chip would be powered, but such a small implant could conceivably leech off the legitimate ethernet interface.

smaili 8y ago

Supermicro shareholders just can't catch a break. Down yet another 25% today.

platinumrad 8y ago

Seeing as they screwed up a bunch of the details yet again[1], I'm still skeptical. 1. https://mobile.twitter.com/marcan42/status/1049690480806584325 https://mobile.twitter.com/marcan42/status/10496904808065843...

creeble 8y ago

Picture or it didn't happen. What is up with Bloomberg?

platinumrad 8y ago
And please no pictures of generic couplers marked in small text as "illustrations" like last time.

gargravarr 8y ago

Some real doublethink going on here, with Bloomberg continuing to insist that supply chain attacks are real, yet seemingly accepting the denials of each company involved. I am really not sure what to make of this.

passwordreset 8y ago
This is just a suggestion, but it might be useful to stop thinking about news reports as "the news report says X is real and true", because that's not what most news reports actually say. The text used usually reads like: "An unnamed source says blah blah", and "Joe Smith, a retied auto-worker, says blah blah blah", or "In response, a spokesman for Large Company, Inc. says blah blah blah blah", and so forth. Bloomberg (probably) isn't insisting that the story or the denial is true or false. They repeat the facts that other people have said. Often those facts are true; often those facts are false. Thinking about it this way helps me make sense of the topic. Maybe I'm weird.
- kickopotomus 8y ago
  It is not that cut and dry though. By publishing this story, they are asserting that the hack happened. They are not saying the attack is possible. They are saying it happened and that halved SMCI's market value. If they are wrong, Bloomberg is going to pay through the teeth for this one.

na85 8y ago

What's interesting to me in all this is not the immediate response but what the medium- to long-term impact will be. Does anyone doubt that semiconductor/electronics manufacturing will become a strategic industry in the same realm as uranium refining? I think we'll see electronics supply chain integrity play a growing role in major project delivery in the years to come.

tomc1985 8y ago

Are there any other sources with these allegations or is it just Bloomberg?

smurfysmurf 8y ago

I've seen several comments regarding whether or not Apple, Amazon etc. would deny the hacking if its true and if that is fraud or not. I work at Amazon now and previously was in the Navy, holding a TS/SCI. My firm belief is if such a hack happened, it would not be disclosed to anyone without a clearance, and the organizations that are denying it have no knowledge that it occurred. Furthermore if there truly was a compromise by a foreign nation it would be classified as a national security threat and subsequently classified and kept from public knowledge. Anyone who disclosed the truth would be at risk of loosing their clearance, job, and could end up getting the snowden treatment.

wil421 8y ago
I can see where the Navy/Military/Government could compartmentalize a hack like this. How could a company like Apple or Amazon keep this under wraps? How could they keep the knowledge of such a hack within the TS/SCI employees?
- smurfysmurf 8y ago
  The cleared department is handled the same way as in the military in terms of security. Amazon has SCIF's etc. So unless a disgruntled employee steps forward who doesn't care about there life, I imagine its easily contained (and symptoms of an employee being disgruntled are highly monitored when they hold a clearance)
  - wil421 8y ago
    I’m thinking about the non cleared data center folk, the sys admins and developers who use the servers for their applications. How do a bunch of Supermicro servers vanish wintout anyone noticing? I’d expect quite a few people would be involved that do not have any clearances. Apple is known for their secrecy but a few other companies named are not.
    - darawk 8y ago
      Maybe they didn't remove them.

noisy_boy 8y ago

This gets me thinking - what if this issue (assuming this is genuine and I'm starting to believe it is), is not limited to server hardware and more pervasive? What if this affects mobile phones as well? I'm looking suspiciously at my Nexus 6p made by Huawei.

kccqzy 8y ago

Might be off topic, but can Bloomberg even consistently use punctuation? I see mixes of dumb quotes and smart quotes in the same paragraph and the same sentence. And the backtick as well. It is those details that reveal to me the article is rushed, possibly very little care has been afforded to the article.

dig1 8y ago

Accidentally, all of this starts happening when trade war between USA and China is raging and when some countries decided to not follow USA orders to reduce business with China...

gepeto42 8y ago

Define "evidence".

thro_a_way 8y ago

why is YCombinator China a thing? I hope the firm gets brought before the Senate to answer for this.

virtualadmin 8y ago

This is not the first nor the last time we see reports about hardware manufactured in China being suspect of tampering like this. It will continue to happen. China steals technology and then manufacturers it for itself.

echevil 8y ago

Oh, it's Bloomberg again...

norseman 8y ago

This smells like a Snowden/Wikileaks strategy. Publish the first article expecting the major players to deny and discredit, but then publish more info after the fact to not only reveal the spying but trap all the big players in their lies.

kyrieeschaton 8y ago

Is no one going to mention that the named source is ex-Israeli Unit 8200, whose alumni themselves have a long record of espionage in the US telecom sector?

contingencies 8y ago
Questions to ask when making up your own mind on this issue: AMDOCS billing customer list includes how many major US telcos? How much customer data is required to generate a bill? How much billing is executed in real or near-real time? How much is hosted off-site on non telco infrastructure? How many non-billing services? (Someone elsewhere here already mentioned 100% of voicemail is now outsourced to AMDOCS at one telco). Incidentally, the source for this article is also conveniently based in Maryland.

deleted 8y ago

[deleted]

ectospheno 8y ago

Wake me up when a news organization other than bloomberg will go on record as saying the story has merit.

jackpirate 8y ago

What if the bloomberg article was a sort of false flag? The Trump administration has been consistently escalating retoric against the Chinese, and it's not hard to imagine the CIA/NSA/etc intentionally leaking facts to bloomberg that would make China look like a national security threat. This could even be done in a way where the security agencies don't leak anything actually false, but let the non-phds at bloomberg run wild with speculation to create a sensationalist story that's not really true. A recent WSJ article [1] has called particular attention to the Trump administration's escalating anti-Chinese rhetoric, calling it the start of a "second cold war". We know for a fact that these sorts of operations happened during the first cold war [2], so it's not at all hard to imagine they would happen now. A false flag attack fits with all the information we have so far about the event: There's no direct evidence of the attack, and if the bloomberg article is ever proven to be false, then only a small number of security researchers (and HNers) will ever learn about the retraction. The vast majority of Americans will only remember reading about how "China hacked major US companies" and create an anti-Chinese atmosphere that will help fuel future anti-Chinese policies. [1] https://www.wsj.com/articles/mike-pence-announces-cold-war-ii-1539039480 https://www.wsj.com/articles/mike-pence-announces-cold-war-i... [2] https://fas.org/sgp/news/2002/02/re022502.html https://fas.org/sgp/news/2002/02/re022502.html

4684499 8y ago

> The security expert, Yossi Appleboum, provided documents, analysis and other evidence > ... said Appleboum, who accompanied them for a visual inspection of the machine. So can we at least get some photos of tampered hardware from these "evidence"?

apo 8y ago

Appleboum said one key sign of the implant is that the manipulated Ethernet connector has metal sides instead of the usual plastic ones. The metal is necessary to diffuse heat from the chip hidden inside, which acts like a mini computer. "The module looks really innocent, high quality and 'original' but it was added as part of a supply chain attack," he said. How uncommon are metal vs plastic ethernet connector sleeves? This seems like a clue that even non-experts could use to track down an implant.

marshray 8y ago
I don't recall seeing a board-mounted Ethernet jack that didn't have metal sides. Reasons for an Ethernet jack to include a small metal enclosure: 1. The jack is subject to physical strain as we insert, remove, and tug on the cables. Most components are affixed to the board using only solder on the signal leads, but these do not provide significant mechanical strength. If you've ever had a bad headphone jack on personal electronics you are familiar with this phenomenon. So the little metal box has additional metal tabs that fit snugly in holes or slots on the board, and provide a stronger mechanical fit and a much larger surface area for solder adhesion. 2. Electromagnetic interference compliance. While Ethernet by definition involves pumping gigahertz signals out a long wire, these signals are carefully shaped and the cables pairs twisted to reduce leakage. But the designer of the jack doesn't know how much EM is flying around inside the case into which the port will be fitted. A metal box around the jack minimizes the size of the unshielded opening in the case. If you've installed a PC motherboard, you know the springy metal fingers on the backplate that seal against the block of external ports. Is it possible the article is talking about an Ethernet cable plug? I have occasionally seen those with metal sides. But they are not normally supplied as part of a motherboard or server.

Balvarez 8y ago

I'm not sure it matters to me whether the original article is 100% correct. What I think the original article points out correctly is that the Chinese supply chain is possibly a pretty easy vector for hardware based hacks. I would suspect most nation states have the pull to bribe/blackmail contractors to make malicious modifications. Though the chinese gov. would be the most likely culprit. It also seems plausible the NSA would prefer techies not to look too closely at their hardware *removes tinfoil hat

narrator 8y ago

And to think a couple of weeks ago I was called a lunatic for saying that backdoors are purposefully put into hardware by government actors: https://news.ycombinator.com/item?id=17736721 https://news.ycombinator.com/item?id=17736721

sunstone 8y ago

This story is so weird as Bloomberg doubles down with their reputation. What if the boards were compromised, but by the USA rather than China?

aranw 8y ago

I've been wondering if Google and Google Cloud is affected by this? Does anyone know who Google uses to build the custom motherboards that they use? Also how does something like the Titan Chip [0] help protect against such attack? I'm curious to understand if theres anything that can protect against this? [0] https://cloud.google.com/blog/products/gcp/titan-in-depth-security-in-plaintext https://cloud.google.com/blog/products/gcp/titan-in-depth-se...

fipple 8y ago

If anything, the fog of war here shows the costs that have been incurred alongside the massive efficiency gains from outsourcing/subcontracting in the global economy.

jhallenworld 8y ago

So I have bought Supermicro motherboards in quantity in the past. We went through a distributor and shipping was not direct from China. So I'm having doubts that Supermicro's Chinese CM has any idea who the end customer of any of their boards is. Maybe it's different for really huge customer, but I still have doubts- it implies they are building to order and have no inventory. This implies that the CM installs the Chinese Spy Chip on every board, or a random sampling of boards (so they should show up in the wild) or the tampering happens later in the supply chain- in the US. Also I have further doubts about embedding a chip in a middle layer of a PCB. I doubt the CM is going to add that much extra expense when it's easier to add a chip to the surface. You don't really buy much secrecy from embedding a chip given that it's easy to x-ray a board, so why bother? Anyway, there are devices called optical comparators. Supermicro could buy some security by providing comparison images that allow customers to perform an incoming inspection. I'm thinking they should do this now to add assurance / help their stock price. Here is an optical one: https://www.visionxinc.com/digital-optical-comparators/pcb https://www.visionxinc.com/digital-optical-comparators/pcb Here is an X-Ray one: http://www.glenbrooktech.com/multi-layer-pcb.php http://www.glenbrooktech.com/multi-layer-pcb.php Maybe the X-Ray inspection could pick up the Ethernet connector attack.

jakobegger 8y ago

This story is getting more incredible every day. Bloomberg only has second hand sources, and all the exploit details are based on speculation from security researchers -- not from insiders. It looks like Bloomberg heard several rumors of supply chain manipulations, mixed that up with plausible scenarios thought up by security researchers, added a few photos from random electronic parts, and voila you have a compelling story to tell. This "new evidence" talks about a completely different type of attack than the original article. It corroborates nothing. It just shows how misleading the original story was. I think the most damning part was the use of so many misleading photos and illustrations. All photos were pure speculation (this is what this chip might look like, this is where it would make sense to put the chip). But neither the captions nor the text made that clear. The only thing I believe about the story is that they have a couple of sources who have vague, second hand rumors about supply chain manipulation.

Havoc 8y ago
Yup. Which is why I'm holding on to my discount bin supermicro shares. The bloomberg story stinks to high heavens even if there are live examples of infiltrated supply chains
Symmetry 8y ago
The reporters in question apparently have a reputation for credulously repeating things they hear about cyber-attacks. https://twitter.com/RobertMLee/status/1049617855396933632 https://twitter.com/RobertMLee/status/1049617855396933632

mmaunder 8y ago

The credibility of the journalists is being called into question by influential people in the infosec community: https://twitter.com/RobertMLee/status/1049617855396933632?s=19 https://twitter.com/RobertMLee/status/1049617855396933632?s=... https://risky.biz/RB517_feature/ https://risky.biz/RB517_feature/ The podcast is an interview with Joe Fitzpatrick, one of the named sources. It's an impartial and mature discussion that left me with the same feeling of unease that Joe says he had when he first read the story. Basically, Joe would describe a theoretical hardware exploit to the Bloomberg journalist and the journo's sources would then confirm exactly that as a real world exploit.

jgamman 8y ago

tl;dr US shocked (shocked i say!) that other countries doing the same sorts of things they do.

mrb 8y ago

It's interesting to note that the organization supposedly at the center of the investigation of this tampering of Supermicro servers—the FBI—has never issued a statement on the veracity of the story. Strange silence...

directusss 8y ago

Damn... sounds like Bloomberg is really hating China, even it's driving directly into the law-suit hell.

browsercoin 8y ago

maybe its just me but this seems all too familiar. we've seen it happen with iraq, reputable media outputs sensational hostile state actions, immediately faces criticism, but more importantly the timing after announcing China tried meddling with elections. i dont know sometimes i like reading into the big picture, what is the purpose of this bloomberg article for those outside HN? it won't invoke feelings of calm but rather moral panic. now half the americans are riled up to think china is attacking usa, and we remember they need just teh right amount of support. also as the noose is tightening around trump, a war or a limited military conflict would be the perfect distraction. /end conspiracy

a3n 8y ago

Maybe this was an NSA operation, but they outsourced the hardware to China, because they are, you know, cheap.

beamatronic 8y ago

Who profits from this story?

mohammedbin 8y ago

I want to mention a. political side of the argument- M Bloomberg himself is very pro open trade and has strongly hinted at running in 2020. Also, newspapers don't usually hurt your owners' candidacy, even for explosive stories. Bloomberg isn't just putting it's reputation on stake here, it's legitimized Trump presidency further. I would say there is definitely something behind this story.

kerng 8y ago

Even if Bloomberg would be incorrect, the good thing will be that everyone starts looking now and in the end it might become a self fulfilling prophecy. It's unlikely that something like this wasn't already done. Scale and targets could differ though

OliverJones 8y ago

Looks like the infosec cold war is in full swing. All secrets eventually leak. All secrets. No exceptions. Not even nation-state players with unlimited budgets can prevent leaks. This is reality. What's a hacker (in the original sense: programmer, not cybercreep) to do? Our task as custodians of secrets for our end-users is to reduce the attack surfaces on our systems, slow down those leaks and mitigate the effects of leaks when they happen. We must do these things to the best of our ability. We must do them whether we rig systems for large organizations or for our grandmothers. Who's trustworthy? Apple? Probably. They're pushing security as a major component of their brand. AWS? Possibly. They have a lot to lose if they're compromised. Microsoft? Possibly. They too have a lot to lose. Seventeen well-placed but unnamed sources in the US security apparatus, babbling to journalists? Possibly. Journalists? Their trustworthiness is eroding. Cryptographers Whit Diffie, Martin Hellman, Ralph Merkle, Bruce Scheier, Ron Rivest and colleagues? Likely. Motherboard vendors? Probably not. Router / switch / firewall vendors? Probably not. Nation-states? No. (They could change this by abolishing "security by obscurity" in their work, but that would require major changes in mindset.)

HillaryBriss 8y ago

my guess is that the NSA, spy agencies from China, Bloomberg, SuperMicro, Apple, Amazon and anyone else involved are carefully watching HN comments to see who interprets the story correctly so they can seek that person out and hire

prirun 8y ago

The article says: "The manipulation of the Ethernet connector appeared to be similar to a method also used by the U.S. National Security Agency, details of which were leaked in 2013." So the US doesn't exactly have clean hands. Here's a theory: 1. The US wants to have more ability to spy 2. With most manufacturing in China, this is hard 3. Implicating Supermicro might cause their customers to switch suppliers. It has already caused Supermicro stock to go down 41% then 27% again. 4. The US / Sepio Systems / former Mossad / former CIA officials come to the rescue with a company manufacturing "secure servers" not made in China. Or more likely, still made in China, only under different control. 5. Result: profit, increase US ability to spy, "bring manufacturing back to US companies" IMO, governments - all governments - have no ethics and will do whatever they want to further their agenda.