10 ms·

I previously submitted this story a few days ago[1], where it garnered some discussion and a bit over 100 upvotes, but never made the front-page due to some dow

I previously submitted this story a few days ago[1], where it garnered some discussion and a bit over 100 upvotes, but never made the front-page due to some down-weights in HN's algorithm. I'm reposting this story at Daniel's suggestion. He agreed it deserved a second chance to break out. -- I noticed that it's commonly accepted that Bloomberg's 2018 stories on the Supermicro hack were bogus, costing them a huge amount of reputational damage. However, Bloomberg stood by the stories. I'm very curious to see how this one will be received. I was never quite convinced by the naysayers or the denials by the government, Apple, or Amazon. I think it's quite likely that Bloomberg will be proven right in the end, this story having been accurate but suppressed for "secrecy" reasons all along. Notably, one of the criticisms at the time was that Bloomberg didn't name credible sources with first-hand accounts of the events — understandably given the sensitivity. This time, it seems they have been able to do so. Either way, I'm thrilled to see an outlet do original reporting and stand by their work in the face of universal condemnation. We are better off for having strong dissenting views informing us and expanding the narrative. I'm finding myself increasingly distrustful of the work of journalism outlets, but this (and those gone independent on Substack) gives me hope. [1]: Previous discussion here: https://news.ycombinator.com/item?id=26112436 https://news.ycombinator.com/item?id=26112436

deleted 6y ago
[deleted]
stevemk14ebr 6y ago
I think this twitter thread is a very reasonable analysis of this article: https://twitter.com/pwnallthethings/status/1360232753623629824 https://twitter.com/pwnallthethings/status/13602327536236298.... I think critically there are a lot of claims here, and none of them provide _concrete_ proof of any form. There's a lot of 'senior official said X' and X number of people in the know declined to comment, but these are just appeals to authority/credentials. People are flawed and second hand rumours of rumours can easily bend the truth. I don't think it's necessarily fair to put out a story claiming this is 'the truth' when alot of the sourcing is indirect and 'declined to comment'. It's just not there with the facts! True or not, we can't know, but this report need more facts.
- jorblumesea 6y ago
  This is `usually how national security problems are reported on, due to the classified nature of the work. You don't have to believe it, but it's naive to think that you're going to see actual chips. Currents and formers can talk about the what, but not the how. I mean even China knows the "what". Methods and means are a big deal, PRB kills entire chapters because of it.
- datavirtue 6y ago
  Head in sand. A outfit like bloomberg isn't going going to screw up this bad and then double down by citing more sources. Get a clue.
- jstrong 6y ago
  > Alarmed by the devices’ sophistication, officials opted to warn a small number of potential targets in briefings that identified Supermicro by name. Executives from 10 companies and one large municipal utility told Bloomberg News that they’d received such warnings. in my book that counts as "concrete proof."
  - deftnerd 6y ago
    Only if you believe that the warnings went out because of the "alarm over the devices' sophistication" and not "alarm over the idea this could happen" or even "alarm over a misidentification of a legitimate component and now we're too embarrassed to issue a retraction to those 10 CEOs"
    - koheripbal 6y ago
      Ok, but now you're making another argument with also zero evidence. I think it's fair to say there are multiple circumstantial pieces of evidence that SuperMicro is inserting security vulnerabilities. That should be enough to take the same decision that both the US gov't and Apple and others have made to avoid that manufacturer.
  - michaelt 6y ago
    The article certainly quotes a wealth of sources - but all of them seem to be vague, third-hand stuff. An unnamed "adviser" to security firms that analyzed Supermicro equipment. An executive for some unnamed company, who received a briefing. A venture capitalist who received a briefing. A retired FBI agent who was told there was an "additional little component" by someone he can't name. Some former FBI officials that refuse to name supermicro, and say only that supply chain attacks are possible. Fifty interviews with officials, all of whom asked not to be named. So apparently this information is so public that everyone and their dog is happy confirm it - yet at the same time, so classified that the victims and the 50 sources can't be named? That's pretty vague compared reports of hardware implants like credit card skimmers [1] which put pictures of the hardware front and centre and make it clear that the author has personally seen it and knows (basically) how it works. [1] https://krebsonsecurity.com/all-about-skimmers/ https://krebsonsecurity.com/all-about-skimmers/
ikeboy 6y ago
>Notably, one of the criticisms at the time was that Bloomberg didn't name credible sources with first-hand accounts of the events — understandably given the sensitivity. It's not just that - it's that no other reputable outlet was able to confirm the story, and many tried. This just doesn't happen with controversial but true stories with many sources - some of those sources will be willing to talk to other outlets.
- jtbayly 6y ago
  Bruce Schneier said last time that he didn’t believe it. This time he said he was wrong. When pressed by a skeptic in the comments, he said, “Independent from this article, I know that the DoD believes this threat is real and is giving away R&D funding for security solutions. So I have more than the Bloomberg reporting to go on.” https://www.schneier.com/blog/archives/2021/02/chinese-supply-chain-attack-on-computer-systems.html https://www.schneier.com/blog/archives/2021/02/chinese-suppl...
  - Daho0n 6y ago
    That "the DoD believes this threat" is not the same as it actually happened. The threat is real, the article is very likely fluff.
    - jtbayly 6y ago
      Yes, but everybody already knows the threat is real, most especially Schneier. Something made him change his mind.
    - datavirtue 6y ago
      All you people are in denial. The first bloomberg story was credible but was protecting sources. They go further and get people on the record and double down....literally stretching their necks out, and you people keep propping up strawmen from the original article. Admit it. You were skeptical, determined that it couldn't be true, ran around shooting your mouth off about how it was fake news and now you can't help but continue smearing the egg all over your face.
ncmncm 6y ago
For a story claiming physical alteration of products, they should have had photographs of altered products. Otherwise it's just "anonymous loudmouths blow smoke". Everything that would be done with the extra component could more easily be done with BIOS code, attracting no attention. So it doesn't pass the sniff test.
- gruez 6y ago
  >Everything that would be done with the extra component could more easily be done with BIOS code This is tricky because on newer platforms the BIOS is signed and verified by the CPU. The signing keys are likely going to be located in the engineering offices (in the US) rather than just lying around in the factories, so getting them would require much more access than just bribing/threatening a few technicians at the factory. > attracting no attention. So it doesn't pass the sniff test. Compromised BIOS is a known threat, so I'd presume paranoid customers would verify/reflash the BIOS upon receiving them. This can be done without booting the motherboard at all (ie. attaching SPI clips to the flash chip and reading/writing directly). A tampered bios would get detected this way so it's not exactly "no attention".
- mdip 6y ago
  I don't think you deserved the downvotes as I felt somewhat the same way. Part of that comes from the reporting, which appears to have been done by someone who doesn't understand the things he's reporting on (or, at least, some of the people quoted are saying things that are nonsense that someone outside of our industry wouldn't necessary recognize) i.e. "had a chip encrypted on the motherboard that would record all the data" ... one does not simply ... encrypt a chip ... With these kinds of explanations or statements used as evidence, it's frustrating trying to figure out what "really happened". Personally, I don't doubt there's something to all of this. It's a "real threat" as has been reported; whether or not it's "theoretically real" or "actually used" isn't that relevant: if it's a "real threat" someone has figured out precisely how to do it or has seen it (or close-enough variations of the "it") being done that it's worth taking action. My hunch is that it's a firmware-related hack; no hardware added to the boards. The article talks around BIOS and such, indicating that some of the code involved was inserted by employees directly connected to the company. It explains the lack of photographic evidence--what's the evidence[0]? And wouldn't the attacker prefer it that way? It also helps explain the "encrypted chip" -- perhaps code, encrypted on the chip (decrypted/executed at boot) was what was intended by that statement[1]. [0] Disassembled code on Github would have been nice, but the US government isn't likely to drop that. [1] And maybe it was said in that bizarre manner intentionally, who knows?
marcan_42 6y ago
It's not just the lack of sources, it's the fact that Bloomberg made up complete nonsense to fill in the bits of the picture that they didn't have. Off the top of my head from that time, I remember they asked someone about the plausibility of some part of the story about the component types that might have been altered, he pointed them at some random example from Digi-Key, and that example became their photoshoot prop. All of their photos were provably fake and staged, and made no technical sense. There was that whole sub-story about how supposedly Ethernet jacks having metal shields was one of the indicators of compromise (what? almost every piece of datacenter kit uses Ethernet jacks with metal shields). The more you looked into the details, the less they added up. The only thing we could confirm was that, at some point, Supermicro shipped compromised drivers. That part I can absolutely believe, and like the last bit of this story update claims, that may well have been an APT hack from a state-actor. That side of the story is entirely believable and self-consistent. But the hardware modifications? Sorry, but the way that story has been reported is complete garbage. I'm not saying it isn't true, I'm saying Bloomberg provided no credible evidence that it is true, and plenty of evidence that they don't have the slightest clue what they're doing, and that their reporting is dishonest. Originally I gave them the benefit of the doubt, but when they published the Ethernet jack story with less than 5 days to check anything from that guy, they lost all semblance of credibility.
- plorkyeran 6y ago
  Yeah, I'm confused by all the posts acting like the objections were just that Bloomberg's first story didn't provide any evidence. It was quite a bit worse than that; they included some very obviously faked evidence and a lot of utter nonsense. I wouldn't be surprised if it turned out to be "based on a true story" but even if every single one of their claims turns out to have been a distorted retelling of something real then I'll still stand by my opinion that it was terrible reporting.
- kenneth 6y ago
  Oftentimes, media will use imagery or illustration that are there for illustrative purposes. I don't think Bloomberg claimed to be showing actual photographic evidence. Not to mention, oftentimes the story and the presentation come from separate sources — the editors taking the written story and deciding the headline, and how to present and illustrate it.
Number157 6y ago
Well actually kenneth is behind everything and we can prove is because we have the sources to back it up. No of course we can't give out the sources, our journalistic integrity requires us to protect our sources. But we fact checked everything and it's 100% true.
cheph 6y ago
> Notably, one of the criticisms at the time was that Bloomberg didn't name credible sources with first-hand accounts of the events — understandably given the sensitivity. Wait is this all of a sudden a prerequisite for something being believable?!?!?! What the hell did I live through since 2016?
JumpCrisscross 6y ago
As a bicoastal American, there is a strong East-West schism in that. West Coast techies, largely disconnected from D.C., being (rightfully albeit overconfidently) sceptical. East Coasters being (almost always overconfidently) confident in their friend at such and such (almost always unrelated) agency vouching for the story.
Puts 6y ago
This was published almost one and a half year before the Bloomberg story: https://www.theinformation.com/articles/apple-severed-ties-with-server-supplier-after-security-concern https://www.theinformation.com/articles/apple-severed-ties-w... Why do you completely cut tie with a big supplier over security concerns and at the same time don't disclose why? I'd say this is exceptional in the business of IT-security.
geofft 6y ago
> We are better off for having strong dissenting views informing us and expanding the narrative. Strong dissenting views is a means to an end, not an end. The truth is some point in the space of plausible ideas. If the region covered by existing ideas doesn't overlap the truth, then yes, you need to expand it. If it does, then looking elsewhere is guaranteed to bring you farther away from the truth. If the story is false (or, more precisely, Bloomberg's reporters are talking to people who aren't technically qualified to analyze the subject and are misinterpreting it and Bloomberg is putting its own spin of misinterpretation on it), then Bloomberg's contribution to the discourse - to the pursuit of truth - is of no more value than the National Enquirer saying that Batboy hacked all our computers. Possibly of less value, in that people know not to take the Enquirer seriously. The ability to have a wide range of views expressed is good for the pursuit of truth, yes. That doesn't mean that a dissenting view is of merit because it is dissenting. And in the case of this article, when it was based on a previous one that was attacked on the merits and not because of how dissenting it was, and seems to be more of the same, a commitment to the search for truth requires us to dismiss it.
vl 6y ago
This is interesting how and why reasonably credible stories instead of being investigated get ignored by mainstream media all the time, recent example being NY Post. In this case Bloomberg comes back with the vengeance, but we need to see if it will picked up or will be buried again.
- stjohnswarts 6y ago
  NYP is garbage, quite famous for splashing completely untrue stories on the front (or top article on WWW) and then retracting it a few days later in the most remote areas of their media.
stjohnswarts 6y ago
Thanks! I didn't see it before. I'd been following it a bit but this fleshed out more details for me. Kudos.
TwoNineFive 6y ago
> I'm finding myself increasingly distrustful of the work of journalism outlets You can't say something like this about Bloomberg News and what happened with the Bloomberg presidential campaign and have credibility.

deleted 6y ago

[deleted]

stevemk14ebr 6y ago

I think this twitter thread is a very reasonable analysis of this article: https://twitter.com/pwnallthethings/status/1360232753623629824 https://twitter.com/pwnallthethings/status/13602327536236298.... I think critically there are a lot of claims here, and none of them provide _concrete_ proof of any form. There's a lot of 'senior official said X' and X number of people in the know declined to comment, but these are just appeals to authority/credentials. People are flawed and second hand rumours of rumours can easily bend the truth. I don't think it's necessarily fair to put out a story claiming this is 'the truth' when alot of the sourcing is indirect and 'declined to comment'. It's just not there with the facts! True or not, we can't know, but this report need more facts.

jorblumesea 6y ago
This is `usually how national security problems are reported on, due to the classified nature of the work. You don't have to believe it, but it's naive to think that you're going to see actual chips. Currents and formers can talk about the what, but not the how. I mean even China knows the "what". Methods and means are a big deal, PRB kills entire chapters because of it.
datavirtue 6y ago
Head in sand. A outfit like bloomberg isn't going going to screw up this bad and then double down by citing more sources. Get a clue.
jstrong 6y ago
> Alarmed by the devices’ sophistication, officials opted to warn a small number of potential targets in briefings that identified Supermicro by name. Executives from 10 companies and one large municipal utility told Bloomberg News that they’d received such warnings. in my book that counts as "concrete proof."
- deftnerd 6y ago
  Only if you believe that the warnings went out because of the "alarm over the devices' sophistication" and not "alarm over the idea this could happen" or even "alarm over a misidentification of a legitimate component and now we're too embarrassed to issue a retraction to those 10 CEOs"
  - koheripbal 6y ago
    Ok, but now you're making another argument with also zero evidence. I think it's fair to say there are multiple circumstantial pieces of evidence that SuperMicro is inserting security vulnerabilities. That should be enough to take the same decision that both the US gov't and Apple and others have made to avoid that manufacturer.
- michaelt 6y ago
  The article certainly quotes a wealth of sources - but all of them seem to be vague, third-hand stuff. An unnamed "adviser" to security firms that analyzed Supermicro equipment. An executive for some unnamed company, who received a briefing. A venture capitalist who received a briefing. A retired FBI agent who was told there was an "additional little component" by someone he can't name. Some former FBI officials that refuse to name supermicro, and say only that supply chain attacks are possible. Fifty interviews with officials, all of whom asked not to be named. So apparently this information is so public that everyone and their dog is happy confirm it - yet at the same time, so classified that the victims and the 50 sources can't be named? That's pretty vague compared reports of hardware implants like credit card skimmers [1] which put pictures of the hardware front and centre and make it clear that the author has personally seen it and knows (basically) how it works. [1] https://krebsonsecurity.com/all-about-skimmers/ https://krebsonsecurity.com/all-about-skimmers/

ikeboy 6y ago

>Notably, one of the criticisms at the time was that Bloomberg didn't name credible sources with first-hand accounts of the events — understandably given the sensitivity. It's not just that - it's that no other reputable outlet was able to confirm the story, and many tried. This just doesn't happen with controversial but true stories with many sources - some of those sources will be willing to talk to other outlets.

jtbayly 6y ago
Bruce Schneier said last time that he didn’t believe it. This time he said he was wrong. When pressed by a skeptic in the comments, he said, “Independent from this article, I know that the DoD believes this threat is real and is giving away R&D funding for security solutions. So I have more than the Bloomberg reporting to go on.” https://www.schneier.com/blog/archives/2021/02/chinese-supply-chain-attack-on-computer-systems.html https://www.schneier.com/blog/archives/2021/02/chinese-suppl...
- Daho0n 6y ago
  That "the DoD believes this threat" is not the same as it actually happened. The threat is real, the article is very likely fluff.
  - jtbayly 6y ago
    Yes, but everybody already knows the threat is real, most especially Schneier. Something made him change his mind.
  - datavirtue 6y ago
    All you people are in denial. The first bloomberg story was credible but was protecting sources. They go further and get people on the record and double down....literally stretching their necks out, and you people keep propping up strawmen from the original article. Admit it. You were skeptical, determined that it couldn't be true, ran around shooting your mouth off about how it was fake news and now you can't help but continue smearing the egg all over your face.

ncmncm 6y ago

For a story claiming physical alteration of products, they should have had photographs of altered products. Otherwise it's just "anonymous loudmouths blow smoke". Everything that would be done with the extra component could more easily be done with BIOS code, attracting no attention. So it doesn't pass the sniff test.

gruez 6y ago
>Everything that would be done with the extra component could more easily be done with BIOS code This is tricky because on newer platforms the BIOS is signed and verified by the CPU. The signing keys are likely going to be located in the engineering offices (in the US) rather than just lying around in the factories, so getting them would require much more access than just bribing/threatening a few technicians at the factory. > attracting no attention. So it doesn't pass the sniff test. Compromised BIOS is a known threat, so I'd presume paranoid customers would verify/reflash the BIOS upon receiving them. This can be done without booting the motherboard at all (ie. attaching SPI clips to the flash chip and reading/writing directly). A tampered bios would get detected this way so it's not exactly "no attention".
mdip 6y ago
I don't think you deserved the downvotes as I felt somewhat the same way. Part of that comes from the reporting, which appears to have been done by someone who doesn't understand the things he's reporting on (or, at least, some of the people quoted are saying things that are nonsense that someone outside of our industry wouldn't necessary recognize) i.e. "had a chip encrypted on the motherboard that would record all the data" ... one does not simply ... encrypt a chip ... With these kinds of explanations or statements used as evidence, it's frustrating trying to figure out what "really happened". Personally, I don't doubt there's something to all of this. It's a "real threat" as has been reported; whether or not it's "theoretically real" or "actually used" isn't that relevant: if it's a "real threat" someone has figured out precisely how to do it or has seen it (or close-enough variations of the "it") being done that it's worth taking action. My hunch is that it's a firmware-related hack; no hardware added to the boards. The article talks around BIOS and such, indicating that some of the code involved was inserted by employees directly connected to the company. It explains the lack of photographic evidence--what's the evidence[0]? And wouldn't the attacker prefer it that way? It also helps explain the "encrypted chip" -- perhaps code, encrypted on the chip (decrypted/executed at boot) was what was intended by that statement[1]. [0] Disassembled code on Github would have been nice, but the US government isn't likely to drop that. [1] And maybe it was said in that bizarre manner intentionally, who knows?

marcan_42 6y ago

It's not just the lack of sources, it's the fact that Bloomberg made up complete nonsense to fill in the bits of the picture that they didn't have. Off the top of my head from that time, I remember they asked someone about the plausibility of some part of the story about the component types that might have been altered, he pointed them at some random example from Digi-Key, and that example became their photoshoot prop. All of their photos were provably fake and staged, and made no technical sense. There was that whole sub-story about how supposedly Ethernet jacks having metal shields was one of the indicators of compromise (what? almost every piece of datacenter kit uses Ethernet jacks with metal shields). The more you looked into the details, the less they added up. The only thing we could confirm was that, at some point, Supermicro shipped compromised drivers. That part I can absolutely believe, and like the last bit of this story update claims, that may well have been an APT hack from a state-actor. That side of the story is entirely believable and self-consistent. But the hardware modifications? Sorry, but the way that story has been reported is complete garbage. I'm not saying it isn't true, I'm saying Bloomberg provided no credible evidence that it is true, and plenty of evidence that they don't have the slightest clue what they're doing, and that their reporting is dishonest. Originally I gave them the benefit of the doubt, but when they published the Ethernet jack story with less than 5 days to check anything from that guy, they lost all semblance of credibility.

plorkyeran 6y ago
Yeah, I'm confused by all the posts acting like the objections were just that Bloomberg's first story didn't provide any evidence. It was quite a bit worse than that; they included some very obviously faked evidence and a lot of utter nonsense. I wouldn't be surprised if it turned out to be "based on a true story" but even if every single one of their claims turns out to have been a distorted retelling of something real then I'll still stand by my opinion that it was terrible reporting.
kenneth 6y ago
Oftentimes, media will use imagery or illustration that are there for illustrative purposes. I don't think Bloomberg claimed to be showing actual photographic evidence. Not to mention, oftentimes the story and the presentation come from separate sources — the editors taking the written story and deciding the headline, and how to present and illustrate it.

Number157 6y ago

Well actually kenneth is behind everything and we can prove is because we have the sources to back it up. No of course we can't give out the sources, our journalistic integrity requires us to protect our sources. But we fact checked everything and it's 100% true.

cheph 6y ago

> Notably, one of the criticisms at the time was that Bloomberg didn't name credible sources with first-hand accounts of the events — understandably given the sensitivity. Wait is this all of a sudden a prerequisite for something being believable?!?!?! What the hell did I live through since 2016?

JumpCrisscross 6y ago

As a bicoastal American, there is a strong East-West schism in that. West Coast techies, largely disconnected from D.C., being (rightfully albeit overconfidently) sceptical. East Coasters being (almost always overconfidently) confident in their friend at such and such (almost always unrelated) agency vouching for the story.

Puts 6y ago

This was published almost one and a half year before the Bloomberg story: https://www.theinformation.com/articles/apple-severed-ties-with-server-supplier-after-security-concern https://www.theinformation.com/articles/apple-severed-ties-w... Why do you completely cut tie with a big supplier over security concerns and at the same time don't disclose why? I'd say this is exceptional in the business of IT-security.

geofft 6y ago

> We are better off for having strong dissenting views informing us and expanding the narrative. Strong dissenting views is a means to an end, not an end. The truth is some point in the space of plausible ideas. If the region covered by existing ideas doesn't overlap the truth, then yes, you need to expand it. If it does, then looking elsewhere is guaranteed to bring you farther away from the truth. If the story is false (or, more precisely, Bloomberg's reporters are talking to people who aren't technically qualified to analyze the subject and are misinterpreting it and Bloomberg is putting its own spin of misinterpretation on it), then Bloomberg's contribution to the discourse - to the pursuit of truth - is of no more value than the National Enquirer saying that Batboy hacked all our computers. Possibly of less value, in that people know not to take the Enquirer seriously. The ability to have a wide range of views expressed is good for the pursuit of truth, yes. That doesn't mean that a dissenting view is of merit because it is dissenting. And in the case of this article, when it was based on a previous one that was attacked on the merits and not because of how dissenting it was, and seems to be more of the same, a commitment to the search for truth requires us to dismiss it.

vl 6y ago

This is interesting how and why reasonably credible stories instead of being investigated get ignored by mainstream media all the time, recent example being NY Post. In this case Bloomberg comes back with the vengeance, but we need to see if it will picked up or will be buried again.

stjohnswarts 6y ago
NYP is garbage, quite famous for splashing completely untrue stories on the front (or top article on WWW) and then retracting it a few days later in the most remote areas of their media.

stjohnswarts 6y ago

Thanks! I didn't see it before. I'd been following it a bit but this fleshed out more details for me. Kudos.

TwoNineFive 6y ago

> I'm finding myself increasingly distrustful of the work of journalism outlets You can't say something like this about Bloomberg News and what happened with the Bloomberg presidential campaign and have credibility.