8 ms·

Supermicro boards were so bug ridden, why would hackers ever need implants?

182 points by drewg123 8y ago

deleted 8y ago
[deleted]
qaq 8y ago
I would guess that large companies are refreshing with known good firmware before deploying servers? So while described approach is easier prob will not get attacker as much.
- bdamm 8y ago
  You would guess wrong because a) large companies are profit motivated and that is a cost that can be cut and b) even if you "refresh" firmware, what actually is happening in there? You don't really know, so the "refresh" may not be as effective at wiping out the attack as you had hoped.
- vel0city 8y ago
  Most BMC updates are handled in software on the BMC. You're giving the BMC a new image file to write and trusting the BMC to actually write it. Who's to say the BMC is dutifully writing that image to the flash memory? Who's to say it doesn't re-infect the image before writing? Even if you do directly connect to the flash module and directly write to it through SPI, if the attack is being loaded by an additional module between the flash memory and the BMC, it could still inject additional data into the BMC's boot. If you're not physically listening to the SPI data being transmitted or knew what to look for in the final environment of the BMC, you wouldn't know it had happened.
throwaway290342 8y ago
Okay, crazy tinfoil hat time: what if this story is a plant from a particular part of the Chinese government (like PLA Unit 61398), designed to give the impression of the ability to disrupt global supply chains and to build respect through fear? If all of these unnamed sources are unnamed because they were adversarial members impersonating government officials, then that would make a little more sense why current government bodies are not just staying mum, but actually denying knowledge of the story. With the software attacks being much more feasible as the Ars article points out than a hardware attack, then it would also make it so that the vehement denials from affected companies would be true as well. The whole thing could be a large disinformation campaign to strike at the very core of what many would otherwise consider reasonable security.
- wycy 8y ago
  It seems like this would be a really bad idea. Scaring companies away from buying Chinese-manufactured products couldn't possibly be worth the respect through fear.
- nakedrobot2 8y ago
  No. A better tinfoil hat theory is that the whole story was fabricated by Russia, to (you know, as always) sow chaos.
- jack6e 8y ago
  But the effect of that would be to cause massive distrust of Chinese suppliers and cause a shift away from electronics being produced there. IC and cyber experts generally identify the Chinese as using intelligence operations for primarily economic purposes, as compared to Russian/Iranian/North Korean objectives being military or political. A Chinese military intelligence agency using cyber espionage to intentionally disrupt one of the most significant export industries of the Chinese economy does not seem likely, nor does it seem to provide such an out-sized strategic benefit as to be worth the economic cost.
  - throwaway290342 8y ago
    Good point, I agree with that thinking. But the actual execution of such a hardware-based attack would surely be discovered at some point anyway, and risk the same negative outcome. So then that would leave the only possible conclusion that the story just isn't true at all. In the end, none of it makes clear sense...
    - jack6e 8y ago
      The difference is two-fold: actively planting a fake story means that first, the espionage is fake and thus no real intelligence can be gathered, so the only benefit is the hypothetical respect you suggested; second, the story will definitely get out, thus the potential for the negative effect is innately 100%. However, as a real intelligence operation the cost/benefit analysis is inverted, because there is a real, tangible benefit to extracting possibly sensitive commercial and national security information. And while an eventual discovery is always a possibility, it seems care was taken to ensure it would only be a small possibility, and that in any case it would be in the future, hopefully after a large amount of useful data is extracted. So in the planted story hypothesis, there is certainty of negative outcomes with only the potential for positive outcomes, and those only intangible, while in the this-is-real hypothesis, there is near certainty of some tangible benefit with good probability of significant tangible benefit, with only a potential, distant, deniable risk of negative outcome.
rayiner 8y ago
I feel like this article reflects some significant technical confusion. The BMC is supposed to be on a trusted network inaccessible from the outside. I've always viewed authentication on the BMC as being like the numeric lock on luggage--it's designed to keep honest people honest, not for real security. Being able to bypass the BMC security is really not a big deal. What the Bloomberg article says about the hardware exploit is much worse: > > The rogue instructions, Bloomberg reported, caused the BMCs to download malicious code from attacker-controlled computers and have it executed by the server’s operating system. It's using the fact that the BMC has unfettered access to the rest of the machine to compromise the code running on the server itself. That's valuable even if the BMC itself is on a private network inaccessible to the attacker.
- stephengillie 8y ago
  BMCs like DRAC or iLO are invaluable when you have hundreds or thousands of fresh servers with no OS. The BMC lets you mount an OS or hypervisor ISO in a way reminiscent of DaemonTools et al., and update bios and other firmware from a shared network folder. I'm pretty sure there's even an API to develop against.
  - rayiner 8y ago
    BMC's are great--all my home builds have them because I'm too old to be fiddling around trying to figure out why a computer won't boot an installer from a USB key. But even on my home network the BMC's are on a separate switch on a subnet that doesn't have internet access except through a VPN gateway.
    - jacquesm 8y ago
      > except There's your problem.
      - xyzzy123 8y ago
        I think the parent meant, you have them on a subnet with no default route, but have a vpn / management system with one interface pointing in to the management network. You can get in via the vpn but they can’t get out. It’s a pretty common configuration.
mannykannot 8y ago
While I don't think Bloomberg's story looks very plausible, perhaps one motivation for cryptic hardware modification at a time when firmware weaknesses were being discovered might be precisely because the easier-to-exploit firmware weaknesses were being discovered, and so might not be exploitable much longer? It might not have seemed plausible that the vulnerabilities would be discovered but then not fixed to the extent that, it turns out, they were not.
mhjas 8y ago
I don't really see why everyone is calling this implausible. Modchips have been around for at least 15 years. The idea of the clipper chip is 25 years old. At every hacker conference there are people "hacking" devices by various buses or interfaces. If there is anything working against the Bloomberg story it is that it is too plausible. Often reality clashes with imagination, but the Bloomberg story contains almost everything you could imagine happening.
- ryanmarsh 8y ago
  Exactly. How much has the US spent on the F-35? How much has China spent on making artificial islands? Yet engineering a chip and bribing/threatening a few factory workers is beyond the pale?
  - deleted 8y ago
    [deleted]
  - kevin_thibedeau 8y ago
    Bloomberg's claim is that a miniature device used for RF analog electronics was coopted and inserted into a board that would never have such a part designed in. This requires modifying the board artwork, the pick and place config, any automated inspection and test equipment, and adding a foreign part reel to the supply chain. It is much easier to compromise firmware directly or modify ICs that are already part of the design. The risk of being caught is much lower and it would be stupid to attempt anything more elaborate.
    - pnematoad 8y ago
      This requires [et c.] Or, they could run their own fabrication facilty, where they can exert total control over the production line, in total secrecy, and you'd never know the difference, or notice a sacrifice in the fidelity of replication. Think that's impossible? Not at the nation-state level. Not in communist countries where everything is property of the government by default. Not in capitalist countries like the United States, where entire nuclear facilities are replicated in secret. [0] [0] https://www.businessinsider.com/the-us-built-a-secret-replica-of-irans-nuclear-facilities-to-help-gain-an-edge-in-nuclear-talks-2015-4 https://www.businessinsider.com/the-us-built-a-secret-replic...
ryanmarsh 8y ago
“There are so many far easier ways to do the same job. It makes no sense—from a capability, cost, complexity, reliability, repudiability perspective—to do it as described in the article.” Considering the US went to the trouble of wiring the North Atlantic for sound to catch Russian submarines during the cold war, and tapped undersea cables using divers and submarines, this is so implausible for a nation state? Large state actors specialize in activities for national defense that make "no sense—from a capability, cost, complexity, reliability, repudiability[sic] perspective".
- crististm 8y ago
  There are even more recent examples with software: Stuxnet
zbentley 8y ago
> a binary file that stored administrator passwords in plaintext. I understand what they mean, but that sentence still hurts to read.
toss1 8y ago
>Supermicro boards were so bug ridden, why would hackers ever need implants? Ummm, because if you need your hack to be reliable, you can't rely on someone else's bugs to be there when you need them. You never know when they'll be fixed, or just replaced by new bugs. A long time ago when setting up computers and networks was driver version hell, we had a short list of manufacturers' computers that we'd do setup included in the price instead of on-the-clock. This came about when a shipment of about 20 Dell computers, all supposedly of the exact same model# and revision, required about about 11 different setups, because the various chips on the board were different. They were clearly just using the chip-of-the-week>from whatever supplier was cheapest -- great for their price points, but every variant required a different driver for some subsystem. So the list was created and Dell was not on it (it was IBM, Compaq, HP, DEC, to show when this was). That's solved now by hiding it with the much more automated OS and networking setups, but it is easy to see how the Chinese spies would be in the same situation -- some buggy boards are wonderfully exploitable, but how do you tell that the version going to your target wasn't changed by some revision that wasn't even noted in the Rev- listings? Better to insert your own bug if you want to actually get the job done.
rbanffy 8y ago
It depends on what you want to do. If you want to extract information from a specific network. maybe custom firmware is a good option. If you want to just disable a very large number of machines to create economic damage or cripple infrastructure, a hardware implant would do just fine. And you wouldn't need to be very careful as to where it ends - if you make enough of them, they'll be everywhere. If 1% of all MacBooks have a similar backdoor, there are about a dozen at my building.
tannhaeuser 8y ago
If Supermicro boards are bug-ridden, then I'd expect other manufacturer's boards to be equally bad or worse. I don't have a reason to defend Supermicro or some such, but where else in retail do you get specialized server boards like Supermicro sells? When Opteron was relatively new in 2004, I bought a two-socket board from Supermicro as the alternatives from well-known Taiwan manufacturers (ASUS, MSI, etc.) weren't as sophisticated (hadn't the PSUs and PSU connectors, power ratings, and rack-mount/tower convertible enclosures).
HillaryBriss 8y ago
One researcher criticizes this type of hardware attack, saying: Once discovered, such an attack would be burned for every affected board as people would replace them. But this article also points out a case where, even after SuperMicro had published a patch to a serious BMC firmware vulnerability, 32,000 servers in the wild had not been updated a year later. So, if software updates aren't always speedily/reliably deployed in the wild by customers, can we really expect hardware to be speedily replaced?
alexeiz 8y ago
The purpose of this whole thing was to manipulate the market. Super Micro stock fell 50% and still has not recovered since October 4th. Before the report its trading volume was invisible. After the report the volume experienced almost 2 orders of magnitude increase.
xkcd-sucks 8y ago
Okay, crazier lead-foil hat time: what if this story is a crappy hoax intended to discredit/prevent from publication a real story with similar details?
- bdamm 8y ago
  I don't follow. If anything it makes the parallel story easier to publish as a sort of "me too" (no disrespect.) Maybe it takes away from the firmware hacking version of the story because now folks are looking at components as being the source of hacks and not the firmware on the components, leading to a false sense of security when they invest mightily in analyzing components with X-rays? I could see that outcome as being plausible. If the ultimate outcome is simply to change corporate priority towards futile component verification and away from firmware verification then indeed the firmware verification vector remains safe for the attacker.
  - xkcd-sucks 8y ago
    More like "Oh look, another story about extra components inserted by a big state actor. Bloomberg just got burned for this, I'm not going to risk my/my organization's reputation on the slight chance that this one is real" Or, "Extra chips on the motherboard? You're about a month behind the news cycle and didn't you hear it was all BS anyway" But the false sense of security interpretation is plausible too. I wonder who holds conferences on the cutting edge research of manipulating media
deleted 8y ago
[deleted]
jiveturkey 8y ago
Very fair article. Raises doubt in a very productive way, not the he-said she-said of previous rebuttals. I'd go further to say it isn't just about the accuracy of the bloomberg piece, but implies bad things about their journalistic integrity. I mean, get real, Ars doesn't have an investigative journalism team. The one-sidedness of the bloomberg article becomes very apparent.
zelon88 8y ago
Not saying I believe in one side or the other, but from a standpoint of avoiding detection I think firmware hacking goes out the window. A deep-pocketed attacker isn't going to risk flashing the firmware with a non-oem one on a brand new board leaving the factory. That probably gets quality inspected somehow later on anyway whereas a visual inspection is just a rubber stamp (IE: OK if the box isn't crushed or wet). Not to mention a customer in the field who experiences problems is likely to report their firmware version to Supermicro support, whose poking around could expose the entire project. There was an article recently about how hardware is "magic" and the IT world mostly takes it for granted. Putting an extra chip on the board but making it completely transparent to software debugging techniques is the best way to go. The board is almost certainly going to be flashed at least once and probably audited several times in it's lifespan by IT, but the hardware is never going to get more than some compressed air blown on it. Nobody repairs these things at the component level on a scale that matches how frequently firmware gets flashed or checked out.
- eridius 8y ago
  Maybe for smaller companies, but Apple is very paranoid and AIUI does indeed inspect the hardware to make sure it hasn't been tampered with. I know less about Amazon in this regard but I would expect Amazon to do at least some level of hardware inspection to detect tampering as well.
  - vermilingua 8y ago
    Forgive my ignorance, but what hardware does Amazon sell?
    - mkl 8y ago
      We're talking about server hardware used by Amazon and Apple, not what they sell.
  - wahern 8y ago
    They also do extensive code reviews, including of imported open source, yet their software is hardly bug free. Bugs can linger for years, often fixed only because someone stumbled upon odd behavior. Spotting a tiny chip sitting on the SPI bus that looks identical to a bunch of other chips? That doesn't do anything unless it's tickled in just the right way? If you believe Amazon and Apple are even remotely capable of protecting against that.... The solution to these problems is to put critical code and critical secrets on discrete, simple SoCs where you actually have a chance of defending both hardware and software attacks. Apple and Amazon understand this because they already do it. The difficulty is building your software systems (firmware, kernel, etc) to make use of these secure elements, not to mention making them available for ad hoc application software. It's an extremely difficult integration problem, and even when you succeed you haven't. For example, AFAIU Amazon's servers have secure elements to perform attestation of the box; it's utilized by their hypervisors to authenticate VMs for things like KMS. But it can't actually protect the data in the VM itself, such as the secrets obtained by dint of the attestation. It can't even prevent taking control of the hypervisor. All it does is help Amazon define a fixed security parameter--that you can't impersonate their hardware nodes on the network. That's extremely useful, but ultimately extremely limited.
csours 8y ago
BMC bug story time: I was working on automating health checks, and I needed some information from a BMC. The information was provided in XML format... fixed width. It's like something produced the document, and then output it to console, then copied from console to web service output.
ThenAsNow 8y ago
While it's ultimately going to help to shame vendors regarding their poor security practices, it's really irritating and unfortunate this is all being framed as a Supermicro issue. How about the other companies in the same market space, like Tyan, that I'm sure are no better? For that matter how about the "Tier 1" OEMs like Dell and HP - how well-written are their BMC firmwares?

182 points by drewg123 8y ago

deleted 8y ago

[deleted]

qaq 8y ago

I would guess that large companies are refreshing with known good firmware before deploying servers? So while described approach is easier prob will not get attacker as much.

bdamm 8y ago
You would guess wrong because a) large companies are profit motivated and that is a cost that can be cut and b) even if you "refresh" firmware, what actually is happening in there? You don't really know, so the "refresh" may not be as effective at wiping out the attack as you had hoped.
vel0city 8y ago
Most BMC updates are handled in software on the BMC. You're giving the BMC a new image file to write and trusting the BMC to actually write it. Who's to say the BMC is dutifully writing that image to the flash memory? Who's to say it doesn't re-infect the image before writing? Even if you do directly connect to the flash module and directly write to it through SPI, if the attack is being loaded by an additional module between the flash memory and the BMC, it could still inject additional data into the BMC's boot. If you're not physically listening to the SPI data being transmitted or knew what to look for in the final environment of the BMC, you wouldn't know it had happened.

throwaway290342 8y ago

Okay, crazy tinfoil hat time: what if this story is a plant from a particular part of the Chinese government (like PLA Unit 61398), designed to give the impression of the ability to disrupt global supply chains and to build respect through fear? If all of these unnamed sources are unnamed because they were adversarial members impersonating government officials, then that would make a little more sense why current government bodies are not just staying mum, but actually denying knowledge of the story. With the software attacks being much more feasible as the Ars article points out than a hardware attack, then it would also make it so that the vehement denials from affected companies would be true as well. The whole thing could be a large disinformation campaign to strike at the very core of what many would otherwise consider reasonable security.

wycy 8y ago
It seems like this would be a really bad idea. Scaring companies away from buying Chinese-manufactured products couldn't possibly be worth the respect through fear.
nakedrobot2 8y ago
No. A better tinfoil hat theory is that the whole story was fabricated by Russia, to (you know, as always) sow chaos.
jack6e 8y ago
But the effect of that would be to cause massive distrust of Chinese suppliers and cause a shift away from electronics being produced there. IC and cyber experts generally identify the Chinese as using intelligence operations for primarily economic purposes, as compared to Russian/Iranian/North Korean objectives being military or political. A Chinese military intelligence agency using cyber espionage to intentionally disrupt one of the most significant export industries of the Chinese economy does not seem likely, nor does it seem to provide such an out-sized strategic benefit as to be worth the economic cost.
- throwaway290342 8y ago
  Good point, I agree with that thinking. But the actual execution of such a hardware-based attack would surely be discovered at some point anyway, and risk the same negative outcome. So then that would leave the only possible conclusion that the story just isn't true at all. In the end, none of it makes clear sense...
  - jack6e 8y ago
    The difference is two-fold: actively planting a fake story means that first, the espionage is fake and thus no real intelligence can be gathered, so the only benefit is the hypothetical respect you suggested; second, the story will definitely get out, thus the potential for the negative effect is innately 100%. However, as a real intelligence operation the cost/benefit analysis is inverted, because there is a real, tangible benefit to extracting possibly sensitive commercial and national security information. And while an eventual discovery is always a possibility, it seems care was taken to ensure it would only be a small possibility, and that in any case it would be in the future, hopefully after a large amount of useful data is extracted. So in the planted story hypothesis, there is certainty of negative outcomes with only the potential for positive outcomes, and those only intangible, while in the this-is-real hypothesis, there is near certainty of some tangible benefit with good probability of significant tangible benefit, with only a potential, distant, deniable risk of negative outcome.

rayiner 8y ago

I feel like this article reflects some significant technical confusion. The BMC is supposed to be on a trusted network inaccessible from the outside. I've always viewed authentication on the BMC as being like the numeric lock on luggage--it's designed to keep honest people honest, not for real security. Being able to bypass the BMC security is really not a big deal. What the Bloomberg article says about the hardware exploit is much worse: > > The rogue instructions, Bloomberg reported, caused the BMCs to download malicious code from attacker-controlled computers and have it executed by the server’s operating system. It's using the fact that the BMC has unfettered access to the rest of the machine to compromise the code running on the server itself. That's valuable even if the BMC itself is on a private network inaccessible to the attacker.

stephengillie 8y ago
BMCs like DRAC or iLO are invaluable when you have hundreds or thousands of fresh servers with no OS. The BMC lets you mount an OS or hypervisor ISO in a way reminiscent of DaemonTools et al., and update bios and other firmware from a shared network folder. I'm pretty sure there's even an API to develop against.
- rayiner 8y ago
  BMC's are great--all my home builds have them because I'm too old to be fiddling around trying to figure out why a computer won't boot an installer from a USB key. But even on my home network the BMC's are on a separate switch on a subnet that doesn't have internet access except through a VPN gateway.
  - jacquesm 8y ago
    > except There's your problem.
    - xyzzy123 8y ago
      I think the parent meant, you have them on a subnet with no default route, but have a vpn / management system with one interface pointing in to the management network. You can get in via the vpn but they can’t get out. It’s a pretty common configuration.

mannykannot 8y ago

While I don't think Bloomberg's story looks very plausible, perhaps one motivation for cryptic hardware modification at a time when firmware weaknesses were being discovered might be precisely because the easier-to-exploit firmware weaknesses were being discovered, and so might not be exploitable much longer? It might not have seemed plausible that the vulnerabilities would be discovered but then not fixed to the extent that, it turns out, they were not.

mhjas 8y ago

I don't really see why everyone is calling this implausible. Modchips have been around for at least 15 years. The idea of the clipper chip is 25 years old. At every hacker conference there are people "hacking" devices by various buses or interfaces. If there is anything working against the Bloomberg story it is that it is too plausible. Often reality clashes with imagination, but the Bloomberg story contains almost everything you could imagine happening.

ryanmarsh 8y ago
Exactly. How much has the US spent on the F-35? How much has China spent on making artificial islands? Yet engineering a chip and bribing/threatening a few factory workers is beyond the pale?
- deleted 8y ago
  [deleted]
- kevin_thibedeau 8y ago
  Bloomberg's claim is that a miniature device used for RF analog electronics was coopted and inserted into a board that would never have such a part designed in. This requires modifying the board artwork, the pick and place config, any automated inspection and test equipment, and adding a foreign part reel to the supply chain. It is much easier to compromise firmware directly or modify ICs that are already part of the design. The risk of being caught is much lower and it would be stupid to attempt anything more elaborate.
  - pnematoad 8y ago
    This requires [et c.] Or, they could run their own fabrication facilty, where they can exert total control over the production line, in total secrecy, and you'd never know the difference, or notice a sacrifice in the fidelity of replication. Think that's impossible? Not at the nation-state level. Not in communist countries where everything is property of the government by default. Not in capitalist countries like the United States, where entire nuclear facilities are replicated in secret. [0] [0] https://www.businessinsider.com/the-us-built-a-secret-replica-of-irans-nuclear-facilities-to-help-gain-an-edge-in-nuclear-talks-2015-4 https://www.businessinsider.com/the-us-built-a-secret-replic...

ryanmarsh 8y ago

“There are so many far easier ways to do the same job. It makes no sense—from a capability, cost, complexity, reliability, repudiability perspective—to do it as described in the article.” Considering the US went to the trouble of wiring the North Atlantic for sound to catch Russian submarines during the cold war, and tapped undersea cables using divers and submarines, this is so implausible for a nation state? Large state actors specialize in activities for national defense that make "no sense—from a capability, cost, complexity, reliability, repudiability[sic] perspective".

crististm 8y ago
There are even more recent examples with software: Stuxnet

zbentley 8y ago

> a binary file that stored administrator passwords in plaintext. I understand what they mean, but that sentence still hurts to read.

toss1 8y ago

>Supermicro boards were so bug ridden, why would hackers ever need implants? Ummm, because if you need your hack to be reliable, you can't rely on someone else's bugs to be there when you need them. You never know when they'll be fixed, or just replaced by new bugs. A long time ago when setting up computers and networks was driver version hell, we had a short list of manufacturers' computers that we'd do setup included in the price instead of on-the-clock. This came about when a shipment of about 20 Dell computers, all supposedly of the exact same model# and revision, required about about 11 different setups, because the various chips on the board were different. They were clearly just using the chip-of-the-week>from whatever supplier was cheapest -- great for their price points, but every variant required a different driver for some subsystem. So the list was created and Dell was not on it (it was IBM, Compaq, HP, DEC, to show when this was). That's solved now by hiding it with the much more automated OS and networking setups, but it is easy to see how the Chinese spies would be in the same situation -- some buggy boards are wonderfully exploitable, but how do you tell that the version going to your target wasn't changed by some revision that wasn't even noted in the Rev- listings? Better to insert your own bug if you want to actually get the job done.

rbanffy 8y ago

It depends on what you want to do. If you want to extract information from a specific network. maybe custom firmware is a good option. If you want to just disable a very large number of machines to create economic damage or cripple infrastructure, a hardware implant would do just fine. And you wouldn't need to be very careful as to where it ends - if you make enough of them, they'll be everywhere. If 1% of all MacBooks have a similar backdoor, there are about a dozen at my building.

tannhaeuser 8y ago

If Supermicro boards are bug-ridden, then I'd expect other manufacturer's boards to be equally bad or worse. I don't have a reason to defend Supermicro or some such, but where else in retail do you get specialized server boards like Supermicro sells? When Opteron was relatively new in 2004, I bought a two-socket board from Supermicro as the alternatives from well-known Taiwan manufacturers (ASUS, MSI, etc.) weren't as sophisticated (hadn't the PSUs and PSU connectors, power ratings, and rack-mount/tower convertible enclosures).

HillaryBriss 8y ago

One researcher criticizes this type of hardware attack, saying: Once discovered, such an attack would be burned for every affected board as people would replace them. But this article also points out a case where, even after SuperMicro had published a patch to a serious BMC firmware vulnerability, 32,000 servers in the wild had not been updated a year later. So, if software updates aren't always speedily/reliably deployed in the wild by customers, can we really expect hardware to be speedily replaced?

alexeiz 8y ago

The purpose of this whole thing was to manipulate the market. Super Micro stock fell 50% and still has not recovered since October 4th. Before the report its trading volume was invisible. After the report the volume experienced almost 2 orders of magnitude increase.

xkcd-sucks 8y ago

Okay, crazier lead-foil hat time: what if this story is a crappy hoax intended to discredit/prevent from publication a real story with similar details?

bdamm 8y ago
I don't follow. If anything it makes the parallel story easier to publish as a sort of "me too" (no disrespect.) Maybe it takes away from the firmware hacking version of the story because now folks are looking at components as being the source of hacks and not the firmware on the components, leading to a false sense of security when they invest mightily in analyzing components with X-rays? I could see that outcome as being plausible. If the ultimate outcome is simply to change corporate priority towards futile component verification and away from firmware verification then indeed the firmware verification vector remains safe for the attacker.
- xkcd-sucks 8y ago
  More like "Oh look, another story about extra components inserted by a big state actor. Bloomberg just got burned for this, I'm not going to risk my/my organization's reputation on the slight chance that this one is real" Or, "Extra chips on the motherboard? You're about a month behind the news cycle and didn't you hear it was all BS anyway" But the false sense of security interpretation is plausible too. I wonder who holds conferences on the cutting edge research of manipulating media

deleted 8y ago

[deleted]

jiveturkey 8y ago

Very fair article. Raises doubt in a very productive way, not the he-said she-said of previous rebuttals. I'd go further to say it isn't just about the accuracy of the bloomberg piece, but implies bad things about their journalistic integrity. I mean, get real, Ars doesn't have an investigative journalism team. The one-sidedness of the bloomberg article becomes very apparent.

zelon88 8y ago

Not saying I believe in one side or the other, but from a standpoint of avoiding detection I think firmware hacking goes out the window. A deep-pocketed attacker isn't going to risk flashing the firmware with a non-oem one on a brand new board leaving the factory. That probably gets quality inspected somehow later on anyway whereas a visual inspection is just a rubber stamp (IE: OK if the box isn't crushed or wet). Not to mention a customer in the field who experiences problems is likely to report their firmware version to Supermicro support, whose poking around could expose the entire project. There was an article recently about how hardware is "magic" and the IT world mostly takes it for granted. Putting an extra chip on the board but making it completely transparent to software debugging techniques is the best way to go. The board is almost certainly going to be flashed at least once and probably audited several times in it's lifespan by IT, but the hardware is never going to get more than some compressed air blown on it. Nobody repairs these things at the component level on a scale that matches how frequently firmware gets flashed or checked out.

eridius 8y ago
Maybe for smaller companies, but Apple is very paranoid and AIUI does indeed inspect the hardware to make sure it hasn't been tampered with. I know less about Amazon in this regard but I would expect Amazon to do at least some level of hardware inspection to detect tampering as well.
- vermilingua 8y ago
  Forgive my ignorance, but what hardware does Amazon sell?
  - mkl 8y ago
    We're talking about server hardware used by Amazon and Apple, not what they sell.
- wahern 8y ago
  They also do extensive code reviews, including of imported open source, yet their software is hardly bug free. Bugs can linger for years, often fixed only because someone stumbled upon odd behavior. Spotting a tiny chip sitting on the SPI bus that looks identical to a bunch of other chips? That doesn't do anything unless it's tickled in just the right way? If you believe Amazon and Apple are even remotely capable of protecting against that.... The solution to these problems is to put critical code and critical secrets on discrete, simple SoCs where you actually have a chance of defending both hardware and software attacks. Apple and Amazon understand this because they already do it. The difficulty is building your software systems (firmware, kernel, etc) to make use of these secure elements, not to mention making them available for ad hoc application software. It's an extremely difficult integration problem, and even when you succeed you haven't. For example, AFAIU Amazon's servers have secure elements to perform attestation of the box; it's utilized by their hypervisors to authenticate VMs for things like KMS. But it can't actually protect the data in the VM itself, such as the secrets obtained by dint of the attestation. It can't even prevent taking control of the hypervisor. All it does is help Amazon define a fixed security parameter--that you can't impersonate their hardware nodes on the network. That's extremely useful, but ultimately extremely limited.

csours 8y ago

BMC bug story time: I was working on automating health checks, and I needed some information from a BMC. The information was provided in XML format... fixed width. It's like something produced the document, and then output it to console, then copied from console to web service output.

ThenAsNow 8y ago

While it's ultimately going to help to shame vendors regarding their poor security practices, it's really irritating and unfortunate this is all being framed as a Supermicro issue. How about the other companies in the same market space, like Tyan, that I'm sure are no better? For that matter how about the "Tier 1" OEMs like Dell and HP - how well-written are their BMC firmwares?