9 ms·
Supermicro server motherboards can be infected with unremovable malware
- qmr 1y agoNSA has entered the chat.
- mindslight 1y agoEvery modern motherboard comes pre-installed with unremoveable malware. Vulnerabilities just let different actors try to be king of the hill. The real answer is that all these persistent storage areas (eg flash chips) need to be documented and writable by the user. Any boot integrity needs to be done on top of an open environment, rather than continuing to rely on security through obscurity.
- bri3d 1y agoI see the point you're trying to make but it's really very orthogonal to this issue: this was an issue in a documented part of a documented feature, it's not some "deeply embedded" system management controller with no documentation, it's "the signed firmware update feature in the big obvious selling point where the server has a backdoor management interface was broken."
- hollerith 1y agoPeople like to criticize secure boot around here, but it prevents these kinds of infections (provided of course there are no vulnerabilities in the implementation of secure boot). Yes, in theory it is possible to prevent these kinds of infections without resorting to secure boot (e.g., by insisting that all the suppliers of components of the motherboard start designing components that cannot be pwned) but so far all the computers you have actually been able to buy that are immune to these kinds of infections achieve that immunity with secure-boot technology.
- amluto 1y agoCan you please explain how Secure Boot helps at all to mitigate this type of attack? I don’t see how it makes it harder to execute the attack, and I don’t see how it has any particular effect on the capabilities of the attack once executed. To the contrary, a BMC compromise of this sort seems like it should be able to arbitrarily override secure boot settings. It seems to me that, in this situation, secure boot’s only role is to provide a false sense of security, which could make recovery from the attack less likely. In contrast, verified boot might somewhat mitigate the damage from being able to use the BMC to write arbitrary data to the SPI flash chip. Emphasis on might — at best I expect that it would require an attacker to be a bit more creative in how they design their exploit payload.
- bri3d 1y agoI have never seen someone make a distinction between Secure Boot and boot verification on UEFI/x86, although I suppose it’s a valid one? I suspect the parent post is referring to Secure Boot in the colloquial sense it’s usually used for on x86: validation of the UEFI boot binary using Secure Boot keys followed by OS level trust chaining, usually accomplished by entangling PCR state with disk encryption in some way. And I think this would deliver a slight level of protection from the BMC: tampering with the firmware image or key enrollment / secure boot state _should_ both break the UEFI root of trust and alter the PCR state and break everything downstream. Of course, all UEFI implementations are holier than Swiss cheese and there are probably a lot of ways to use the BMC to dump or modify memory post-boot anyway, but it does do something.
- amluto 1y agoYou mean the same Secure Boot that I mean. This is a software mechanism (almost universally implemented as an unholy mixture of ordinary code in firmware and a pile of immensely complex SMM code that, IMO, is entirely unnecessary if the hardware or the BMC gives even the slightest bit of help). And Secure Boot is implemented in, and configured by, the firmware that the BMC can overwrite at its whim while entirely bypassing all the fancy CPU-hardware and SMM protections that are supposed to prevent writing arbitrary data to it. To the extent that a mechanism not controlled by firmware will detect such an attack and extend PCRs accordingly before executing a single instruction from the compromised flash chip, it might partially mitigate attacks. But that part isn’t Secure Boot.
- toast0 1y agoHow does secure boot help? If you control the BMC, you can enroll whatever keys you want. The BMC usually has full access to system memory as well, so if you can get the timing right, you could replace the secure boot verified image with your own after verification. Also, re: BusinessWeek, hey look a hardware backdoor installed on servers. Pretty sure IPMI vulnerability fits the bill for most of what was described.
- tiffanyh 1y agoIs this related to controversial Bloomberg 2021 piece about China hacking Supermicro servers? https://www.bloomberg.com/features/2021-supermicro/ https://www.bloomberg.com/features/2021-supermicro/
- dlcarrier 1y agoNot at all, that piece described a supply chain attack replacing a component with a look-alike part analyzing tens to hundreds of gigabits if data, in a form factor so small that it wouldn't be physically possible without semiconducting fabricating processes years in advance of what existed at the time. What this article is describing is something far more likely— a firmware attack that doesn't require specialized hardware.
- bri3d 1y agoWasn’t the implant supposedly (illogically) implanting custom BMC firmware? This actually always struck me as a somewhat unbelievable part of the story: why install a hardware implant when you could just clip the flash chip and implant something without a physical trace?
- wmf 1y agoFlash can always be reflashed (you'd lose the implant if the customer does any firmware update) but a separate implant chip can remain infected forever.
- bri3d 1y agoThat’s fair - the first thing I’d put in my implant firmware would be a fake firmware updater, but I suppose trading guaranteed persistence for physical detection would be a reasonable tradeoff in some places.
- dlcarrier 1y agoThe Bloomberg article specifically mentioned the attack occurring through the addition of extra hardware: “In early 2018, two security companies that I advise were briefed by the FBI’s counterintelligence division investigating this discovery of added malicious chips on Supermicro’s motherboards,” said Mike Janke, a former Navy SEAL who co-founded DataTribe, a venture capital firm. “These two companies were subsequently involved in the government investigation, where they used advanced hardware forensics on the actual tampered Supermicro boards to validate the existence of the added malicious chips.”
- dlcarrier 1y agoUnremovable through remote access, but reliably removed by reflashing the firmware through JTAG, or whatever interface used during manufacturing to initially load the firmware.
- deleted 1y ago[deleted]
- vetrom 1y agobios flashes are typically preflashed prior to soldering to boards -- some vendors route jtag or spi contacts, but you're more likely to need a vampire/pogo clip on for the TSOP or equivalent chip, or have fun with resoldering the bios flash if you're needing to recover from this. It's not impossible to do in the field, but you can't really count on vendors surfacing that interface usually.
- bpye 1y agoDo any server boards still socket the BIOS flash?
- BobbyTables2 1y agoWouldn’t matter if the BMC can infect the BIOS flash…
- arianvanp 1y agoYes ASRock Rack has socketed bios and BMC flash. At least on their Ampere motherboards
- jacquesm 1y agoThis is a real problem with some laptops. You can't always get to the chip contacts.
- dlcarrier 1y agoI've worked with automated EEPROM/Flash programmers (earlier versions of this line: https://www.bpmmicro.com/device-programmers/automated-programmers/ https://www.bpmmicro.com/device-programmers/automated-progra...), and used pre-programming services from distributors, like Digi-Key, but that was the exception. It's almost exclusively faster, cheaper, and easier to load firmware from a test fixture. You need to test the assembly anyway, and it's much easier to update a test procedure, when a new firmware is developed, than to update and track inventory of pre-programmed devices, especially when different firmware versions are needed for different hardware variations. Pogo pins are only really needed for mass production, especially for reducing repetitive stress injuries. For one-off updates, if a header isn't populated, it's easy to hold an unsoldered header in place, for long enough to flash an update.
- iberator 1y agoBusiness idea: eprom based firmware
- nine_k 1y agoAnd put the EPROM in a socket, like it's 1987? Some motherboards just have a physical jumper that prevents BIOS flashing. This happens infrequently enough as to warrant it for one server, or 10 servers, or maybe 100 servers. Likely unpractical for 1000 servers though.
- boredatoms 1y agoIf they can put the jumper on the exterior it might be feasible, if its inside its out of the question if you have to unrack the chassis to change. Rolling in a server lift for an 8u thats half full of copper is not a nice process
- c0balt 1y agoThe next idea, a second oob management for the first oob managemen. A BMC for the BMC. It only does updates and maybe credential management. Make this one simple enough and add an EPROM for it. Effectively a security chip for the oob. Extra points for secure enclave-like verified boot.
- wmf 1y agoThe security chip for the BMC is called root of trust.
- transpute 1y agoOpenCompute (OCP) Caliptra is an effort by hyperscalers, AMD and others to enforce a platform root of trust with OSS firmware and open silicon, mandating dual signature by server OEM and hyperscaler customer. The platform RoT is responsible for validating device firmware and OS boot, https://www.youtube.com/watch?v=p9PlCm4tLb8&t=2764s https://www.youtube.com/watch?v=p9PlCm4tLb8&t=2764s > Often we see.. great security.. compromised by other great ideas for mgmt and other things.. starts to weaken its security posture.. want to keep Caliptra very clean [via OSS firmware transparency]
- kj4ips 1y agoDo we know if this is also the case for other systems that use Aspeed/ami BMCs, or if the key pair in question is exclusive to SM?
- buildbot 1y agoYes it is. Supermicro is one of the only vendors that tries to prevent this attack at all through RoT. Other vendors you can flash whatever unsigned firmware you want. It’s very useful for adding in microcode for intel engineering samples, or malware…
- gpapilion 1y agoThis is not true. Almost all firmware is signed by every vendor, and there are standards from Intel and amd on implementation of code signing. Look up Intel pfr.
- sgt 1y agoSupermicro just can't catch a break!
- SoftTalker 1y ago"If a potential attacker already has administrative access to the BMC..." Then you've already lost. The BMC needs to be ideally on a physically isolated network, or at least a separate one that has no route from the outside nor on the machine itself.
- Aurornis 1y agoThat’s a cop-out. It should be the case that even administrator access should not be abusable to implant permanent backdoors. Anything that makes privileges escalation exploits more damaging is a real problem. I’m getting tired of how these are being dismissed as if admin access should mean that you can ignore any security issues. There are things that even admin accounts should not be able to change at the hardware level, or if they can they must be reversible in the future by another user with admin access. > The BMC needs to be ideally on a physically isolated network, or at least a separate one that has no route from the outside nor on the machine itself. This is good practice but it shouldn’t excuse poor security at the hardware level. Supermicro motherboards also commonly default to having a feature that bonds the BMC network interface to one of the main NICs if you don’t plug a cable into the BMC interface. It’s common for people to be surprised that their BMC is exposed on their main network because they didn’t plug in a cable on the BMC NIC port at all.
- SoftTalker 1y agoI don't disagree, but if you have admin access to the BMC you can access the console, reboot the machine into single-user mode or even into another OS entirely, and then implant any malware you want, wipe the storage, etc. Some of the Supermicro boards don't even have a separate BMC NIC, the only choice is to bond it to one of the main NICs or sacrifice one of them to be BMC only. I try to pay attention to that now after being surprised by that once on some servers we bought.
- Aurornis 1y ago> I don't disagree, but if you have admin access to the BMC you can access the console, reboot the machine into single-user mode or even into another OS entirely, and then implant any malware you want, wipe the storage, etc. Yes, all of which can be reversed by another admin in the future. That is expected. It should not be the case that getting admin access one time can result in modifying the hardware in a way that can’t be reversed by future admin, short of physically reflashing the chip on the board.
- temp0826 1y agoMy favorite supermicro facepalm will always be when you could set the IPMI encryption cipher to "none" (ipmitool -C0) and bypass actually needing any password at all. (Though I don't think this was unique to supermicro actually?)
- transpute 1y agoWith some server vendors, if you don't connect an ethernet cable to the BMC, it can intercept BMC-targeted traffic from the OS-connected ethernet port.
- nyrikki 1y agoDell also had this problem, you still needed to provide a password, it simply didn’t check the password.
- kj4ips 1y agoPretty much all of them allow unrestricted access from KMS from factory, tough all of them have a way to disable it once configured, and HPE even throws shade until it's limited. KMS only works from the host itself.
- buckle8017 1y agoThis is true for nearly all computers. Regular desktop motherboards can be flashed from windows. Yawnnnn.
- protimewaster 1y agoDo nearly all computers really have a signature bypass flaw?
- buckle8017 1y agoThey don't have signatures.
- rubatuga 1y agoHP begs to differ
- thisisit 1y agoAm I having a deja vu or wasn't this discussed earlier like 5-6 years ago?
- TacticalCoder 1y agoYup definitely. It was about SMCI too. It's not just you: at least two of us are having deja vu.
- formerly_proven 1y agoIf you expect the BMCs of other vendors to be significantly more secure than the average home router from 2010 I have several bridges you may be interested in.
- tecleandor 1y agoDifferent. In 2018 there was articles [0] talking about hacked Supermicro servers being in the wild that had (simplifying) extra ICs in their motherboard, like some short of supply chain attack. But AFAIK, tangible evidence never surfaced. [1] -- 0: https://news.ycombinator.com/item?id=18176620 1: https://news.ycombinator.com/item?id=19910537
- alex77456 1y agoI remember it too, but vaguely remember evidence of compromised boards
- PoorlyNamed 1y agoThe story is called "The Big Hack" and was run by Bloomberg in 2018. All the major companies named in the story conducted investigations and never found anything, but the story was never retracted. https://www.bloomberg.com/2018-the-big-hack https://www.bloomberg.com/2018-the-big-hack
- 4ggr0 1y agowonder who got short-options on SMCI in 2018 :)
- metalman 1y agoisn't the simple answer to infect all.of these servers with a benign infection, a vacine if you will, that blocks further attempts to remotely flash the bios
- jiggawatts 1y agoSomething to note is that there are special BMC-less motherboards that are made for organisations like the NSA. That should tell you everything you need to know about the security risks involved.
- tinco 1y agoAnd of course there's oxide.computer making motherboards with full control of the entire hardware stack running open source software specifically for customers like the Idaho National Laboratory.
- selkin 1y agoOxide’s efforts are commendable, but I don’t know what compliance regime their customer adheres to. In other words, it’s a neat feature, but maybe not one many customers actually request.
- tinco 1y agoAs far as I can tell it's one of their most significant features, and certainly one of its most capital intensive. And I don't know how many startups land institutions like INL as their launching customer, and enough other big enterprise leads to warrant their follow up investment. If you remove that component from their value prop, they're not that much different from Dell.
- bcantrill 1y agoJust a quick point of clarification that while our boot architecture is very important (e.g., a service processor in lieu of a BMC, the elimination of UEFI entirely, etc.), we are quite a bit different from Dell beyond that. There are certainly many hardware-level differentiators (e.g. DC busbar-based design, blindmated networking, built-in switch, etc.) but the big differentiator is really what these things allow: entirely integrated software. The Oxide rack comes with all of the software to run elastic infrastructure (that is, the distributed system that comprises the control plane), including switch software, storage software, etc. And then (critically!) the capacity to update all of this.[0] All of it is a far cry from the offerings of Dell/HPE/Supermicro, which rely on others to provide the software that turns the hardware into real infrastructure. [0] https://oxide.computer/blog/systems-software-in-the-large https://oxide.computer/blog/systems-software-in-the-large
- xenadu02 1y agoIf you've never used the BMC on a server... it is all 100% garbage. Software mostly written by embedded folks who haven't got a clue. It is absolutely garbage software on the whole (and no matter what vendor you get the board from). Go ahead and hit up the web interface then do a bit of "View Source". If you are imagining the rest of that stack is any better than my friend have I got a Beautiful Bridge in Brooklyn to sell you! If it were me I'd assume the majority of BMC firmware out there from all vendors: 1. Is full of many many exploitable vulnerabilities 2. To the extent they patch holes it will be whack-a-mole because the economics do not permit large investments in software quality. 3. Many server owners will never install a patch anyway.
- preisschild 1y agoThere is the open source OpenBMC software nowaydays, which is pretty good. Unfortunately, Supermicro doesn't use it yet for most of their servers. Probably because they sell an extremely expansive license for their own software so you can use the Redfish API.
- gpapilion 1y agoThe one vendor mentioned in the comments, AMI, is switching this code base to openbmc. Also it should be noted that often this software is system specific.
- citrin_ru 1y agoBMC software quality is low but what's the alternative? Without BMC it is more expensive to manage a fleet of servers. In a better word hardware vendors will publish specs to allow open-source BMC firmware but for some reason they resist this idea. Having only insecure BMC available a semi-separate management network (connected via a bastion host or a VPN) provides balance between cost and security.
- einsteinx2 1y ago> BMC software quality is low but what's the alternative? Dedicated KVM devices?
- snvzz 1y agoWhy the BMC is not designed around seL4, I cannot comprehend.
- justincormack 1y agoBecause people dont want to pay? Openbmc clearly has difficulty getting widespread support as nothing is documented.
- kj4ips 1y agoTwo main reasons I can think of: Most current BMC platforms are older than seL4 Most run on hardware that is not supported by seL4, or at least on hardware where it has not been validated. Not to mention that a task manager would be needed as well as tons of other services which aren't provided out of the box, and don't share the verification provided guarantees.
- egorfine 1y agoIt bugs me that it is near impossible to purchase a server without a backdoor present.
- Craighead 1y ago[dead]
- deleted 1y ago[deleted]