23 ms·
Making sense of the alleged Supermicro motherboard attack
- lovelearning 8y agoIt was reported that the security auditor used during Elemental's acquisition detected this compromise. I assume they found it in a randomly selected board. Either they were very lucky, or hundreds of boards were compromised. Now, I think all motherboard manufacturers - and especially high end server manufacturers like SM - use sophisticated automated tests and quality control on boards. Under what circumstances is it possible that SM's QC missed this out on so many boards? Won't it affect things like the power budget, weight, and latency? What do professional EE people here think?
- auslander 8y ago> what circumstances is it possible that SM's QC missed this out Rogue insider, paid or patriotic, likely both. > affect things like the power budget, weight, and latency All three would have no measurable change, falling into measurement error margin. Its a death sentence for Supermicro, nothing will help to restore trust. They should publish a detailed post-mortem analysis, though.
- toomuchtodo 8y ago> They should publish a detailed post-mortem analysis, though. More of an obituary at this point.
- gscott 8y agoThey took pains to say that this was happening with sub-contractors in China building motherboards in excess of what Taiwan could handle. So to build trust they can pull back on that extra manufacturing capacity and move it elsewhere maybe South Korea.
- auslander 8y ago> So to build trust they can You can't put the toothpaste back in the tube
- lovelearning 8y agoYeah, SM has the most to lose here. Regardless of whether the compromise is true or non-existent, they really should consider becoming more transparent about their QC, I think.
- LeonM 8y agoEE here, looking at the size of the chip, the weight and power difference is going to be too small to measure. Latency will probably not be affected when the chip is dormant
- scld 8y agoIf it replaced a DNP part, it could have been found at a flying probe test. Half the expected impedance on a few of those pins? Also AOI might have picked up a difference.
- dboreham 8y agoFlying probe testing is unlikely to be used for this type of product.
- pixl97 8y agoSounds like a randomized sample of boards needs tested against a reference then.
- nvusuvu 8y agoAOI would miss the chip if it were buried between layers or look exactly like a part it was replaced for.
- lovelearning 8y agoI wonder how the auditing team detected it. X-rays?
- bri3d 8y agoThey might have noted an anomaly in the SMC given the acknowledged SMC firmware exploits (maybe a beta test run?) and investigated the SPI lines to the SMC as a response.
- mediocrejoker 8y ago
- luma 8y agoDepends on where it was added. Doing it as part of the assembly process would require having access to the pick and place software along with being able to load reels of the implant into the machine. A well placed insider may be able to make that happen, and automated optical inspections would be fed by the modified pick and place program, so checks would pass as the new component is part of the build.
- lovelearning 8y agoThanks for that info. I assumed that once the boards came back to SM's US premises, a second level of optical and other tests are done by SM employees. But perhaps that's not - or was not - their actual workflow.
- admax88q 8y agoWould be pretty wasteful to wait until the hardware was shipped to the US to look for manufacturing defects.
- JohnJamesRambo 8y agoThat’s my main takeaway from all of this- Elemental is great at their job and didn’t just do a half assed attempt at an audit. They actually did what they were hired to do. I wonder how rare that is.
- coolspot 8y agoYou perhaps meant not Elemental, but Elemental's auditors is great at their job. > ... In late spring of 2015, Elemental’s staff boxed up several servers and sent them to Ontario, Canada, for the third-party security company to test, the person says. Nested on the servers’ motherboards, the testers found a tiny microchip ...
- deleted 8y ago[deleted]
- JohnJamesRambo 8y agoOh yes, my mistake yes, I thought Elemental was the name of the third party security company. Do we know the name of that company? They need some major kudos for this.
- neffy 8y agoThere used to be a Nortel Division up there that had an electro scanning microscope for this sort of thing - I imagine they´re still there under a different name.
- crispyambulance 8y ago> ... high end server manufacturers like SM - use sophisticated automated tests and quality control on boards. MFG test guy here (not supermicro!) Not necessarily. Automated production tests are there to configure and exercise the system and confirm it works as specified. Such tests are good at things like finding bad solder joints, pick-and-place mishaps, misconfiguration of firmware and weeding out product that fails functional test. That, by itself, is hard enough. Such tests are not _really_ able to detect things which no one is expecting, or worse, things which an adversary has specifically designed to avoid detection. Sure, if something gets "discovered" during a failure analysis, a test or process can be created to specifically address THAT problem in the future. To find "unknown unknowns", you need an audit of some kind and that is definitely different from production test.
- lovelearning 8y agoThank you for the insight. I was rather naively imagining that every board is subject to some kind of x-ray image matching with the original PCB design to find differences. Not that simple, I see.
- crispyambulance 8y agoSome manufacturers do X-ray boards. Typically this is done to check that BGA devices (https://en.wikipedia.org/wiki/Ball_grid_array https://en.wikipedia.org/wiki/Ball_grid_array) are soldered properly. Usually not done on every board, but only if there's a problem suspected. When it happens they tend to focus only on particular BGA components or suspect copper traces rather than the whole PCB.
- extrapickles 8y agoUsually you scan boards at a percentage determined by statistics (SPC can get complex). However the checks are automated, so you would have to perform the checks in a different trust zone (eg: different country).
- scoggs 8y ago
- ig1 8y agoThe Bloomberg article says suspicions were raised during a routine audit that resulted in a deeper audit. The routine audit could have uncovered hackers in Elemental's corporate network doing things like ensure certain machines go to certain customers, etc.
- qualsiasi 8y agoI'm no expert, but wouldn't be more efficient to completely replace the BMC with a malicious one? That way it should be virtually impossible to detect
- jonathankoren 8y agoIn true HN fashion, I’m not an expert either, but I can think of two plausible reasons. 1) Directly compromising via software the BMC is easy to detect if you bothered to look. (i.e. checksums don’t match), and would only last as long as the BMC isn’t flashed. 2) It’s simply currently easier to make a second component, rather than to integrate it into the main chip directly. Of course, if I was in charge of this project, I’d already be investing resources into exploring this option.
- dboreham 8y ago3) the purpose of the operation was to produce headlines and fear in the western press. A few bytes changed in some firmware, or even a malicious BMC design (presumably they're gate-array designs, not full-custom) wouldn't allow for macro photos of a chip the size of a grain of rice and all that.
- creeble 8y agoOr even why two SPI roms? Why not just one with special code? It's not like anyone is routinely reverse engineering the BMC boot code. It seems like an awful lot of provable trouble to go through (note that there is no physical evidence in the public eye yet) when you could do the same thing, at the factory, with just software.
- simplecomplex 8y agoWhere did all the boards in question go? Why wouldn’t a company notice any of the outbound traffic using firewalls? Two pieces of the story that don’t add up for me.
- radicalbyte 8y agoThere wouldn't be any weird outbound traffic; the attacker would use another cloud instance as a controller. They would use a US-based account (and IPs via VPNs) to control the controller.
- simplecomplex 8y agoThe traffic has to travel over wire as TCP/IP, regardless of what device generated it. It has to travel over wire somewhere! The device can’t magically communicate with China. Any outbound firewall could detect that, especially traffic going to ports that aren’t even open/used.
- true_religion 8y agoI think the idea is that the traffic goes over intranet to another computer that they legitimately control within the datacenter, then goes to China. Or if they can insert themselves into the maintenance team, they can retrieve their chips at a later date.
- lostmsu 8y agoAren't we talking about virtual machines running on this hardware?
- Filligree 8y agoUnless, just to provide a nightmare scenario, the routers that would detect it are also compromised.
- paulie_a 8y agoThe NSA did that, they literally intercepted routers en route. I'm guessing other agencies have too. Or you can just use the default credentials Cisco is addicted to leaving in their code.
- wyldfire 8y ago> There are few facts, and much supposition. Can't we watch it in action? Probe it and run the system, watch for activity on the SPI lines. Can we dissect it and look for how it might interact with the world? Does it have anything like an RF front end, or does it wait for some specific BMC/network activity before triggering?
- creeble 8y agoWe could do all of those things if anyone were to come forward with actual, physical evidence (i.e., a board). But so far no one has.
- docker_up 8y agoIs this really that hard to imagine? I am willing to bet that there are teams of spies who have infiltrated Google, Facebook, Amazon, etc. Spies from US, Russia, China, Britain, Israel, Germany, etc, must have dozens of spies working as engineers are getting access to all that data as we speak. To think that they aren't would be rather naive, in my opinion. If I were head of the spy agencies in any one of those countries, it would definitely be the first thing I would do.
- coolspot 8y agoEspecially if you: 1) happen to be country that makes all electronics 2) Company with a lot of ties to your country is happen to be top server supplier 3) Companies that use these servers happen to work for DoD, CIA and all major points of interest.
- aswanson 8y agoI wonder if China is making all these cheap wifi chips (esp8266, esp32) etc as backdoors into US infrastructure.
- baq 8y agoNo more Chinese phones for me either way. It's enough that Google knows everything about me, PLA doesn't have to.
- conceptme 8y agoWhat phones do not have Chinese components?
- e12e 8y agoNot only that, which base stations don't have Chinese components? But cpu, baseband, ram etc is certainly more serious consideration... And I don't see how you can get around that. Unless you plan to build a dumb phone around a Motorola 68k or something?
- jt3 8y agoWould be bad if a DoS attack could occur by killing the chip at a certain date or on command. Imagine electronic devices going down, and only way to fix is to get new hardware...ouch. Maybe this will open up more manufacturing jobs in the US.
- ivoras 8y agoWhile it's practically certain that hacking attempts do happen from both state- and non-state actors, this particular instance is so "alleged" that it's practically theoretical. Where's the actual hardware? Why didn't someone decap the tiny chip and probe it? Its design should be well within today's reverse engineering labs' capabilities.
- jonathankoren 8y agoWhat makes you think that people aren't going through they're POs and server farms and looking today? Just because a couple huge players told prior to publication, doesn't mean everyone was. These things literally take time, and there's no indication of how widespread the targeting was.
- dphack 8y agoIn the security world, there is always a new attack vector. With the advent of Spectre and Meltdown, I am expecting to see more hardware-based attacks in the future.
- abledon 8y agoand subsidies to enrol more students in EE to build up local national talent?
- cmurf 8y agoThis article is worth skimming/reading in order to fill in the technical gaps in the Bloomberg article; but also describes the complexity of modern computers in a way I didn't previously appreciate. What's still missing in the overall story, is what Apple and Amazon denials mean. The Bloomberg article says Apple and Amazon independently discovered these chips (Amazon via a 3rd party) in 2015; but the blanket denials by both companies appear to deny this discovery and reporting to U.S. authorities. I'd argue it's unethical for a company to apply falsus in uno, falsus in omnibus to a PR statement, but it's plausible there is a sufficiently misleading or false claim in the Bloomberg article that they feel it's legitimate to dismiss the entire article. In the meantime, the company denials have to be treated as conjecture.
- dboreham 8y agoSo the point of the attack was to subvert the BMC firmware in a way that was resistant to firmware re-flashing, but not resistant to firmware integrity checks. But there are no firmware integrity checks?
- baybal2 8y agoWhat is likely: they connected their own flash in place of recovery flash, that was not soldered on on that particular board version, that for what that soic8 or tsop8 pad is on photos; the chip will boot from the recovery, but if someone were to read the main flash, it would output the correct content. Only if somebody specifically poked the recovery, would the bug be revealed.
- parliament32 8y agoWhy would an integrity check show you anything different? It's all the same code running on the actual BMC, it'd be identical. That's the entire point of having an external package do all the malicious stuff -- reflash/dump/check will come up clean every time.
- uep 8y agoI think the attacks are real. A year ago, Google announced their Titan firmware security chip[1], which would limit these kinds of attacks. I don't believe they designed and built this chip, and surrounding infrastructure, because of purely theoretical attacks. Besides that, over the last couple years there has also been a lot of work trying to neuter the Intel ME, because of how dangerous it is. Another example is that Google had also done work to replace the EFI/Intel ME with a Linux kernel [2]. This has limited usefulness against the most recent attack, but it is related (since it's still a chip that has more privileges than the primary CPU). I suspect that there are a very small number of people in these big companies who are aware of these attacks. It's hard for me to guess why the companies involved would deny that these exist. 1. https://www.zdnet.com/article/google-opens-up-on-titan-security-heres-how-chip-combats-hardware-backdoors/ https://www.zdnet.com/article/google-opens-up-on-titan-secur... 2. https://schd.ws/hosted_files/osseu17/84/Replace%20UEFI%20with%20Linux.pdf https://schd.ws/hosted_files/osseu17/84/Replace%20UEFI%20wit...
- wjnc 8y agoFirst guess: not being allowed to admit it due to national security reasons and it being an ongoing investigation. On the same day several Russians were exposed trying to attack OPCW. They were exposed by Dutch military intelligence. At the press briefing the UK ambassador was there. Same day US indicts several Russian spies. This to show that these are major, international events and that proper disclosure towards investors is probably on the back burner as long as operations are continuing. It must be maddening for those involved not to be allowed to discuss it, or even having to actively lie about it while government officials leak freely in the game of thrones.
- lern_too_spel 8y agoIf they are under a gag order, they would simply not comment on it. Lying about it is never required and puts them at risk for shareholder lawsuits.
- lozaning 8y agoI wonder if there isn't some level of national security super mega secret scenario situation where the government requires companies to do something and the government indemnifies them against all risks associated with the required action.
- romed 8y agoI’m just going to throw this out there: BMCs and ILOMs are for tiny shops where “the IT guy” might have to do something from the beach at Cannes on their vacation. If you are a large operator like Apple you absolutely do not need BMCs.
- acdha 8y agoNot just tiny – any place where you don't want work to be rate-limited by the time it takes to get a body in front of a server. The only scale where it's not a big win is when you're so large that you have rock-solid auto-recovery in your software stack and hardware failures are handled by disabling a server and leaving it in the rack until it's periodically reaped at a regular interval. Most businesses are in that gap – definitely not tiny, but definitely not able to just ignore remote management – especially if they have lots of physical locations and it's not practical to have 24x7 IT staffing at every branch office.
- romed 8y agoRemote office management can be handled by a KVM with remote access. These can be security hazards in their own right, but it’s a lot safer than having some vendor junk literally hard-wired to your PCI bus.
- acdha 8y agoNow you have a second device to buy, secure and support, and it can't do everything that an ILOM can (most importantly, remote power management). Having had security updates for KVMs, I'm also not sure your assumption that it's safer is true in any meaningful sense. Management engine vulnerabilities have gotten a lot of hype over the last year or two but most of it has been marketing for security companies rather than a serious risk (e.g. what percentage of bugs required local network access, which should not be easy to get on your management VLAN). During a similar period, we've also had Spectre/Meltdown and the usual slew of software vulnerabilities so I have trouble with the conclusion that the answer is to stop using a useful tool rather than continuing the industry-wide effort to improve the level of security competency among development teams.
- ChuckMcM 8y agoOk, that finally makes a bit of sense about "if" this is true, how it might be carried out. And I agree with the author that the simplest action for a chip on the SPI bus would be to hold the MISO line low during power on to suggest to the BMC chip that its QSPI flash isn't programmed (note that QSPI starts up as 'regular' SPI and then switches over[1]). I would guess that the next thing the BMC would do is assume its on the factory floor and hasn't had firmware loaded yet and so it would attempt to boot via TFTP from some server to start the firmware loading process. FWIW at NetApp the firmware engineers called the BMC the 'BiteMeChip' because they always caused issues when bringing up a new filer motherboard. They were finicky, were often hard to update, and when misbehaving could completely screw up the system. So I can see this as a vulnerability but really I'd want to stick some probes on that 'mystery' chip and make sure it wasn't just some LC filter or something which is shunting off noise on the data lines. [1] https://github.com/ChuckM/stm32f469i/tree/master/demos/qspi https://github.com/ChuckM/stm32f469i/tree/master/demos/qspi
- SlowRobotAhead 8y agoAgreed on finally makes sense. Depending on what article it made it sound like a chip that sat between CPU and RAM or some external cache that was intercepting some pattern of data and real time buffering then modifying the output - which at CPU/RAM speeds made no sense. A chip holding and SPI line down - makes way more sense.
- ispiansclsda 8y ago> the simplest action for a chip on the SPI bus would be to hold the MISO line low during power on MISO is master-in, slave-out for anyone not familiar with serial peripheral interface jargon. Usually, the slave quad-SPI memory would send the configuration over this line, so pulling it low should dump the real data to ground.
- fkd221 8y agoBruce Schneier talked in 2013 already on BMC security linking to a paper [0] from Dan Farmer who did an Internet scan to find that ~230,000 BMCs were exposed to the Internet of which 90% could be compromised [1], yikes. [0] https://www.schneier.com/blog/archives/2013/01/the_eavesdroppi.html [1] http://fish2.com/ipmi/river.pdf
- zaroth 8y agoThe fact that cursory examination finds the attack is not only entirely feasible, and completely undefended against, but that the hardware shown in the Bloomberg animation is precisely the hardware which would be required to pull off the attack is quite astonishing. Whose “law” is it that the closer you are to an event the more you can see that the reporting on the event is desperately flawed? This has almost always been my own personal experience. To have the article hit the nail on the head with a entirely plausible attack means one of two things; 1) This is exactly what happened 2) A nation state wanted us to think this is exactly what happened. Either way the NSA is involved. The people on the ground who knew it happened would have to be NSL’d from telling their superiors that it happened, so they leaked it. Perhaps the denials from above are entirely sincere because the people who discovered it can’t pass it up the chain. The only thing I’m having trouble with is that would have required either a low-level plant to have discovered it and someone else at a low-level found out and was sworn to secrecy, or a middle-manager plant intercepted the message as it went up the line. The alternative, that this is a false flag, is pretty fascinating in its own right. TAO would have had to conceptualize the attack vector, and then someone planted the story. Perhaps this is equally likely. In fact, this is actually an attack I assume TAO is using in the field, which would burn this attack vector presumably because they have an even better one. Even more interesting, the idea TAO was actively using this attack vector, saw evidence in the field their attack was discovered (devices going dark) and so preemptively planted a story to blame the opposition before they accused the US of the same thing.
- gerwitz 8y agoI'm voting for false flag because it's the most interesting hypothesis to imagine. Especially with the NCSC statement.
- kromem 8y agoOr, it could be both. It could have been a real attack vector that didn't actually infiltrate the companies in question due to having been caught upstream. But as "attempted attack" sounds far less sexy than "actual attack," the sources may have exaggerated the impact/discovery to further trade negotiation objectives with the spin. In fact, the BS discovery part of the story might be an attempt to parallel construct a reason it was discovered to cover for a program that might do independent testing of hardware which, if revealed, would give attackers some sense of sampling methodology to be able to counter. That would explain the vehement denials from the companies allegedly compromised and yet the realistic attack vector published along with the US entity bans on using Chinese hardware around that time.
- mirekrusin 8y agoIf it is false, can/will/should Apple/Amazon sue Bloomberg?
- mindslight 8y ago> Let’s assume an implant was added to the motherboard at manufacture time. This needed modification of both the board design, and the robotic component installation process. It intercepts the SPI lines between the flash and the BMC controller Circuit wise, you don't really need to "intercept" (place in series with) the SPI lines. Two parallel drivers [0] will generally fight it out quietly, and you can guarantee yours will win by designing your implant to have the beefier driver (and perhaps only changing 1's to 0's, as P-channel FETs are generally weaker). If there already was a footprint for an extra chip (debugging, part flexibility, etc), then it would not require modification of the board design. And an extra component can easily be hand soldered afterwards outside of the standard robotic component placing. [0] The standard topology of SPI, but in proper operation only one is active due to CS lines.
- theomarkettos 8y agoPost author here. That was what I was thinking of - simply overdriving the lines to drag them high or low in opposition to the 'official' driver. That's pretty much all you can do with 6 pins. I was trying not to overcomplicate the explanation, since the material is already complicated enough! Fair point about hand soldering, however it would be more obvious to the manufacturing employees for an entire production batch to be diverted to have parts added manually than to simply sneak in another reel to the hundred on the pick and place machine.
- mindslight 8y agoI just thought that was phrased as if it were a necessary requirement for the attack, rather than a description of a likely approach. I could also see a scenario where only the occasional unit was bugged, and then an agent at the distributor made sure the right customers received the bugged units. Thank you for writing your article! I'm currently a bit out of the security headspace and reading the Bloomberg article had me scratching my head like what are the actual details here.
- fhood 8y agoCan someone outline some reasons for me why nobody has come up with an actual physical example of a compromised board? I'm not trying to make a point, I just want to get a more complete picture of the issue, and the biggest thing that stands out to me is the lack of physical evidence.
- netman21 8y agoJust wait. If they are out there they will be found. Of course you could ask why none of the ANT Catalog software implants have been found yet or why Juniper/Cisco/Fortinet have not (publicly) announced tools to detect them.
- lolc 8y agoIt's not like these boards are found in every living room. I would assume that if somebody found a suspicious board at their org, they would kick off internal processes first.
- rando444 8y agoWell according to the original article, the attack was targeted, implying the only way to get such a thing would be from one of the compromised customers, who are probably involved in an investigation into the matter, don't have all of the info themselves, and aren't super eager to release details to the public before a full picture emerges and mitigation procedures are in place. Further, going back to the original article, the majority of the information comes from alleged government sources, so not the people directly impacted, but rather those just helping deal with the fallout and coordination. Assuming there is merit to the story, it will likely be some time before more details emerge, unless having the story out there now helps accelerate that process.
- est 8y ago> the attack was targeted, Supermicro assemble their products in US/Taiwan. So in theory China need to accurately predict the pattern of how non-China factories install batches of the motherboard. I think it's extremely difficult to pull this off.
- devy 8y agoRegardless this attack is real or not, this exposure and assessments (just like Charlie Miller and Chris Valasek did for exposing the car hacks) raised good awareness of how vulnerable BMCs are and hopefully the baseboard hardware security and firmware security will be securer in the future when they evolve.
- tomc1985 8y agoI really hope that this is the straw that breaks the back of all these "management engines" Like seriously, why does my hobby consumer motherboard need that feature? Corp IT only ever deploys to large fleets of OEM machines.
- bri3d 8y agoThe management engine in question is an ASpeed standalone part which is only used in large-scale server deployments, not the CPU MEs that you're forced into.
- contingencies 8y agoTake a look at ASpeed (BMC supplier) stock movement: https://finance.yahoo.com/quote/5274.TWO/chart?p=5274.TWO https://finance.yahoo.com/quote/5274.TWO/chart?p=5274.TWO
- duskwuff 8y agoThat movement seems to have started about two weeks ago. I don't think it's related to this news.
- contingencies 8y agoI'm sure it didn't help!
- tomc1985 8y agoI know, but seeing the article mention "management engines" made me hope that it will direct a bunch of newfound scrutiny on Intel ME/AMD TrustZone/etc
- beiller 8y agoDoes this mean I can get some great deals on Supermicro boards now? :)
- calimac 8y agoHey OP, call it what the article was and nix the "Alleged" it spills your bias.
- TickleSteve 8y agoQSPI has 4 data lines, a clock and a chip-select line. In order to intercept a QSPI bus, you would therefore need 4+1+1+2xpower lines = 8 pins. This is not sufficient for QSPI. Single SPI on the other hand requires Clock, Data, ChipSelect, Ground & Power = 5 lines, so that would be plausible. (Yes, I know you can potentially get rid of the CS line, but that depends on what else is on the bus).
- mindslight 8y agoYou need two lines - data/ground, or data/Vdd. Probably whichever gives you voltage when idle. You harvest power when data is idle. CS is irrelevant. The rough clock speed is fixed, and you can match the precise timing from the data line. QSPI actually gets you access to data in both directions with the tradeoff of only getting one quarter of the bits. Logically, you likely only need to recognize a few patterns that are each say 30 bits long. If you could shrink this down, it would look exactly like a boring pullup.
- TickleSteve 8y agoCS isn't irrelevant if there are multiple devices on the bus, we havent seen a schematic.
- mindslight 8y agoYes it is - a tap doesn't need to be 100% sure that a given chip is selected like how we think when designing a circuit - it just needs to key off patterns that look like memory reads of interest. If you've got ambiguity you don't need to pull in more parallel bits, just record a longer serial history. In fact I would think the most stealthy and robust way of bugging code would be to just ignore addresses and look for a context-free stream of instructions that does some security-sensitive initialization (either of the platform or of the application code). Turn off some flag that's supposed to be on, causing the code to continue running as normal but silently skipping certain checks. Now that you mention other devices on the bus - those in fact would be a great chip to subvert with such a backdoor. Why bother fitting everything into a two terminal hack package when you could just have a malicious temperature sensor sourced from the "lowest bidder" that does the attack and jumps to its own ROM? (I think those are usually I2C but you get the point). Bonus points for only being triggered through some sidechannel - recruit a low level employee at the target, while they retain plausible deniability.
- deleted 8y ago[deleted]
- dsdsdsds2 8y agoThe Register's article on the matter seems also good: https://www.theregister.co.uk/2018/10/04/supermicro_bloomberg/ https://www.theregister.co.uk/2018/10/04/supermicro_bloomber...
- hugelgupf 8y agoI think the board that is pictured by Bloomberg is just an illustration. For example, Elemental never used that board. See my tweets here for the actual specs: https://twitter.com/hugelgupf/status/1048160794565861377?s=19 https://twitter.com/hugelgupf/status/1048160794565861377?s=1... Elemental boards come with GPUs. The blade pictured has neither GPUs nor PCIe.
- alexnewman 8y agoThese types of attacks are distractions from the real attacks. Doping attacks are the big attack vector now, and they can't be picked up with electron microscopes
- bayindirh 8y agoThere's a problem with exfiltrate via BMC network theory. In a sane setup, your BMC connection cannot access internet. You should build an isolated intranet for it (including VLAN or hardware isolation, not just subnet/IP), and put a VPN in the front gate. As a result, you login to your data center, or go to there if you like metaphors. If nobody’s there via VPN, BMC network is a silent and dark place. No connection to outside, no unknown traffic, just silence. Only exception may be the discovery packets of some BMCs, which can find similar servers and form federations for easier management. Even this needs some setup beforehand.
- EvanAnderson 8y agoIf the BMC has write access to host memory it could surely use that access to create a side channel using the host's network interfaces. Having said that, it would be nice if networks were segmented in the way you describe. I've been appalled at the lack of segmentation I've seen in companies of all sizes that I've had gigs for.
- bayindirh 8y agoToday's network cards are small computers of their own. So it'd be very hard to inject packages with all this kernel-hardware integration at the module level IMHO. The card would probably throw a tantrum if you try to access it directly. TBH, while I'm knowledgeable about hardware, I'm a total beginner in attack side of cybersecurity. At least, we are segmenting our networks like that.
- arminiusreturns 8y agoBlame management.
- deleted 8y ago[deleted]
- andrewflnr 8y agoAccording to this comment, the BMC is at least capable of working off the real NIC instead, and will do so if you don't hook up the management NIC. https://news.ycombinator.com/item?id=18138411 https://news.ycombinator.com/item?id=18138411
- DenisM 8y agoA decade ago when I worked at Microsoft I shopped around the idea of using XBox as a basis for secure computing. XBox was designed to function in the hands of the adversary, to be robust against peripheral attacks and even motherboard mods. Even the main memory was encrypted by the on-CPU controller. Obviously, no open JTAGs. A lot of expertise there. In my fantasies it would form the basis of the DoD infrastructure and then trickle down to finance. It was deemed to be not solving any practical problems, so it didn't go anywhere. Oh well, I found other fun thins to do in my life.
- joshstrange 8y agoAre you referring to the original Xbox? Modchips hit the scene for that in ~ 2 years after release. It's not exactly what I would consider a basis for secure computing. Or maybe you are just referencing the procedures "main memory was encrypted by the on-CPU controller. Obviously, no open JTAGs" should be ratified to create a basis for secure computing, a checklist of things to do/prevent before considering a device "Secure".
- DenisM 8y agoXBox 360 had the on-CPU memory controller with encryption, and it was very tamper-resistant overall. A good model to follow.
- 394549 8y agoIIRC, the original XBox made the assumption that the main bus was secure due to its high-speed, which turned out to be false. That was one of the few things that was unencrypted, and it offered a way in.
- dawnerd 8y agoDidn't even need a modchip. There was a technique where you could softmod by simply unplugging the ide harddrive right after boot and hot plugging it into a desktop. https://www.reddit.com/r/originalxbox/wiki/hotswap https://www.reddit.com/r/originalxbox/wiki/hotswap
- 8y ago
- agumonkey 8y agoFew years ago https://hackaday.com/2013/08/02/sprite_tm-ohm2013-talk-hacking-hard-drive-controller-chips/ https://hackaday.com/2013/08/02/sprite_tm-ohm2013-talk-hacki... popped up, basically hdd have many arm co-controllers. Nowadays everything is a processor, everything is thus virtual and fakeable. Enjoy the paranoia :)
- PaulHoule 8y agoThe NSA has been caught doing this kind of stuff to Cisco routers so I can believe it.
- exabrial 8y agoRegardless if someone of the details turn out to be true, there's a couple of really important network security takeaways here. First you need to run an air gapped network or at least switch enforced vnet for your bmcs. Three firmware on these is only updated every few years anyway, so they're likely full of security holes anyway. Second, in general, outbound connections need to be monitored everywhere in your data center.
- osterwood 8y agoIf this attack vector is true, it seems overly complicated to me. Why not have the assembly house load the normal SPI flash with firmware you've altered to control the main CPU the way you want? That is WAY easier than hardware changes to the PCB design and installing a part which isn't on the approved Bill of Materials.
- blitmap 8y agoProtecting tech in transit would be a new business. Tamper-evident packaging and such.
- GaryNumanVevo 8y agoobviously we need block-chain now, I'll start up the ICO generator script
- cimi_ 8y ago> But there’s another trick a bad BMC can do — it can simply read and write main memory once the machine is booted. Doesn't ASLR[0] mitigate this? [0] https://en.wikipedia.org/wiki/Address_space_layout_randomization https://en.wikipedia.org/wiki/Address_space_layout_randomiza...
- ATsch 8y agoI don't see how it would. ASLR only really helps making it harder for attackers to gain full control when they manage to execute some instructions in your process via memory corruption. It relies on the memory layout being hard to guess, however, the BMC can already just read from arbitrary memory, so it can just look it up. What would help here is isolating PCIe devices with the IOMMU, but this is currently rarely enabled, only for virtualization, apparently due to it's relatively high overhead.
- stordoff 8y agoAs I understand it, no - if you can read from all of main memory, you can just look for the function you need.
- monocasa 8y agoASLR is a virtual memory technique, an attack via the BMC would attack physical memory.
- aussieguy1234 8y agoSmartphones, both iPhone and Android are manufactured in China
- shawwn 8y agoTest comment, please ignore.
- alexandercrohde 8y agoIn another thread Walterbell points out: From 2016, https://arstechnica.com/information-technology/2016/03/repor.. https://arstechnica.com/information-technology/2016/03/repor.... > Apple has begun designing its own servers partly because of suspicions that hardware is being intercepted before it gets delivered to Apple, according to a report yesterday from The Information. "Apple has long suspected that servers it ordered from the traditional supply chain were intercepted during shipping, with additional chips and firmware added to them by unknown third parties in order to make them vulnerable to infiltration, according to a person familiar with the matter," the report said. "At one point, Apple even assigned people to take photographs of motherboards and annotate the function of each chip, explaining why it was supposed to be there. Building its own servers with motherboards it designed would be the most surefire way for Apple to prevent unauthorized snooping via extra chips."
- branchless 8y agoUltimately the west deserves to lose, should this come to conflict. Our system, which elevates the corporation above all else, is so damaging.
- Tempest1981 8y agoThe industry was aware of intercepted/modified hardware shipments long before this, also giving reason to be suspicious: From 2013: “NSA reportedly intercepting laptops purchased online to install spy malware” https://www.theverge.com/2013/12/29/5253226/nsa-cia-fbi-laptop-usb-plant-spy https://www.theverge.com/2013/12/29/5253226/nsa-cia-fbi-lapt... Also https://www.wired.com/2015/02/nsa-firmware-hacking/ https://www.wired.com/2015/02/nsa-firmware-hacking/
- ezVoodoo 8y agoTechnical possibility is one thing; proving the story has actually happened is another thing. Until now, what we get is a categorical denial of the story from all related parties. And all the evidence Bloomberg can provide so far is just vague anonymous sources. Talk is cheap, show me the code/server/chip if they ever exist. Otherwise, the story is just a blunt lie fabricated by Bloomberg serving as a propaganda to bash China amid the Sino-America trade war.
- KaiserPro 8y agoBMCs have always been an Achilles heel. HP's iLo could be got into if you used `curl -H "Connection: AAAAAAAAAAAAAAAAAAAAAAAAAAAAA"` Supermicro's horrid BMCs, of which there are many, all were horrendously lax in security, long before this chip was, or was not inserted. You didn't buy supermicro if you were worried about security, you bought them if you needed to stack stuff high, cheap and dense. There is a reason why its best practice to put them on separate networks, with as much stuff between it and anyone else. BMCs are massive backdoors, and should be treated with caution.
- novaRom 8y agoThis chip looks for me like a simple fuse. This is not for spying, but rather a doomsday switch. If activated, it will short the circuit destroying full system operations. I don't even think it contains any sophisticated digital logic inside.
- VCinvestorSF 8y agoThis story is ridiculous. I believe the arguments made int eh article didn’t happen. Apple, amazon, and SMCI have all rebuffed. The Bloomberg journalist was duped. SMCI did nothing wrong and their business remains strong as we speaker. They will shortly: 1) Issue a detailed rebuttal with QC procedures and 2) become current with their audited financials by the end of October. As a result, the stock will revisit $30 per share within 3-6 months which is a triple from today’s price. Who remembers the Johnson and Johnson Tylenol scar? Someone posiomed Tylenol and the stock cratered. This is no different, except the “poisoning” never happened. And even if it did, it was 3 years ago and was discovered.
- bobbob1921 8y agoevery supermicro motherboard that I have dealt with (several 100s) always has a jumper which can enable(shorted)/disable the bmc/ipmi. i’ve been surprised at how many we have bought used off eBay/surplus that had the jumper open (ie bmc disabled). wouldn’t having this disabled physically (jumper open), thwart this attack? (i’m asking not suggesting) and I do realize the majority of people do have the BMC enabled, including me) also the fact that supermicro still offers this jumper to disable the BMC might speak to the fact that they (sm) are not in nor had any knowledge of this issue (ie they were not “In on it”)
- wahern 8y agoI don't understand why everybody assumes it must be pulling data over the network autonomously. It could simply compromise the host OS to augment other, targeted attacks. Say, by reintroducing a buffer overflow or race condition. It would be incredibly short-sighted to go to such lengths to compromise these machines just to naively pull from a command+control server, virtually advertising its presence. Defenders simply do not think like attackers. If you're a defender it's lethal to try to think like an attacker. "I can't imagine how this would be useful therefore it must not be useful" are the famous last words of everybody who has cast doubt on a new and novel attack vector. As a defender you need to first estimate the unknowns and unknown unknowns, which over the last year alone have exploded (e.g. actual and potential Intel microcode vulnerabilities). If you needed this article to convince you of feasibility or even just practicality, please don't pretend to be capable of assessing the security posture of any complex system. And let's be clear: this is not a new and novel attack vector. There are companies that have existed for quite some time researching and selling products to deal with this sort of attack vector. On the spectrum of hardware based attacks feasible today, this chip isn't at the complex end of the spectrum but rather the simple end of the spectrum. The complex end of the spectrum involves hiding logic deep within existing ICs, and there's ample literature to demonstrate feasibility of both implementation and detection. The difficulty in pulling off these attacks lies not in the software or hardware, but the political, intelligence, and military apparatus of attacking countries. The economic costs of detection are huge precisely because, at the end of the day, fundamental security relies on trust, not technological hurdles per se.