6 ms·
It was reported that the security auditor used during Elemental's acquisition detected this compromise. I assume they found it in a randomly selected board. Eit
by lovelearning 8y ago
It was reported that the security auditor used during Elemental's acquisition detected this compromise. I assume they found it in a randomly selected board. Either they were very lucky, or hundreds of boards were compromised.
Now, I think all motherboard manufacturers - and especially high end server manufacturers like SM - use sophisticated automated tests and quality control on boards. Under what circumstances is it possible that SM's QC missed this out on so many boards? Won't it affect things like the power budget, weight, and latency? What do professional EE people here think?
- auslander 8y ago> what circumstances is it possible that SM's QC missed this out Rogue insider, paid or patriotic, likely both. > affect things like the power budget, weight, and latency All three would have no measurable change, falling into measurement error margin. Its a death sentence for Supermicro, nothing will help to restore trust. They should publish a detailed post-mortem analysis, though.
- toomuchtodo 8y ago> They should publish a detailed post-mortem analysis, though. More of an obituary at this point.
- gscott 8y agoThey took pains to say that this was happening with sub-contractors in China building motherboards in excess of what Taiwan could handle. So to build trust they can pull back on that extra manufacturing capacity and move it elsewhere maybe South Korea.
- auslander 8y ago> So to build trust they can You can't put the toothpaste back in the tube
- lovelearning 8y agoYeah, SM has the most to lose here. Regardless of whether the compromise is true or non-existent, they really should consider becoming more transparent about their QC, I think.
- LeonM 8y agoEE here, looking at the size of the chip, the weight and power difference is going to be too small to measure. Latency will probably not be affected when the chip is dormant
- scld 8y agoIf it replaced a DNP part, it could have been found at a flying probe test. Half the expected impedance on a few of those pins? Also AOI might have picked up a difference.
- dboreham 8y agoFlying probe testing is unlikely to be used for this type of product.
- pixl97 8y agoSounds like a randomized sample of boards needs tested against a reference then.
- nvusuvu 8y agoAOI would miss the chip if it were buried between layers or look exactly like a part it was replaced for.
- lovelearning 8y agoI wonder how the auditing team detected it. X-rays?
- bri3d 8y agoThey might have noted an anomaly in the SMC given the acknowledged SMC firmware exploits (maybe a beta test run?) and investigated the SPI lines to the SMC as a response.
- mediocrejoker 8y ago
- luma 8y agoDepends on where it was added. Doing it as part of the assembly process would require having access to the pick and place software along with being able to load reels of the implant into the machine. A well placed insider may be able to make that happen, and automated optical inspections would be fed by the modified pick and place program, so checks would pass as the new component is part of the build.
- lovelearning 8y agoThanks for that info. I assumed that once the boards came back to SM's US premises, a second level of optical and other tests are done by SM employees. But perhaps that's not - or was not - their actual workflow.
- admax88q 8y agoWould be pretty wasteful to wait until the hardware was shipped to the US to look for manufacturing defects.
- JohnJamesRambo 8y agoThat’s my main takeaway from all of this- Elemental is great at their job and didn’t just do a half assed attempt at an audit. They actually did what they were hired to do. I wonder how rare that is.
- coolspot 8y agoYou perhaps meant not Elemental, but Elemental's auditors is great at their job. > ... In late spring of 2015, Elemental’s staff boxed up several servers and sent them to Ontario, Canada, for the third-party security company to test, the person says. Nested on the servers’ motherboards, the testers found a tiny microchip ...
- deleted 8y ago[deleted]
- JohnJamesRambo 8y agoOh yes, my mistake yes, I thought Elemental was the name of the third party security company. Do we know the name of that company? They need some major kudos for this.
- neffy 8y agoThere used to be a Nortel Division up there that had an electro scanning microscope for this sort of thing - I imagine they´re still there under a different name.
- crispyambulance 8y ago> ... high end server manufacturers like SM - use sophisticated automated tests and quality control on boards. MFG test guy here (not supermicro!) Not necessarily. Automated production tests are there to configure and exercise the system and confirm it works as specified. Such tests are good at things like finding bad solder joints, pick-and-place mishaps, misconfiguration of firmware and weeding out product that fails functional test. That, by itself, is hard enough. Such tests are not _really_ able to detect things which no one is expecting, or worse, things which an adversary has specifically designed to avoid detection. Sure, if something gets "discovered" during a failure analysis, a test or process can be created to specifically address THAT problem in the future. To find "unknown unknowns", you need an audit of some kind and that is definitely different from production test.
- lovelearning 8y agoThank you for the insight. I was rather naively imagining that every board is subject to some kind of x-ray image matching with the original PCB design to find differences. Not that simple, I see.
- crispyambulance 8y agoSome manufacturers do X-ray boards. Typically this is done to check that BGA devices (https://en.wikipedia.org/wiki/Ball_grid_array https://en.wikipedia.org/wiki/Ball_grid_array) are soldered properly. Usually not done on every board, but only if there's a problem suspected. When it happens they tend to focus only on particular BGA components or suspect copper traces rather than the whole PCB.
- extrapickles 8y agoUsually you scan boards at a percentage determined by statistics (SPC can get complex). However the checks are automated, so you would have to perform the checks in a different trust zone (eg: different country).
- scoggs 8y ago
- ig1 8y agoThe Bloomberg article says suspicions were raised during a routine audit that resulted in a deeper audit. The routine audit could have uncovered hackers in Elemental's corporate network doing things like ensure certain machines go to certain customers, etc.