4 ms·

> I mean, people were tearing these boards apart, looking for anything. There's no way they could keep that quiet, even if they were willing to disappear people

> I mean, people were tearing these boards apart, looking for anything. There's no way they could keep that quiet, even if they were willing to disappear people. Without commenting on whether or not I believe the Bloomberg story, I very much doubt that "people were tearing these boards apart", doing reverse hardware-level engineering looking for an unknown. Even in the case of open-source software, with full source code and documentation available, the number of people who analyze code is miniscule. Doing the same thing with ICs and PCBs, without engineering diagrams and documentation, and needing very specialized skills and tools, must be 1000x more difficult. Maybe Supermicro and some intelligence agencies did internal studies, but they aren't going to tell you anything. Perhaps Apple and Amazon asked around in their engineering departments to see if anyone saw something. Possibly a couple security researchers took a look to see if they saw something obvious. But I don't think teams of experts were mobilized all around the world trying to pinpoint a deeply hidden hardware vulnerability. I'd love to be proven wrong. Hacker News is read by ~50,000 engineers and programmers and related professionals a day. Admittedly my comment won't reach 50,000 people, but I'll ask anyway: Is there anyone reading here who personally analyzed a Supermicro board in any meaningful way, or knows anyone who personally did so?

derefr 7y ago
I mean, it's not that expensive to just take some hi-res X-ray photos of old and new boards of the same make, and compare the trace paths near each component. Maybe you can even take a visual diff of the X-ray negatives, if you can get everything lined up tightly enough. You can parallelize this search, because the people doing it don't need to know much about what they're looking for—they're just hunting for visual differences. Unless/until one of them actually finds anything different. At that point, you need better resources to figure out what's different; but at that point, all hell also breaks loose, and you'll get all the resources and help you need to do better analyses.
- cantrevealname 7y ago
  > Unless/until one of them actually finds anything different ... at that point, all hell also breaks loose Boards bought weeks or months apart might have lots of differences in traces and components due to engineering modifications and normal changes in suppliers. Do you really think that all hell would break loose if you showed that there were differences between a Supermicro board made in March 2018 and one made in September 2018? Other than maybe Supermicro or intelligence agencies (who aren't talking), I doubt that anyone has done what you've suggested. People are busy and it's a lot of work based on an allegation and no exact place to look.
  - errantspark 7y ago
    Is that really true? Granted I'm working on low volume stuff (1000's of units) in a niche B2B setting, but the times for rolling out a new rev of a board can sometimes be measured in quarters. Changing traces/components can have unforeseen effects for even fairly minor things, you can't just have a Tier1/2 CM tool up without a round of DVT. I doubt boards with the same PN change very often if at all, but I'd be curious to know if i'm wrong.
  - nitrogen 7y ago
    I used to work for an electronics manufacturer. They did use xrays on a regular basis, but mostly to find cracked solder joints under ICs.
- cyphar 7y ago
  > I mean, it's not that expensive to just take some hi-res X-ray photos of old and new boards of the same make, and compare the trace paths near each component. Maybe you can even take a visual diff of the X-ray negatives, if you can get everything lined up tightly enough. There are examples of attacks which cannot be detected by staring at X-rays of components[1]. Now, to be fair they are very advanced attacks but we know they exist. [1]: https://link.springer.com/article/10.1007/s13389-013-0068-0 https://link.springer.com/article/10.1007/s13389-013-0068-0
rossenberg79 7y ago
I know a cybersecurity professional who was contracted by a hedgefund to investigate the claims by reverse engineering and present a report so it’s not entirely unlikely.
lsc 7y ago
so by my reading of the article, it was extra chips... as I recall, it looked like some of them were hidden in the power supply (totally possible, if you can find some exploit on the I2C interface, but also probably fairly easy to detect) (the story was that it was a Chinese subcontractor. Remember that SuperMicro is a Taiwanese company; I'm pretty sure that SuperMicro has less Chinese made content than any of the other major manufacturers of commodity server kit. ) I did know people cutting open power supplies, cutting open capacitors (a place chips have been hidden in the past) and otherwise examining the systems at the macro level. I also knew people who managed networks of these things... and one of the bloomburg stories, as I recall, claimed that the hack was discovered via looking at the network as the BMC tried to phone home. I mean, yes, you could do something more clever than that, replace a chip with one that looks identical, which uses the same traces that did something different, but the article certainly implied it was a more simple sort of thing. I mean, if I was trying to compromise a BMC in that way, I'd just flash it with firmware that did what I needed (and this has happened before... to many different vendors) - my understanding of the bloomburg story was that there was some bit of hardware to re-flash the firmware to the compromised version after the user upgraded to a non-compromised version.
lsc 7y ago
>Without commenting on whether or not I believe the Bloomberg story, I very much doubt that "people were tearing these boards apart", doing reverse hardware-level engineering looking for an unknown. Even in the case of open-source software, with full source code and documentation available, the number of people who analyze code is miniscule. I can tell you people were literally tearing them apart; cutting open capacitors and the like (common place to hide extra components) pulling apart power supplies (as I recall, one of the bloomburg illustrations implied that something was in a power supply; I remember a long discussion of how you could compromise something over i2c from a chip actually in the psu) - I mean, sure, these things could be better hidden to the point where such techniques would be ineffective, but my reading of the bloomburg article was that they weren't particularly well-hidden, and that therefore they were discovered. Hiding that sort of thing from another EE who designs that sort of thing seems like it's probably pretty difficult. (I mean, it is common to have compromised firmware; that's way easier. but that's also not what the bloomburg article was about) But... yeah, I guess a lot of my assumption was based on my reading of the bloomburg article implying that this hardware was something tacked on in a detectible way; the sort of thing that could be done by some third-tier subcontractor. I acknowledge that if your attacker has the capability to produce ASICs that look just like the real thing and act just like the real thing out the same I/O pins, except in some rare case? yeah, that's not going to fall to this level of examination. but that wasn't my reading of the bloomburg story. Furthermore, if an attacker went through that level of effort, they'd probably not just do it for supermicro. Most motherboards of the same generation are pretty similar; pretty heavily based on reference designs. If you were going to do something that sophisticated, that's where you'd attack. If you were going to attack just one particular rev of one particular brand of a motherboard? well, something tacked on seems a lot more feasible.
lsc 7y ago
>Even in the case of open-source software, with full source code and documentation available, the number of people who analyze code is miniscule. I think... we might just live in different worlds? I'm a sysadmin and I maintain computers, but I am surrounded by serious programmers (I mean, kernel and system verilog types, not frontend people) I have worked places where it was just me and the frontend folks, and in that case? yeah, It's kinda lonely when you need help on anything serious. But even then, with my social life being what it is now, I'd have people to ask if I was in that situation again at work. I... usually don't have any trouble getting someone qualified to look at my backtraces when I have kernel panics, and I usually don't have trouble getting those people to give me patches. that requires analyzing the code. I mean, sure, people don't spend a lot of time reading random code for no reason... but you give them a clue that something interesting is somewhere, and my experience? you can get plenty of help.

derefr 7y ago

I mean, it's not that expensive to just take some hi-res X-ray photos of old and new boards of the same make, and compare the trace paths near each component. Maybe you can even take a visual diff of the X-ray negatives, if you can get everything lined up tightly enough. You can parallelize this search, because the people doing it don't need to know much about what they're looking for—they're just hunting for visual differences. Unless/until one of them actually finds anything different. At that point, you need better resources to figure out what's different; but at that point, all hell also breaks loose, and you'll get all the resources and help you need to do better analyses.

cantrevealname 7y ago
> Unless/until one of them actually finds anything different ... at that point, all hell also breaks loose Boards bought weeks or months apart might have lots of differences in traces and components due to engineering modifications and normal changes in suppliers. Do you really think that all hell would break loose if you showed that there were differences between a Supermicro board made in March 2018 and one made in September 2018? Other than maybe Supermicro or intelligence agencies (who aren't talking), I doubt that anyone has done what you've suggested. People are busy and it's a lot of work based on an allegation and no exact place to look.
- errantspark 7y ago
  Is that really true? Granted I'm working on low volume stuff (1000's of units) in a niche B2B setting, but the times for rolling out a new rev of a board can sometimes be measured in quarters. Changing traces/components can have unforeseen effects for even fairly minor things, you can't just have a Tier1/2 CM tool up without a round of DVT. I doubt boards with the same PN change very often if at all, but I'd be curious to know if i'm wrong.
- nitrogen 7y ago
  I used to work for an electronics manufacturer. They did use xrays on a regular basis, but mostly to find cracked solder joints under ICs.
cyphar 7y ago
> I mean, it's not that expensive to just take some hi-res X-ray photos of old and new boards of the same make, and compare the trace paths near each component. Maybe you can even take a visual diff of the X-ray negatives, if you can get everything lined up tightly enough. There are examples of attacks which cannot be detected by staring at X-rays of components[1]. Now, to be fair they are very advanced attacks but we know they exist. [1]: https://link.springer.com/article/10.1007/s13389-013-0068-0 https://link.springer.com/article/10.1007/s13389-013-0068-0

rossenberg79 7y ago

I know a cybersecurity professional who was contracted by a hedgefund to investigate the claims by reverse engineering and present a report so it’s not entirely unlikely.

lsc 7y ago

so by my reading of the article, it was extra chips... as I recall, it looked like some of them were hidden in the power supply (totally possible, if you can find some exploit on the I2C interface, but also probably fairly easy to detect) (the story was that it was a Chinese subcontractor. Remember that SuperMicro is a Taiwanese company; I'm pretty sure that SuperMicro has less Chinese made content than any of the other major manufacturers of commodity server kit. ) I did know people cutting open power supplies, cutting open capacitors (a place chips have been hidden in the past) and otherwise examining the systems at the macro level. I also knew people who managed networks of these things... and one of the bloomburg stories, as I recall, claimed that the hack was discovered via looking at the network as the BMC tried to phone home. I mean, yes, you could do something more clever than that, replace a chip with one that looks identical, which uses the same traces that did something different, but the article certainly implied it was a more simple sort of thing. I mean, if I was trying to compromise a BMC in that way, I'd just flash it with firmware that did what I needed (and this has happened before... to many different vendors) - my understanding of the bloomburg story was that there was some bit of hardware to re-flash the firmware to the compromised version after the user upgraded to a non-compromised version.

lsc 7y ago

>Without commenting on whether or not I believe the Bloomberg story, I very much doubt that "people were tearing these boards apart", doing reverse hardware-level engineering looking for an unknown. Even in the case of open-source software, with full source code and documentation available, the number of people who analyze code is miniscule. I can tell you people were literally tearing them apart; cutting open capacitors and the like (common place to hide extra components) pulling apart power supplies (as I recall, one of the bloomburg illustrations implied that something was in a power supply; I remember a long discussion of how you could compromise something over i2c from a chip actually in the psu) - I mean, sure, these things could be better hidden to the point where such techniques would be ineffective, but my reading of the bloomburg article was that they weren't particularly well-hidden, and that therefore they were discovered. Hiding that sort of thing from another EE who designs that sort of thing seems like it's probably pretty difficult. (I mean, it is common to have compromised firmware; that's way easier. but that's also not what the bloomburg article was about) But... yeah, I guess a lot of my assumption was based on my reading of the bloomburg article implying that this hardware was something tacked on in a detectible way; the sort of thing that could be done by some third-tier subcontractor. I acknowledge that if your attacker has the capability to produce ASICs that look just like the real thing and act just like the real thing out the same I/O pins, except in some rare case? yeah, that's not going to fall to this level of examination. but that wasn't my reading of the bloomburg story. Furthermore, if an attacker went through that level of effort, they'd probably not just do it for supermicro. Most motherboards of the same generation are pretty similar; pretty heavily based on reference designs. If you were going to do something that sophisticated, that's where you'd attack. If you were going to attack just one particular rev of one particular brand of a motherboard? well, something tacked on seems a lot more feasible.

lsc 7y ago

>Even in the case of open-source software, with full source code and documentation available, the number of people who analyze code is miniscule. I think... we might just live in different worlds? I'm a sysadmin and I maintain computers, but I am surrounded by serious programmers (I mean, kernel and system verilog types, not frontend people) I have worked places where it was just me and the frontend folks, and in that case? yeah, It's kinda lonely when you need help on anything serious. But even then, with my social life being what it is now, I'd have people to ask if I was in that situation again at work. I... usually don't have any trouble getting someone qualified to look at my backtraces when I have kernel panics, and I usually don't have trouble getting those people to give me patches. that requires analyzing the code. I mean, sure, people don't spend a lot of time reading random code for no reason... but you give them a clue that something interesting is somewhere, and my experience? you can get plenty of help.