3 ms·

> ...every hardware pentesting shop will be going after these boards like they're looking for golden tickets. The way the original story was written, it sugges

> ...every hardware pentesting shop will be going after these boards like they're looking for golden tickets. The way the original story was written, it suggested that four subcontractors were identified, and almost 30 targets selected, with the implied suggestion that either the boards were custom special order boards, or destined for a specific lot order made by a customer. If true, then it is unlikely these boards will be found in the wild except by slipping through those almost 30 targets into the used market. Supermicro has about 600 board SKUs, so finding these needles in the haystack is likely more feasible by approaching data centers offering to help find the boards for free, in return for allowing the pentesting firms to take physical possession of such found boards. The story did reveal that the original "tell" that gave away the chip was odd but not obviously malicious at first glance network traffic, and that the suspected intent was implementing an Advanced Persistent Threat model. The article also mentioned the chips were connected with the BMC, but it wasn't specified in the article whether or not the chips got out onto the Net. So finding the boards in the wild probably will focus upon finding them in the same manner, over the network. Power down the server, let the BMC stay powered on, and watch for unwarranted network activity. Or boot a Linux on a stick that deliberately does as little as possible and premises networking allows it to do just enough routing out to the Net to capture traces of what unauthorized network traffic is trying to do, and watch for unwarranted network activity, in case the chip design is clever, and hides its activity until it detects the mainboard is already running before trying to inject its network payloads onto the mainboard's network interfaces. What are others' thoughts on how to find these "golden ticket" boards?

dvfjsdhgfv 8y ago
That's the interesting bit. Having done so much to hide the exploit your most important aim is to hide it's presence. Anything that would just "connect to a Chinese server" would be discovered immediately. If the Bloomberg story is true, the network traffic scheme must have been extremely sophisticated to fool so many network security specialists at top companies for such a long time. This, or the story is false, purposely or not.
- ytjohn 8y ago
  Most places I know of isolate their OOB management network, requiring a vpn or jumpbox to access it. However, if someone did let their OOB network full outbound access, I could see this slipping through. I could imagine that simply going to a CDN or cloud provider like AWS/Cloudfront/cloudflare/akamai with a dns lookup along the lines of updates.supermicro.cdn-front.com wouldn't be too suspicious. At that point, you'd be looking for dns lookups and not firewall hits. If you are blocking outbound, I could still this going unnoticed if you're not actively reviewing denials. But, if you are properly watching dns lookups from OOB and it's anything other than necessary services (ntp, ldap, syslog), then this would get picked up pretty quickly.
  - fulafel 8y ago
    Sounds like many things would have to go right in the defender's court. Optimism is not a good defense strategy :)
    - ytjohn 8y ago
      Heh. One colleague has "Hope is not a valid deployment strategy" as a signature. Most places might be blocking this type of activity by default. For most of our security audits, it's just assumed that the SM IPMI or Dell idrac is vulnerable to one exploit or another. We mitigate that by controlling the traffic. I feel this is common practice in most places that understand vlans and firewalls. However, while blocking is easy, being aware of something like this is on another level altogether. Unicorn jumping over a rainbow level rare. You really have to be logging outbound attempts and dns lookups. Where I work, there is a full security team and they are at an insane level where they log the allowed traffic. One told me that the allowed traffic is more interesting than the denied traffic. Denied just tells them what we anticipated, while active helps them establish a pattern and look for deviations.
      - dvfjsdhgfv 8y ago
        That's my point. Security people were analyzing network traffic for decades trying to spot something that doesn't fit, host-wise, pattern-wise or even packet-wise (see The Museum of Broken Packets[0], for example). And someone managed to somehow hide all this traffic from security experts working for Amazon and Apple, for months or years? I'm very curious to see how. [0] http://lcamtuf.coredump.cx/mobp/ http://lcamtuf.coredump.cx/mobp/
hef19898 8y ago
That is a good question, did the Chinese target specific customers, identifying the relevant sub-contractors and modify these boards? Or are these boards commercial-of-the-shelf things that are mass produced? In the latter case these boards can just end up anywhere, in the first case not so much. As a result using commodity boards would provide some degree of protection against such an attack as slipping a manipulated one into a specific target's servers would be a game of chance. Also the risk that the attack is discovered would be much higher.
- jnbiche 8y ago
  > That is a good question, did the Chinese target specific customers, identifying the relevant sub-contractors and modify these boards? Why not go read the story? It addresses this question and many others HNers are asking now. It's actually a fairly detailed story, although annoyingly short on technical detail (which could have been because their sources refused to go into much detail).
  - hef19898 8y ago
    I did read it, so if it was mentioned I missed the part where Bloomberg stated if it was targeted at specific orders and thus customers or if it was seeded into the supply chain to see what sticks. So, if you know the answer, why not just providing the information? ;-)
sitkack 8y ago
I think the Command and Control traffic will be much harder to spot. In fact it might be dormant until an operative is on the same network and wakes it up.

dvfjsdhgfv 8y ago

That's the interesting bit. Having done so much to hide the exploit your most important aim is to hide it's presence. Anything that would just "connect to a Chinese server" would be discovered immediately. If the Bloomberg story is true, the network traffic scheme must have been extremely sophisticated to fool so many network security specialists at top companies for such a long time. This, or the story is false, purposely or not.

ytjohn 8y ago
Most places I know of isolate their OOB management network, requiring a vpn or jumpbox to access it. However, if someone did let their OOB network full outbound access, I could see this slipping through. I could imagine that simply going to a CDN or cloud provider like AWS/Cloudfront/cloudflare/akamai with a dns lookup along the lines of updates.supermicro.cdn-front.com wouldn't be too suspicious. At that point, you'd be looking for dns lookups and not firewall hits. If you are blocking outbound, I could still this going unnoticed if you're not actively reviewing denials. But, if you are properly watching dns lookups from OOB and it's anything other than necessary services (ntp, ldap, syslog), then this would get picked up pretty quickly.
- fulafel 8y ago
  Sounds like many things would have to go right in the defender's court. Optimism is not a good defense strategy :)
  - ytjohn 8y ago
    Heh. One colleague has "Hope is not a valid deployment strategy" as a signature. Most places might be blocking this type of activity by default. For most of our security audits, it's just assumed that the SM IPMI or Dell idrac is vulnerable to one exploit or another. We mitigate that by controlling the traffic. I feel this is common practice in most places that understand vlans and firewalls. However, while blocking is easy, being aware of something like this is on another level altogether. Unicorn jumping over a rainbow level rare. You really have to be logging outbound attempts and dns lookups. Where I work, there is a full security team and they are at an insane level where they log the allowed traffic. One told me that the allowed traffic is more interesting than the denied traffic. Denied just tells them what we anticipated, while active helps them establish a pattern and look for deviations.
    - dvfjsdhgfv 8y ago
      That's my point. Security people were analyzing network traffic for decades trying to spot something that doesn't fit, host-wise, pattern-wise or even packet-wise (see The Museum of Broken Packets[0], for example). And someone managed to somehow hide all this traffic from security experts working for Amazon and Apple, for months or years? I'm very curious to see how. [0] http://lcamtuf.coredump.cx/mobp/ http://lcamtuf.coredump.cx/mobp/

hef19898 8y ago

That is a good question, did the Chinese target specific customers, identifying the relevant sub-contractors and modify these boards? Or are these boards commercial-of-the-shelf things that are mass produced? In the latter case these boards can just end up anywhere, in the first case not so much. As a result using commodity boards would provide some degree of protection against such an attack as slipping a manipulated one into a specific target's servers would be a game of chance. Also the risk that the attack is discovered would be much higher.

jnbiche 8y ago
> That is a good question, did the Chinese target specific customers, identifying the relevant sub-contractors and modify these boards? Why not go read the story? It addresses this question and many others HNers are asking now. It's actually a fairly detailed story, although annoyingly short on technical detail (which could have been because their sources refused to go into much detail).
- hef19898 8y ago
  I did read it, so if it was mentioned I missed the part where Bloomberg stated if it was targeted at specific orders and thus customers or if it was seeded into the supply chain to see what sticks. So, if you know the answer, why not just providing the information? ;-)

sitkack 8y ago

I think the Command and Control traffic will be much harder to spot. In fact it might be dormant until an operative is on the same network and wakes it up.