9 ms·

This is the key excerpt, its quite shocking: Crowdstrike said Sunspot was written to be able to detect when it was installed on a SolarWinds developer syst

by f430 6y ago

This is the key excerpt, its quite shocking: Crowdstrike said Sunspot was written to be able to detect when it was installed on a SolarWinds developer system, and to lie in wait until specific Orion source code files were accessed by developers. This allowed the intruders to “replace source code files during the build process, before compilation,” Crowdstrike wrote. The attackers also included safeguards to prevent the backdoor code lines from appearing in Orion software build logs, and checks to ensure that such tampering wouldn’t cause build errors. “The design of SUNSPOT suggests [the malware] developers invested a lot of effort to ensure the code was properly inserted and remained undetected, and prioritized operational security to avoid revealing their presence in the build environment to SolarWinds developers,” CrowdStrike wrote. So how do we guard against this type of attack? How do we know this hasn't already happened to some of us? What is the potential fallout from this hack, it seems quite significant. This must be why the Japanese Intelligence agencies prefer paper over computer systems. The digitization of critical national security apparatus is the Archilles Heel that is being exploited successfully. One example is Japan's intelligence gathering capabilities in East Asia, especially China, which is bar none. Japan has a better linguistic understanding of the Chinese language (Kanji and all) but also interestingly much of PRC's massive public surveillance equipment like CCTV cameras are made in Japan. Even if they hire Krebs, I believe that if its digital, it can be hacked given long enough time period and unlimited state level backing and head hunting essentially geniuses of their country to do their bidding. I wonder how Biden-Harris administration will respond, it is very clear who the state actor is here. I'm very nervous about the ramifications of this hack.

j_walter 6y ago
It was not done by a bunch of amateurs that is for sure. Now everything points to Russia, but that is also the most obvious clues to leave as who would question it. However...we know the NSA wants to be in every system and this kind of operational security and evasion screams of the NSA to me.
- f430 6y ago
  > Now everything points to Russia How do we know this is Russians? To my knowledge its very common practice to obfusticate origins before launching a campaign like this by washing through several different countries. You could leave stuff like comments or references that would suggest it was the Russians, there's just no way of knowing, so I follow the fundamentals of political sabotage: whoever benefits most is the culprit. Who has the most to lose and gain here?
  - NikolaeVarius 6y ago
    Yeah, no. The various Russian APTs have tooling they prefer to use and are attributable to them. This is generally fairly stable because these are professionals who spend years learning specific toolchains, programming, and skills, and do not really change it up much, since they don't have to. Even if they're attributed, what is the world going to do? Toss a bomb into Russia? And before you get started, yes, security professionals are aware that you can obfuscate that, but there are already techniques to defeat this second layer of obfuscation. If multiple sources are saying this was probably Russia, they probably have a decent bit of proof.
    - f430 6y ago
      Hmm I hadn't considered that but how do you find out what tools were being used to produce the payload source code? How can you be certain? Could another adversary simply use the same tooling or perhaps it is shared to an allied nation (enemy of my enemy is a friend) to do its bidding.
      - NikolaeVarius 6y ago
        These people are incredibly smart. https://link.springer.com/article/10.1186/s42400-020-00048-4 https://link.springer.com/article/10.1186/s42400-020-00048-4 https://www.blackhat.com/html/webcast/07072020-how-attackers-confuse-investigators-with-cyber-false-flag-attacks.html https://www.blackhat.com/html/webcast/07072020-how-attackers... TLDR. > They highlight that not only malware samples and their specific properties (such as compiler settings, language settings, certain re-occurring patterns and the like) are useful, but also information available outside the actually attacked infrastructure, including data on the command & control infrastructure. Yes you can obfuscate certain things, buts its hard to obfuscate EVERYTHING, and if you dig deep enough, you can make a decent effort finding the owner.
- zests 6y ago
  Could the NSA just tell the rest of the government to pretend it was Russia? It's not like any of us would know better.
deleted 6y ago
[deleted]
dilyevsky 6y ago
> So how do we guard against this type of attack? Don’t give people running windows machines access to your source code/production
- _wldu 6y ago
  Especially if they are domain joined.
  - dilyevsky 6y ago
    Just don’t. Inb4 “but Microsoft has most sophisticated security” crowd, every major hack of the last 15 years always starts with compromised windows box that allows attacker an outpost to move laterally.
    - intern4tional 6y ago
      Then you should go review major security events. Attackers have been able to compromise major environments regardless of OS type. Windows having a larger market share is clearly going to have more compromises, but other OS based environments are just as prone to compromise. There also have been major incidents with standalone applications that are OS agnostic. Heartbleed is a great example of an OS agnostic flaw that was used for major attacks. As noted earlier while the OS does play an issue, its more than likely just correlation rather than causation due to Microsoft having a dominant position in the market. Reference: https://en.wikipedia.org/wiki/Heartbleed#Exploitation https://en.wikipedia.org/wiki/Heartbleed#Exploitation
      - dilyevsky 6y ago
        Source of major leak/intrusion as a result of heartbleed? That was 2013 btw and I remember it clearly - since it was closed pretty quickly I don’t recall anything major coming out of it Edit: i mean when it was 0day. Obviously if some idiot didn’t patch their servers it shouldn’t count
        6y ago
rodgerd 6y ago
> So how do we guard against this type of attack? One big issue with a lot of security and enterprise ops tooling is that it doesn't follow good practice around, well, security. For example, security code analysis software with hard-coded passwords that you hook into your build tooling, or in this case, ops software that instructs you to disable Microsoft's security tools so they don't flag problems with the agent. In a similar vein I've had BMC software want the highest levels of access to Oracle DBs to do simply monitoring, and so on and so forth. The other observation I heard Bruce Schneier make at a hacker con is more profound, and probably going to take a lot longer for national security actors to accept is this: the norms need to change. There is no longer a clear separation between "our stuff" and "their stuff" the way that there was a few decades ago, when espionage was more on "your telco network" or "my telco network". As we've moved to pervasive connectivity it's no longer possible to say, "oh that backdoor will only be used by our side", or "that hack will only affect their SCADA systems" or whatever.
- f430 6y ago
  I think this is the best answer out of all the replies.
bostik 6y ago
> So how do we guard against this type of attack? You can't ever prevent it, but you can raise the attack cost/complexity and make detection much more likely. Go for immutable infra and transient CI/CD systems. Provision the build nodes on demand, from trusted images, and wipe them out completely after just a few hours. The attacker will have to re-infect the nodes as they come online, and risk leaving more network traces. Anything they deploy on the systems goes away when the node goes away. The attack against SolarWinds worked so well because the CI system was persistent and it was upgraded over time. For a reliable and secure environment, the correct amount of host configuration is zero: if you need to modify anything at all after a system has been launched, that's a bad smell. Don't upgrade. Provision a completely new system with the more recent versions. This kind of architecture requires the attacker to compromise the CI image build and/or storage instead. (Or the upstream, directly.) It's an extra step for adversary, and a simpler point of control to the defender. Recon. Compromise. Persist. Exfiltrate. -- As a defender you want to make every step as expensive and brittle as possible. You can't prevent a compromise, but you can make it less useful, and you can arrange things so that it must happen often and leave enough traces to increase the probability of getting caught.
- joe_the_user 6y ago
  Another thing that might have made this hard - if Solar Winds were distributed as source code and each client built it themselves, with their own options (though the old back-doored c-compiler "thought experiment" may not be as much of a thought experiment anymore). Moreover, achieving the hack was likely costly given the effort and the benefit of the hack appeared once the Solar Winds binary was distributed. You can reduce the benefit of such a hack by not having information critical enterprises all running the same binary blob.
  - SahAssar 6y ago
    > You can reduce the benefit of such a hack by not having information critical enterprises all running the same binary blob. That assumes that you actually inspect the source, right?
Spooky23 6y ago
You have to segment and have monitoring tools monitored by people with a clue. But that is very expensive to do. The average SaaS or software company does nothing.
benlivengood 6y ago
> So how do we guard against this type of attack? How do we know this hasn't already happened to some of us? What is the potential fallout from this hack, it seems quite significant. Verified builds. That means deterministic builds (roughly, from a given git commit the same binaries should result no matter who compiles them. It requires compiler support and sometimes changes to the code) plus trusted build infrastructure. To verify that you haven't been compromised do a verified build from two independent roots of trust and compare the resulting binaries. Add more roots of trust to reduce the probability that all of them are compromised. Establishing a trusted root build environment is tricky because very little software has deterministic builds yet. Once they do it'll be much easier. Here's my best shot at it: Get a bunch of fresh openbsd machines. Don't network them together. Add some windows machines if you're planning to use VS. Pick 3 or more C compilers. Grab the source, verify with pgp on a few machines using a few different clients. For each one, compile it as much as possible with the others. This won't be possible in whole due to some extensions only available in a particular compiler used in its source, but is the best we can do at this point. Build all your compilers with each of these stage-2 compilers. Repeat until you have N-choose-N stage-N compilers. At this point any deterministic builds by a particular compiler (gcc, llvm, VS) should exactly match despite the compilers themselves being compiled in different orders by different compilers. This partially addresses Ken Thompson's paper "reflections on trusting trust" by requiring any persistent compiler backdoors to be mutually compatible across many different compilers otherwise it'll be detected as mismatched output from some compiler build ancestries but not others. Now you have some trusted compiler binaries. Git repository hashes can be the root of trust for remaining software. Using a few different github client implementations verify all the hashes match on the entire merkle tree. Build them with trusted compilers of your choice on multiple machines and verify the results match where possible. At this point you should have compilers, kernels, and system libraries that are most likely true to the verified source code. Make a couple build farms and keep them administratively separate. No common passwords, ssh keys, update servers, etc. Make sure builds on both farms match before trusting the binaries. The good news is that most of this can be done by the open source community; if everyone starts sharing the hashes of their git trees before builds and the hashes of the resulting binaries we could start making a global consensus of what software can currently be built deterministically and out of those which are very likely to be true translations from source code. EDIT: https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds is Debian's attempt at this.
nvader 6y ago
This reminds me of the Ken Thompson hack: https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html
twistedpair 6y ago
We're all screwed. Predicted long ago. See _Reflections On Trusting Trust_ [1] [1] https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
stelfer 6y ago
> So how do we guard against this type of attack? Looks like they compromised the editor. If so, then I imagine checking checksums for each component of the toolchain would work. Though if they compromised the filesystem or runtime then that would complicate things. But still, a hash tree or certificate of the OS and toolchain as part of CI would seem to be a good idea in 2021.
marcinzm 6y ago
>So how do we guard against this type of attack? Only allow restricted CI servers to build and deploy production code. Whitelisted list of software and security monitoring for CI servers so malicious software is harder to install. Whitelisted list of software that developer machines can run (ie: no arbitrary third party code). Restricted docker containers for all local testing. There's a reason various companies do not give developers admin access to their machines. edit: This or related concepts (SDLC, audit logs reviewed weekly, security team sign offs on system changes, etc.) are in my experience pretty standard in large enterprise security reviews for vendors that were high risk. The issue is that everyone probably lies and doesn't actually do any of these things even if they say they do. The incentives are set up to certify vendors rather than fail them.

by f430 6y ago

j_walter 6y ago

It was not done by a bunch of amateurs that is for sure. Now everything points to Russia, but that is also the most obvious clues to leave as who would question it. However...we know the NSA wants to be in every system and this kind of operational security and evasion screams of the NSA to me.

f430 6y ago
> Now everything points to Russia How do we know this is Russians? To my knowledge its very common practice to obfusticate origins before launching a campaign like this by washing through several different countries. You could leave stuff like comments or references that would suggest it was the Russians, there's just no way of knowing, so I follow the fundamentals of political sabotage: whoever benefits most is the culprit. Who has the most to lose and gain here?
- NikolaeVarius 6y ago
  Yeah, no. The various Russian APTs have tooling they prefer to use and are attributable to them. This is generally fairly stable because these are professionals who spend years learning specific toolchains, programming, and skills, and do not really change it up much, since they don't have to. Even if they're attributed, what is the world going to do? Toss a bomb into Russia? And before you get started, yes, security professionals are aware that you can obfuscate that, but there are already techniques to defeat this second layer of obfuscation. If multiple sources are saying this was probably Russia, they probably have a decent bit of proof.
  - f430 6y ago
    Hmm I hadn't considered that but how do you find out what tools were being used to produce the payload source code? How can you be certain? Could another adversary simply use the same tooling or perhaps it is shared to an allied nation (enemy of my enemy is a friend) to do its bidding.
    - NikolaeVarius 6y ago
      These people are incredibly smart. https://link.springer.com/article/10.1186/s42400-020-00048-4 https://link.springer.com/article/10.1186/s42400-020-00048-4 https://www.blackhat.com/html/webcast/07072020-how-attackers-confuse-investigators-with-cyber-false-flag-attacks.html https://www.blackhat.com/html/webcast/07072020-how-attackers... TLDR. > They highlight that not only malware samples and their specific properties (such as compiler settings, language settings, certain re-occurring patterns and the like) are useful, but also information available outside the actually attacked infrastructure, including data on the command & control infrastructure. Yes you can obfuscate certain things, buts its hard to obfuscate EVERYTHING, and if you dig deep enough, you can make a decent effort finding the owner.
zests 6y ago
Could the NSA just tell the rest of the government to pretend it was Russia? It's not like any of us would know better.

deleted 6y ago

[deleted]

dilyevsky 6y ago

> So how do we guard against this type of attack? Don’t give people running windows machines access to your source code/production

_wldu 6y ago
Especially if they are domain joined.
- dilyevsky 6y ago
  Just don’t. Inb4 “but Microsoft has most sophisticated security” crowd, every major hack of the last 15 years always starts with compromised windows box that allows attacker an outpost to move laterally.
  - intern4tional 6y ago
    Then you should go review major security events. Attackers have been able to compromise major environments regardless of OS type. Windows having a larger market share is clearly going to have more compromises, but other OS based environments are just as prone to compromise. There also have been major incidents with standalone applications that are OS agnostic. Heartbleed is a great example of an OS agnostic flaw that was used for major attacks. As noted earlier while the OS does play an issue, its more than likely just correlation rather than causation due to Microsoft having a dominant position in the market. Reference: https://en.wikipedia.org/wiki/Heartbleed#Exploitation https://en.wikipedia.org/wiki/Heartbleed#Exploitation
    - dilyevsky 6y ago
      Source of major leak/intrusion as a result of heartbleed? That was 2013 btw and I remember it clearly - since it was closed pretty quickly I don’t recall anything major coming out of it Edit: i mean when it was 0day. Obviously if some idiot didn’t patch their servers it shouldn’t count
      - 6y ago

rodgerd 6y ago

> So how do we guard against this type of attack? One big issue with a lot of security and enterprise ops tooling is that it doesn't follow good practice around, well, security. For example, security code analysis software with hard-coded passwords that you hook into your build tooling, or in this case, ops software that instructs you to disable Microsoft's security tools so they don't flag problems with the agent. In a similar vein I've had BMC software want the highest levels of access to Oracle DBs to do simply monitoring, and so on and so forth. The other observation I heard Bruce Schneier make at a hacker con is more profound, and probably going to take a lot longer for national security actors to accept is this: the norms need to change. There is no longer a clear separation between "our stuff" and "their stuff" the way that there was a few decades ago, when espionage was more on "your telco network" or "my telco network". As we've moved to pervasive connectivity it's no longer possible to say, "oh that backdoor will only be used by our side", or "that hack will only affect their SCADA systems" or whatever.

f430 6y ago
I think this is the best answer out of all the replies.

bostik 6y ago

> So how do we guard against this type of attack? You can't ever prevent it, but you can raise the attack cost/complexity and make detection much more likely. Go for immutable infra and transient CI/CD systems. Provision the build nodes on demand, from trusted images, and wipe them out completely after just a few hours. The attacker will have to re-infect the nodes as they come online, and risk leaving more network traces. Anything they deploy on the systems goes away when the node goes away. The attack against SolarWinds worked so well because the CI system was persistent and it was upgraded over time. For a reliable and secure environment, the correct amount of host configuration is zero: if you need to modify anything at all after a system has been launched, that's a bad smell. Don't upgrade. Provision a completely new system with the more recent versions. This kind of architecture requires the attacker to compromise the CI image build and/or storage instead. (Or the upstream, directly.) It's an extra step for adversary, and a simpler point of control to the defender. Recon. Compromise. Persist. Exfiltrate. -- As a defender you want to make every step as expensive and brittle as possible. You can't prevent a compromise, but you can make it less useful, and you can arrange things so that it must happen often and leave enough traces to increase the probability of getting caught.

joe_the_user 6y ago
Another thing that might have made this hard - if Solar Winds were distributed as source code and each client built it themselves, with their own options (though the old back-doored c-compiler "thought experiment" may not be as much of a thought experiment anymore). Moreover, achieving the hack was likely costly given the effort and the benefit of the hack appeared once the Solar Winds binary was distributed. You can reduce the benefit of such a hack by not having information critical enterprises all running the same binary blob.
- SahAssar 6y ago
  > You can reduce the benefit of such a hack by not having information critical enterprises all running the same binary blob. That assumes that you actually inspect the source, right?

Spooky23 6y ago

You have to segment and have monitoring tools monitored by people with a clue. But that is very expensive to do. The average SaaS or software company does nothing.

benlivengood 6y ago

> So how do we guard against this type of attack? How do we know this hasn't already happened to some of us? What is the potential fallout from this hack, it seems quite significant. Verified builds. That means deterministic builds (roughly, from a given git commit the same binaries should result no matter who compiles them. It requires compiler support and sometimes changes to the code) plus trusted build infrastructure. To verify that you haven't been compromised do a verified build from two independent roots of trust and compare the resulting binaries. Add more roots of trust to reduce the probability that all of them are compromised. Establishing a trusted root build environment is tricky because very little software has deterministic builds yet. Once they do it'll be much easier. Here's my best shot at it: Get a bunch of fresh openbsd machines. Don't network them together. Add some windows machines if you're planning to use VS. Pick 3 or more C compilers. Grab the source, verify with pgp on a few machines using a few different clients. For each one, compile it as much as possible with the others. This won't be possible in whole due to some extensions only available in a particular compiler used in its source, but is the best we can do at this point. Build all your compilers with each of these stage-2 compilers. Repeat until you have N-choose-N stage-N compilers. At this point any deterministic builds by a particular compiler (gcc, llvm, VS) should exactly match despite the compilers themselves being compiled in different orders by different compilers. This partially addresses Ken Thompson's paper "reflections on trusting trust" by requiring any persistent compiler backdoors to be mutually compatible across many different compilers otherwise it'll be detected as mismatched output from some compiler build ancestries but not others. Now you have some trusted compiler binaries. Git repository hashes can be the root of trust for remaining software. Using a few different github client implementations verify all the hashes match on the entire merkle tree. Build them with trusted compilers of your choice on multiple machines and verify the results match where possible. At this point you should have compilers, kernels, and system libraries that are most likely true to the verified source code. Make a couple build farms and keep them administratively separate. No common passwords, ssh keys, update servers, etc. Make sure builds on both farms match before trusting the binaries. The good news is that most of this can be done by the open source community; if everyone starts sharing the hashes of their git trees before builds and the hashes of the resulting binaries we could start making a global consensus of what software can currently be built deterministically and out of those which are very likely to be true translations from source code. EDIT: https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds is Debian's attempt at this.

nvader 6y ago

This reminds me of the Ken Thompson hack: https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html

twistedpair 6y ago

We're all screwed. Predicted long ago. See _Reflections On Trusting Trust_ [1] [1] https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...

stelfer 6y ago

> So how do we guard against this type of attack? Looks like they compromised the editor. If so, then I imagine checking checksums for each component of the toolchain would work. Though if they compromised the filesystem or runtime then that would complicate things. But still, a hash tree or certificate of the OS and toolchain as part of CI would seem to be a good idea in 2021.

marcinzm 6y ago

>So how do we guard against this type of attack? Only allow restricted CI servers to build and deploy production code. Whitelisted list of software and security monitoring for CI servers so malicious software is harder to install. Whitelisted list of software that developer machines can run (ie: no arbitrary third party code). Restricted docker containers for all local testing. There's a reason various companies do not give developers admin access to their machines. edit: This or related concepts (SDLC, audit logs reviewed weekly, security team sign offs on system changes, etc.) are in my experience pretty standard in large enterprise security reviews for vendors that were high risk. The issue is that everyone probably lies and doesn't actually do any of these things even if they say they do. The incentives are set up to certify vendors rather than fail them.