16 ms·

SolarWinds: What Hit Us Could Hit Others

196 points by parsecs 6y ago

f430 6y ago
This is the key excerpt, its quite shocking: Crowdstrike said Sunspot was written to be able to detect when it was installed on a SolarWinds developer system, and to lie in wait until specific Orion source code files were accessed by developers. This allowed the intruders to “replace source code files during the build process, before compilation,” Crowdstrike wrote. The attackers also included safeguards to prevent the backdoor code lines from appearing in Orion software build logs, and checks to ensure that such tampering wouldn’t cause build errors. “The design of SUNSPOT suggests [the malware] developers invested a lot of effort to ensure the code was properly inserted and remained undetected, and prioritized operational security to avoid revealing their presence in the build environment to SolarWinds developers,” CrowdStrike wrote. So how do we guard against this type of attack? How do we know this hasn't already happened to some of us? What is the potential fallout from this hack, it seems quite significant. This must be why the Japanese Intelligence agencies prefer paper over computer systems. The digitization of critical national security apparatus is the Archilles Heel that is being exploited successfully. One example is Japan's intelligence gathering capabilities in East Asia, especially China, which is bar none. Japan has a better linguistic understanding of the Chinese language (Kanji and all) but also interestingly much of PRC's massive public surveillance equipment like CCTV cameras are made in Japan. Even if they hire Krebs, I believe that if its digital, it can be hacked given long enough time period and unlimited state level backing and head hunting essentially geniuses of their country to do their bidding. I wonder how Biden-Harris administration will respond, it is very clear who the state actor is here. I'm very nervous about the ramifications of this hack.
- j_walter 6y ago
  It was not done by a bunch of amateurs that is for sure. Now everything points to Russia, but that is also the most obvious clues to leave as who would question it. However...we know the NSA wants to be in every system and this kind of operational security and evasion screams of the NSA to me.
  - f430 6y ago
    > Now everything points to Russia How do we know this is Russians? To my knowledge its very common practice to obfusticate origins before launching a campaign like this by washing through several different countries. You could leave stuff like comments or references that would suggest it was the Russians, there's just no way of knowing, so I follow the fundamentals of political sabotage: whoever benefits most is the culprit. Who has the most to lose and gain here?
    - NikolaeVarius 6y ago
      Yeah, no. The various Russian APTs have tooling they prefer to use and are attributable to them. This is generally fairly stable because these are professionals who spend years learning specific toolchains, programming, and skills, and do not really change it up much, since they don't have to. Even if they're attributed, what is the world going to do? Toss a bomb into Russia? And before you get started, yes, security professionals are aware that you can obfuscate that, but there are already techniques to defeat this second layer of obfuscation. If multiple sources are saying this was probably Russia, they probably have a decent bit of proof.
      - f430 6y ago
        Hmm I hadn't considered that but how do you find out what tools were being used to produce the payload source code? How can you be certain? Could another adversary simply use the same tooling or perhaps it is shared to an allied nation (enemy of my enemy is a friend) to do its bidding.
        NikolaeVarius 6y ago
        These people are incredibly smart. https://link.springer.com/article/10.1186/s42400-020-00048-4 https://link.springer.com/article/10.1186/s42400-020-00048-4 https://www.blackhat.com/html/webcast/07072020-how-attackers-confuse-investigators-with-cyber-false-flag-attacks.html https://www.blackhat.com/html/webcast/07072020-how-attackers... TLDR. > They highlight that not only malware samples and their specific properties (such as compiler settings, language settings, certain re-occurring patterns and the like) are useful, but also information available outside the actually attacked infrastructure, including data on the command & control infrastructure. Yes you can obfuscate certain things, buts its hard to obfuscate EVERYTHING, and if you dig deep enough, you can make a decent effort finding the owner.
- deleted 6y ago
  [deleted]
- dilyevsky 6y ago
  > So how do we guard against this type of attack? Don’t give people running windows machines access to your source code/production
  - _wldu 6y ago
    Especially if they are domain joined.
    - dilyevsky 6y ago
      Just don’t. Inb4 “but Microsoft has most sophisticated security” crowd, every major hack of the last 15 years always starts with compromised windows box that allows attacker an outpost to move laterally.
      - intern4tional 6y ago
        Then you should go review major security events. Attackers have been able to compromise major environments regardless of OS type. Windows having a larger market share is clearly going to have more compromises, but other OS based environments are just as prone to compromise. There also have been major incidents with standalone applications that are OS agnostic. Heartbleed is a great example of an OS agnostic flaw that was used for major attacks. As noted earlier while the OS does play an issue, its more than likely just correlation rather than causation due to Microsoft having a dominant position in the market. Reference: https://en.wikipedia.org/wiki/Heartbleed#Exploitation https://en.wikipedia.org/wiki/Heartbleed#Exploitation
        dilyevsky 6y ago
        Source of major leak/intrusion as a result of heartbleed? That was 2013 btw and I remember it clearly - since it was closed pretty quickly I don’t recall anything major coming out of it Edit: i mean when it was 0day. Obviously if some idiot didn’t patch their servers it shouldn’t count
        6y ago
- rodgerd 6y ago
  > So how do we guard against this type of attack? One big issue with a lot of security and enterprise ops tooling is that it doesn't follow good practice around, well, security. For example, security code analysis software with hard-coded passwords that you hook into your build tooling, or in this case, ops software that instructs you to disable Microsoft's security tools so they don't flag problems with the agent. In a similar vein I've had BMC software want the highest levels of access to Oracle DBs to do simply monitoring, and so on and so forth. The other observation I heard Bruce Schneier make at a hacker con is more profound, and probably going to take a lot longer for national security actors to accept is this: the norms need to change. There is no longer a clear separation between "our stuff" and "their stuff" the way that there was a few decades ago, when espionage was more on "your telco network" or "my telco network". As we've moved to pervasive connectivity it's no longer possible to say, "oh that backdoor will only be used by our side", or "that hack will only affect their SCADA systems" or whatever.
  - f430 6y ago
    I think this is the best answer out of all the replies.
- bostik 6y ago
  > So how do we guard against this type of attack? You can't ever prevent it, but you can raise the attack cost/complexity and make detection much more likely. Go for immutable infra and transient CI/CD systems. Provision the build nodes on demand, from trusted images, and wipe them out completely after just a few hours. The attacker will have to re-infect the nodes as they come online, and risk leaving more network traces. Anything they deploy on the systems goes away when the node goes away. The attack against SolarWinds worked so well because the CI system was persistent and it was upgraded over time. For a reliable and secure environment, the correct amount of host configuration is zero: if you need to modify anything at all after a system has been launched, that's a bad smell. Don't upgrade. Provision a completely new system with the more recent versions. This kind of architecture requires the attacker to compromise the CI image build and/or storage instead. (Or the upstream, directly.) It's an extra step for adversary, and a simpler point of control to the defender. Recon. Compromise. Persist. Exfiltrate. -- As a defender you want to make every step as expensive and brittle as possible. You can't prevent a compromise, but you can make it less useful, and you can arrange things so that it must happen often and leave enough traces to increase the probability of getting caught.
  - joe_the_user 6y ago
    Another thing that might have made this hard - if Solar Winds were distributed as source code and each client built it themselves, with their own options (though the old back-doored c-compiler "thought experiment" may not be as much of a thought experiment anymore). Moreover, achieving the hack was likely costly given the effort and the benefit of the hack appeared once the Solar Winds binary was distributed. You can reduce the benefit of such a hack by not having information critical enterprises all running the same binary blob.
    - SahAssar 6y ago
      > You can reduce the benefit of such a hack by not having information critical enterprises all running the same binary blob. That assumes that you actually inspect the source, right?
- Spooky23 6y ago
  You have to segment and have monitoring tools monitored by people with a clue. But that is very expensive to do. The average SaaS or software company does nothing.
- benlivengood 6y ago
  > So how do we guard against this type of attack? How do we know this hasn't already happened to some of us? What is the potential fallout from this hack, it seems quite significant. Verified builds. That means deterministic builds (roughly, from a given git commit the same binaries should result no matter who compiles them. It requires compiler support and sometimes changes to the code) plus trusted build infrastructure. To verify that you haven't been compromised do a verified build from two independent roots of trust and compare the resulting binaries. Add more roots of trust to reduce the probability that all of them are compromised. Establishing a trusted root build environment is tricky because very little software has deterministic builds yet. Once they do it'll be much easier. Here's my best shot at it: Get a bunch of fresh openbsd machines. Don't network them together. Add some windows machines if you're planning to use VS. Pick 3 or more C compilers. Grab the source, verify with pgp on a few machines using a few different clients. For each one, compile it as much as possible with the others. This won't be possible in whole due to some extensions only available in a particular compiler used in its source, but is the best we can do at this point. Build all your compilers with each of these stage-2 compilers. Repeat until you have N-choose-N stage-N compilers. At this point any deterministic builds by a particular compiler (gcc, llvm, VS) should exactly match despite the compilers themselves being compiled in different orders by different compilers. This partially addresses Ken Thompson's paper "reflections on trusting trust" by requiring any persistent compiler backdoors to be mutually compatible across many different compilers otherwise it'll be detected as mismatched output from some compiler build ancestries but not others. Now you have some trusted compiler binaries. Git repository hashes can be the root of trust for remaining software. Using a few different github client implementations verify all the hashes match on the entire merkle tree. Build them with trusted compilers of your choice on multiple machines and verify the results match where possible. At this point you should have compilers, kernels, and system libraries that are most likely true to the verified source code. Make a couple build farms and keep them administratively separate. No common passwords, ssh keys, update servers, etc. Make sure builds on both farms match before trusting the binaries. The good news is that most of this can be done by the open source community; if everyone starts sharing the hashes of their git trees before builds and the hashes of the resulting binaries we could start making a global consensus of what software can currently be built deterministically and out of those which are very likely to be true translations from source code. EDIT: https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds is Debian's attempt at this.
- nvader 6y ago
  This reminds me of the Ken Thompson hack: https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html
- twistedpair 6y ago
  We're all screwed. Predicted long ago. See _Reflections On Trusting Trust_ [1] [1] https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
- stelfer 6y ago
  > So how do we guard against this type of attack? Looks like they compromised the editor. If so, then I imagine checking checksums for each component of the toolchain would work. Though if they compromised the filesystem or runtime then that would complicate things. But still, a hash tree or certificate of the OS and toolchain as part of CI would seem to be a good idea in 2021.
- marcinzm 6y ago
  >So how do we guard against this type of attack? Only allow restricted CI servers to build and deploy production code. Whitelisted list of software and security monitoring for CI servers so malicious software is harder to install. Whitelisted list of software that developer machines can run (ie: no arbitrary third party code). Restricted docker containers for all local testing. There's a reason various companies do not give developers admin access to their machines. edit: This or related concepts (SDLC, audit logs reviewed weekly, security team sign offs on system changes, etc.) are in my experience pretty standard in large enterprise security reviews for vendors that were high risk. The issue is that everyone probably lies and doesn't actually do any of these things even if they say they do. The incentives are set up to certify vendors rather than fail them.
omgbobbyg 6y ago
As a citizen, I am shocked and appalled by this backdoor. As a software engineer, I can't help but marvel at the creativity and thoughtfulness put into the exploit.
- f430 6y ago
  The average engineer isn't an infosec expert and love automation so they found the weakest link in the chain: CI/CD
  - robbyt 6y ago
    We're not talking expert-level infosec here. They should have firewalled their CI systems. That's pretty basic stuff.
HALtheWise 6y ago
https://www.crowdstrike.com/blog/sunspot-malware-technical-analysis/ https://www.crowdstrike.com/blog/sunspot-malware-technical-a... is the key link with more technical analysis for those interested, including source code of the implant. "If the decryption of the parameters (target file path and replacement source code) is successful and if the MD5 checks pass, SUNSPOT proceeds with the replacement of the source file content. The original source file is copied with a .bk extension (e.g., InventoryManager.bk), to back up the original content. The backdoored source is written to the same filename, but with a .tmp extension (e.g., InventoryManager.tmp), before being moved using MoveFileEx to the original filename (InventoryManager.cs). After these steps, the source file backdoored with SUNBURST will then be compiled as part of the standard process."
- woliveirajr 6y ago
  This, combined with comments from other threads, makes me think that SolarWinds wasn't the real target. They were just means to some specific high-value ends. When low-value companies were using it, the remote control would even remove the backdoor to avoid some accidental discovery. How, and how much, were the real targets affected?
  - jaywalk 6y ago
    SolarWinds was absolutely not the real target. The malware wouldn't even execute if the machine it was installed on was joined to a domain containing the string "solarwinds".
fancyfredbot 6y ago
Could and likely already has hit others
candiddevmike 6y ago
I'm surprised this hasn't caused the software industry to completely halt and rewrite/audit all third party libraries and dependencies. The entire software supply chain is highly trust-based, npm especially. Why aren't we seeing the start of a NIH dark age?
- cobookman 6y ago
  NPM is not trusted by major fortune enterprises. Many of the tech companies I've worked at banned using NPM in prod. And instead created their own NPM Clone with source code pinned into a private repo, which security audited. Adding a single NPM library became a total PITA, as it linked to another 100 other NPM libraries, which in-tern linked to an additional 100+ other NPM libraries. So adding a single NPM library to the private repo, meant adding 100s to 1000s of other NPM libraries. (E.g. left-pad [1]) Personally, I think this was a major reason for node.js not picking up more in the enterprise space. With Python & golang getting more traction. [1] https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/ https://qz.com/646467/how-one-programmer-broke-the-internet-...
  - mandelbrotwurst 6y ago
    > With Python & golang getting more traction. Does pip not have the same issues?
  - dgellow 6y ago
    Python and go have similar issues though. People do not audit and do no vendor their dependencies.
    - jniedrauer 6y ago
      At least python and go packages don't execute arbitrary code on developer's workstations at install time. If you pin dependencies to a known safe version, then you're relatively safe. With NPM, just typing `npm update` can pwn your workstation.
Veserv 6y ago
Oh good. The attackers were only in their systems since 9/4/19 before being detected on 12/12/20, so only 15 months of infiltration into SolarWinds' systems before detection. At least the payload was only deployed 2/20/20, so their customers were only completely infiltrated without detection for 8 months. Assuming the attackers could only get a 10 MB/s channel total per target even though they probably infected thousands to tens of thousands of machines per target, at ~20 million seconds that would constitute ~200 TB exfiltrated per customer or ~19 years of 1080p video. If an attacker has just one day to root around and exfiltrate they can easily get valuable information. If they are given 8 months they have already gotten everything of value for months and are just waiting around for any new data to come in. Think how inadequate your systems must be to let an attacker sit around in your systems for 8 months, it is mind-boggling how unqualified their systems are for their problems. And this is not just an indictment of SolarWinds. Just in case anybody forgets, it was the top-flight security company FireEye who discovered this breach after realizing they themselves were breached. A "best of the best" security company took 8 months before realizing that they or any of their customers had been breached. This is what "top-class" security buys you.
- vasco 6y ago
  Whenever I read a post as strongly worded as yours I wonder that its author must really be the best security minded engineer out there, and has never compromised on anything for a deadline or missed something because they simply didn't know it should be configured in a particular safe way. This definitely doesn't look good and there's probably many failures along the way, but jeez.
  - spijdar 6y ago
    I think it's fair to say that these problems shouldn't be blamed on individuals, but systemic failure. In my imagination, someone should have seen this at some point. The system should have had safeguards to ensure that, regardless of any individual's personal failures. That said, I agree it's a little harsh, since there's no evidence that anyone else could/has done better in this incident.
throwawaybutwhy 6y ago
Oh. Brian Krebs regurgitates a corporate press release by a company that has recently hired Chris Krebs (no relation), all the while skirting around the solarwinds123 gross negligence. A well-funded PR campaign has already resulted in a NY Times smear hit piece accusing another software company of colluding with the Russians. Back to the substance, it appears the investigation keeps a close lid on the real extent of the breach. It's nasty.
dbg31415 6y ago
Yes, but... Having worked at SolarWinds they're especially susceptible to demands from sales and marketing. "Go faster, ignore tech best practices, etc." It's not unique, but their culture is not a dev-first, or security-first, culture to say the least. Many product managers answer to marketing first, and don't have earnest tech backgrounds that would let them know right from wrong past sales numbers. The culture changed significantly when they went public the first time; it went from a place where devs built good tools... to a place looking to buy products / competitor products so they could charge more for their services. Look at how long it took them to get into cloud tools -- great example of how marketing and sales missed the boat because they were only focused on things they had sold before and not focused on systemic changes to the industry -- because technologists weren't driving. Anyway, like I've worked a lot places with better security built into the culture, better tech best practices built into the culture... that's all I'm trying to say. Knowing that attacks like this are out there... and it was just a matter of time before it happened, SolarWinds did next to nothing to avoid it happening to them.
- josho 6y ago
  > SolarWinds did next to nothing to avoid it happening to them And how does one stop this from happening to them? Assume 'standard' security practices are in place, e.g. firewalls. Clearly the attacker was able to get their payload to be executed by a developer in SolarWinds. This payload clearly didn't trigger antivirus software, and hid itself from the developer and the build systems. How does a typical software dev. team guard against this?
  - josephg 6y ago
    As others have mentioned, building on secured CI/CD infrastructure (+ ideally deterministic builds) would have helped here. But I'm not sure if thats good enough. Presumably if CI/CD was set up, some of the engineers would have had security credentials to configure any build servers anyway. And if the hackers could leverage that, they might have been able to just infect the build servers too. As annoying as it is, I can't help but think we need to bring parts of ios / android's security model to our desktop machines. Its ridiculous that a single rogue npm module can access any file on my computer, including SSH credentials and (probably) secure cookies in my browser's trust store. Our desktop security model is obsolete, and should do more to protect us from threats like this.
afrcnc 6y ago
From what we've seen until now, this company deserves everything that happened to it. Hope they go under. Ignoring security best practices for the chance of a quick buck.
- josho 6y ago
  What best practices were ignored? From what I'm reading here I'm terrified as I don't think the industry can protect from a targeted attack like this. The only thing we have going for us is defence in depth to make a targeted attack expensive (like this attack was), and that there are sophisticated attackers and very many software shops which means I have a low probability of being the next victim.
slt2021 6y ago
I am 99% percent sure the hackers are among the CUSTOMERS of Solar Winds. That way they were able to live-test infected SolarWinds distro in their own controlled environment and develop all possible mitigations and techniques - the sheer amount of these evading techniques suggests they were built up over time, and not instantly. Being Solar Winds customer and receiving infected updated versions every time gave them opportunity to perfect their techniques and hide for so long At least that what I would do if I were a hacker and wanted to persist and be very careful about not getting detected
- nallerooth 6y ago
  At least it would be safe to assume that they had access to several systems which received Orion updates. Given the attacker's dedication to the whole process, I'd say someone else's servers probably ran the tests for them.
- totalZero 6y ago
  Alternatively, it is not unimaginable for a foreign entity to bribe or otherwise compromise a SolarWinds employee.
  - somurzakov 6y ago
    it is much easier for highly skilled hacker to get fake identity and get employed at SW than to compromise an employee - although this is longer term play
    - cutemonster 6y ago
      > get fake identity Wouldn't Google, Apple etc do background checks that discovered fake identities? Or is it not so easy with background checks? Maybe SolarWinds would be less careful though?
      - slt2021 6y ago
        nation state spies can get fake identities issued legally by their own government and will be confirmed as usual during the background check
Stierlitz 6y ago
"So how do we guard against this type of attack?" Don't allow access to your 'secret' source code from the open Internet.
- m4rtink 6y ago
  Or actually do that and let everyone verify and build their own binaries. This is preatty much how Linux distros work, the multiple third party distro packagers packaging upstream code make it preatty hard to sneak in mallware unnoticed and almost impossible to affect multiple distros at once. In comparison proprietary software companies are a single point of failure where customers can't access the source and have no means to verify what ghe binaries they get actually contain.
exabrial 6y ago
At one end of the spectrum I need to make sure people aren't choosing dumb passwords and are applying software updates. On the other end, centralizing control makes a very juicy target for hackers.
wolverine876 6y ago
Similar attacks are going on right now, you just won't read about it until the year 202x, if ever. What are you doing about it?
dinny 6y ago
Do we know how the SUNSPOT malware got access to the build VMs?
ThinkBeat 6y ago
The scope of this thing is staggering to me. It must have taken a significant amount of time in prep and dev and then deploy and control. A serious investment in time and resources so money. But why? In order to burn so many nice tricks they had to be after something quite valuable in one way or another. What was the motherload they were after and did they get it? Will we ever know? Or was this harvesting of the intelligence and information there were needed for the real gold? They went wide, which might have been to obscure the real target, or they needed a lot of pieces from different sources
vondur 6y ago
I wonder is something as simple as two factor authentication for those who have access to the build servers may have helped prevent something like this attack?

196 points by parsecs 6y ago

f430 6y ago

This is the key excerpt, its quite shocking: Crowdstrike said Sunspot was written to be able to detect when it was installed on a SolarWinds developer system, and to lie in wait until specific Orion source code files were accessed by developers. This allowed the intruders to “replace source code files during the build process, before compilation,” Crowdstrike wrote. The attackers also included safeguards to prevent the backdoor code lines from appearing in Orion software build logs, and checks to ensure that such tampering wouldn’t cause build errors. “The design of SUNSPOT suggests [the malware] developers invested a lot of effort to ensure the code was properly inserted and remained undetected, and prioritized operational security to avoid revealing their presence in the build environment to SolarWinds developers,” CrowdStrike wrote. So how do we guard against this type of attack? How do we know this hasn't already happened to some of us? What is the potential fallout from this hack, it seems quite significant. This must be why the Japanese Intelligence agencies prefer paper over computer systems. The digitization of critical national security apparatus is the Archilles Heel that is being exploited successfully. One example is Japan's intelligence gathering capabilities in East Asia, especially China, which is bar none. Japan has a better linguistic understanding of the Chinese language (Kanji and all) but also interestingly much of PRC's massive public surveillance equipment like CCTV cameras are made in Japan. Even if they hire Krebs, I believe that if its digital, it can be hacked given long enough time period and unlimited state level backing and head hunting essentially geniuses of their country to do their bidding. I wonder how Biden-Harris administration will respond, it is very clear who the state actor is here. I'm very nervous about the ramifications of this hack.

j_walter 6y ago
It was not done by a bunch of amateurs that is for sure. Now everything points to Russia, but that is also the most obvious clues to leave as who would question it. However...we know the NSA wants to be in every system and this kind of operational security and evasion screams of the NSA to me.
- f430 6y ago
  > Now everything points to Russia How do we know this is Russians? To my knowledge its very common practice to obfusticate origins before launching a campaign like this by washing through several different countries. You could leave stuff like comments or references that would suggest it was the Russians, there's just no way of knowing, so I follow the fundamentals of political sabotage: whoever benefits most is the culprit. Who has the most to lose and gain here?
  - NikolaeVarius 6y ago
    Yeah, no. The various Russian APTs have tooling they prefer to use and are attributable to them. This is generally fairly stable because these are professionals who spend years learning specific toolchains, programming, and skills, and do not really change it up much, since they don't have to. Even if they're attributed, what is the world going to do? Toss a bomb into Russia? And before you get started, yes, security professionals are aware that you can obfuscate that, but there are already techniques to defeat this second layer of obfuscation. If multiple sources are saying this was probably Russia, they probably have a decent bit of proof.
    - f430 6y ago
      Hmm I hadn't considered that but how do you find out what tools were being used to produce the payload source code? How can you be certain? Could another adversary simply use the same tooling or perhaps it is shared to an allied nation (enemy of my enemy is a friend) to do its bidding.
      - NikolaeVarius 6y ago
        These people are incredibly smart. https://link.springer.com/article/10.1186/s42400-020-00048-4 https://link.springer.com/article/10.1186/s42400-020-00048-4 https://www.blackhat.com/html/webcast/07072020-how-attackers-confuse-investigators-with-cyber-false-flag-attacks.html https://www.blackhat.com/html/webcast/07072020-how-attackers... TLDR. > They highlight that not only malware samples and their specific properties (such as compiler settings, language settings, certain re-occurring patterns and the like) are useful, but also information available outside the actually attacked infrastructure, including data on the command & control infrastructure. Yes you can obfuscate certain things, buts its hard to obfuscate EVERYTHING, and if you dig deep enough, you can make a decent effort finding the owner.
deleted 6y ago
[deleted]
dilyevsky 6y ago
> So how do we guard against this type of attack? Don’t give people running windows machines access to your source code/production
- _wldu 6y ago
  Especially if they are domain joined.
  - dilyevsky 6y ago
    Just don’t. Inb4 “but Microsoft has most sophisticated security” crowd, every major hack of the last 15 years always starts with compromised windows box that allows attacker an outpost to move laterally.
    - intern4tional 6y ago
      Then you should go review major security events. Attackers have been able to compromise major environments regardless of OS type. Windows having a larger market share is clearly going to have more compromises, but other OS based environments are just as prone to compromise. There also have been major incidents with standalone applications that are OS agnostic. Heartbleed is a great example of an OS agnostic flaw that was used for major attacks. As noted earlier while the OS does play an issue, its more than likely just correlation rather than causation due to Microsoft having a dominant position in the market. Reference: https://en.wikipedia.org/wiki/Heartbleed#Exploitation https://en.wikipedia.org/wiki/Heartbleed#Exploitation
      - dilyevsky 6y ago
        Source of major leak/intrusion as a result of heartbleed? That was 2013 btw and I remember it clearly - since it was closed pretty quickly I don’t recall anything major coming out of it Edit: i mean when it was 0day. Obviously if some idiot didn’t patch their servers it shouldn’t count
        6y ago
rodgerd 6y ago
> So how do we guard against this type of attack? One big issue with a lot of security and enterprise ops tooling is that it doesn't follow good practice around, well, security. For example, security code analysis software with hard-coded passwords that you hook into your build tooling, or in this case, ops software that instructs you to disable Microsoft's security tools so they don't flag problems with the agent. In a similar vein I've had BMC software want the highest levels of access to Oracle DBs to do simply monitoring, and so on and so forth. The other observation I heard Bruce Schneier make at a hacker con is more profound, and probably going to take a lot longer for national security actors to accept is this: the norms need to change. There is no longer a clear separation between "our stuff" and "their stuff" the way that there was a few decades ago, when espionage was more on "your telco network" or "my telco network". As we've moved to pervasive connectivity it's no longer possible to say, "oh that backdoor will only be used by our side", or "that hack will only affect their SCADA systems" or whatever.
- f430 6y ago
  I think this is the best answer out of all the replies.
bostik 6y ago
> So how do we guard against this type of attack? You can't ever prevent it, but you can raise the attack cost/complexity and make detection much more likely. Go for immutable infra and transient CI/CD systems. Provision the build nodes on demand, from trusted images, and wipe them out completely after just a few hours. The attacker will have to re-infect the nodes as they come online, and risk leaving more network traces. Anything they deploy on the systems goes away when the node goes away. The attack against SolarWinds worked so well because the CI system was persistent and it was upgraded over time. For a reliable and secure environment, the correct amount of host configuration is zero: if you need to modify anything at all after a system has been launched, that's a bad smell. Don't upgrade. Provision a completely new system with the more recent versions. This kind of architecture requires the attacker to compromise the CI image build and/or storage instead. (Or the upstream, directly.) It's an extra step for adversary, and a simpler point of control to the defender. Recon. Compromise. Persist. Exfiltrate. -- As a defender you want to make every step as expensive and brittle as possible. You can't prevent a compromise, but you can make it less useful, and you can arrange things so that it must happen often and leave enough traces to increase the probability of getting caught.
- joe_the_user 6y ago
  Another thing that might have made this hard - if Solar Winds were distributed as source code and each client built it themselves, with their own options (though the old back-doored c-compiler "thought experiment" may not be as much of a thought experiment anymore). Moreover, achieving the hack was likely costly given the effort and the benefit of the hack appeared once the Solar Winds binary was distributed. You can reduce the benefit of such a hack by not having information critical enterprises all running the same binary blob.
  - SahAssar 6y ago
    > You can reduce the benefit of such a hack by not having information critical enterprises all running the same binary blob. That assumes that you actually inspect the source, right?
Spooky23 6y ago
You have to segment and have monitoring tools monitored by people with a clue. But that is very expensive to do. The average SaaS or software company does nothing.
benlivengood 6y ago
> So how do we guard against this type of attack? How do we know this hasn't already happened to some of us? What is the potential fallout from this hack, it seems quite significant. Verified builds. That means deterministic builds (roughly, from a given git commit the same binaries should result no matter who compiles them. It requires compiler support and sometimes changes to the code) plus trusted build infrastructure. To verify that you haven't been compromised do a verified build from two independent roots of trust and compare the resulting binaries. Add more roots of trust to reduce the probability that all of them are compromised. Establishing a trusted root build environment is tricky because very little software has deterministic builds yet. Once they do it'll be much easier. Here's my best shot at it: Get a bunch of fresh openbsd machines. Don't network them together. Add some windows machines if you're planning to use VS. Pick 3 or more C compilers. Grab the source, verify with pgp on a few machines using a few different clients. For each one, compile it as much as possible with the others. This won't be possible in whole due to some extensions only available in a particular compiler used in its source, but is the best we can do at this point. Build all your compilers with each of these stage-2 compilers. Repeat until you have N-choose-N stage-N compilers. At this point any deterministic builds by a particular compiler (gcc, llvm, VS) should exactly match despite the compilers themselves being compiled in different orders by different compilers. This partially addresses Ken Thompson's paper "reflections on trusting trust" by requiring any persistent compiler backdoors to be mutually compatible across many different compilers otherwise it'll be detected as mismatched output from some compiler build ancestries but not others. Now you have some trusted compiler binaries. Git repository hashes can be the root of trust for remaining software. Using a few different github client implementations verify all the hashes match on the entire merkle tree. Build them with trusted compilers of your choice on multiple machines and verify the results match where possible. At this point you should have compilers, kernels, and system libraries that are most likely true to the verified source code. Make a couple build farms and keep them administratively separate. No common passwords, ssh keys, update servers, etc. Make sure builds on both farms match before trusting the binaries. The good news is that most of this can be done by the open source community; if everyone starts sharing the hashes of their git trees before builds and the hashes of the resulting binaries we could start making a global consensus of what software can currently be built deterministically and out of those which are very likely to be true translations from source code. EDIT: https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds is Debian's attempt at this.
nvader 6y ago
This reminds me of the Ken Thompson hack: https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html
twistedpair 6y ago
We're all screwed. Predicted long ago. See _Reflections On Trusting Trust_ [1] [1] https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
stelfer 6y ago
> So how do we guard against this type of attack? Looks like they compromised the editor. If so, then I imagine checking checksums for each component of the toolchain would work. Though if they compromised the filesystem or runtime then that would complicate things. But still, a hash tree or certificate of the OS and toolchain as part of CI would seem to be a good idea in 2021.
marcinzm 6y ago
>So how do we guard against this type of attack? Only allow restricted CI servers to build and deploy production code. Whitelisted list of software and security monitoring for CI servers so malicious software is harder to install. Whitelisted list of software that developer machines can run (ie: no arbitrary third party code). Restricted docker containers for all local testing. There's a reason various companies do not give developers admin access to their machines. edit: This or related concepts (SDLC, audit logs reviewed weekly, security team sign offs on system changes, etc.) are in my experience pretty standard in large enterprise security reviews for vendors that were high risk. The issue is that everyone probably lies and doesn't actually do any of these things even if they say they do. The incentives are set up to certify vendors rather than fail them.

omgbobbyg 6y ago

As a citizen, I am shocked and appalled by this backdoor. As a software engineer, I can't help but marvel at the creativity and thoughtfulness put into the exploit.

f430 6y ago
The average engineer isn't an infosec expert and love automation so they found the weakest link in the chain: CI/CD
- robbyt 6y ago
  We're not talking expert-level infosec here. They should have firewalled their CI systems. That's pretty basic stuff.

HALtheWise 6y ago

https://www.crowdstrike.com/blog/sunspot-malware-technical-analysis/ https://www.crowdstrike.com/blog/sunspot-malware-technical-a... is the key link with more technical analysis for those interested, including source code of the implant. "If the decryption of the parameters (target file path and replacement source code) is successful and if the MD5 checks pass, SUNSPOT proceeds with the replacement of the source file content. The original source file is copied with a .bk extension (e.g., InventoryManager.bk), to back up the original content. The backdoored source is written to the same filename, but with a .tmp extension (e.g., InventoryManager.tmp), before being moved using MoveFileEx to the original filename (InventoryManager.cs). After these steps, the source file backdoored with SUNBURST will then be compiled as part of the standard process."

woliveirajr 6y ago
This, combined with comments from other threads, makes me think that SolarWinds wasn't the real target. They were just means to some specific high-value ends. When low-value companies were using it, the remote control would even remove the backdoor to avoid some accidental discovery. How, and how much, were the real targets affected?
- jaywalk 6y ago
  SolarWinds was absolutely not the real target. The malware wouldn't even execute if the machine it was installed on was joined to a domain containing the string "solarwinds".

fancyfredbot 6y ago

Could and likely already has hit others

candiddevmike 6y ago

I'm surprised this hasn't caused the software industry to completely halt and rewrite/audit all third party libraries and dependencies. The entire software supply chain is highly trust-based, npm especially. Why aren't we seeing the start of a NIH dark age?

cobookman 6y ago
NPM is not trusted by major fortune enterprises. Many of the tech companies I've worked at banned using NPM in prod. And instead created their own NPM Clone with source code pinned into a private repo, which security audited. Adding a single NPM library became a total PITA, as it linked to another 100 other NPM libraries, which in-tern linked to an additional 100+ other NPM libraries. So adding a single NPM library to the private repo, meant adding 100s to 1000s of other NPM libraries. (E.g. left-pad [1]) Personally, I think this was a major reason for node.js not picking up more in the enterprise space. With Python & golang getting more traction. [1] https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/ https://qz.com/646467/how-one-programmer-broke-the-internet-...
- mandelbrotwurst 6y ago
  > With Python & golang getting more traction. Does pip not have the same issues?
- dgellow 6y ago
  Python and go have similar issues though. People do not audit and do no vendor their dependencies.
  - jniedrauer 6y ago
    At least python and go packages don't execute arbitrary code on developer's workstations at install time. If you pin dependencies to a known safe version, then you're relatively safe. With NPM, just typing `npm update` can pwn your workstation.

Veserv 6y ago

Oh good. The attackers were only in their systems since 9/4/19 before being detected on 12/12/20, so only 15 months of infiltration into SolarWinds' systems before detection. At least the payload was only deployed 2/20/20, so their customers were only completely infiltrated without detection for 8 months. Assuming the attackers could only get a 10 MB/s channel total per target even though they probably infected thousands to tens of thousands of machines per target, at ~20 million seconds that would constitute ~200 TB exfiltrated per customer or ~19 years of 1080p video. If an attacker has just one day to root around and exfiltrate they can easily get valuable information. If they are given 8 months they have already gotten everything of value for months and are just waiting around for any new data to come in. Think how inadequate your systems must be to let an attacker sit around in your systems for 8 months, it is mind-boggling how unqualified their systems are for their problems. And this is not just an indictment of SolarWinds. Just in case anybody forgets, it was the top-flight security company FireEye who discovered this breach after realizing they themselves were breached. A "best of the best" security company took 8 months before realizing that they or any of their customers had been breached. This is what "top-class" security buys you.

vasco 6y ago
Whenever I read a post as strongly worded as yours I wonder that its author must really be the best security minded engineer out there, and has never compromised on anything for a deadline or missed something because they simply didn't know it should be configured in a particular safe way. This definitely doesn't look good and there's probably many failures along the way, but jeez.
- spijdar 6y ago
  I think it's fair to say that these problems shouldn't be blamed on individuals, but systemic failure. In my imagination, someone should have seen this at some point. The system should have had safeguards to ensure that, regardless of any individual's personal failures. That said, I agree it's a little harsh, since there's no evidence that anyone else could/has done better in this incident.

throwawaybutwhy 6y ago

Oh. Brian Krebs regurgitates a corporate press release by a company that has recently hired Chris Krebs (no relation), all the while skirting around the solarwinds123 gross negligence. A well-funded PR campaign has already resulted in a NY Times smear hit piece accusing another software company of colluding with the Russians. Back to the substance, it appears the investigation keeps a close lid on the real extent of the breach. It's nasty.

dbg31415 6y ago

Yes, but... Having worked at SolarWinds they're especially susceptible to demands from sales and marketing. "Go faster, ignore tech best practices, etc." It's not unique, but their culture is not a dev-first, or security-first, culture to say the least. Many product managers answer to marketing first, and don't have earnest tech backgrounds that would let them know right from wrong past sales numbers. The culture changed significantly when they went public the first time; it went from a place where devs built good tools... to a place looking to buy products / competitor products so they could charge more for their services. Look at how long it took them to get into cloud tools -- great example of how marketing and sales missed the boat because they were only focused on things they had sold before and not focused on systemic changes to the industry -- because technologists weren't driving. Anyway, like I've worked a lot places with better security built into the culture, better tech best practices built into the culture... that's all I'm trying to say. Knowing that attacks like this are out there... and it was just a matter of time before it happened, SolarWinds did next to nothing to avoid it happening to them.

josho 6y ago
> SolarWinds did next to nothing to avoid it happening to them And how does one stop this from happening to them? Assume 'standard' security practices are in place, e.g. firewalls. Clearly the attacker was able to get their payload to be executed by a developer in SolarWinds. This payload clearly didn't trigger antivirus software, and hid itself from the developer and the build systems. How does a typical software dev. team guard against this?
- josephg 6y ago
  As others have mentioned, building on secured CI/CD infrastructure (+ ideally deterministic builds) would have helped here. But I'm not sure if thats good enough. Presumably if CI/CD was set up, some of the engineers would have had security credentials to configure any build servers anyway. And if the hackers could leverage that, they might have been able to just infect the build servers too. As annoying as it is, I can't help but think we need to bring parts of ios / android's security model to our desktop machines. Its ridiculous that a single rogue npm module can access any file on my computer, including SSH credentials and (probably) secure cookies in my browser's trust store. Our desktop security model is obsolete, and should do more to protect us from threats like this.

afrcnc 6y ago

From what we've seen until now, this company deserves everything that happened to it. Hope they go under. Ignoring security best practices for the chance of a quick buck.

josho 6y ago
What best practices were ignored? From what I'm reading here I'm terrified as I don't think the industry can protect from a targeted attack like this. The only thing we have going for us is defence in depth to make a targeted attack expensive (like this attack was), and that there are sophisticated attackers and very many software shops which means I have a low probability of being the next victim.

slt2021 6y ago

I am 99% percent sure the hackers are among the CUSTOMERS of Solar Winds. That way they were able to live-test infected SolarWinds distro in their own controlled environment and develop all possible mitigations and techniques - the sheer amount of these evading techniques suggests they were built up over time, and not instantly. Being Solar Winds customer and receiving infected updated versions every time gave them opportunity to perfect their techniques and hide for so long At least that what I would do if I were a hacker and wanted to persist and be very careful about not getting detected

nallerooth 6y ago
At least it would be safe to assume that they had access to several systems which received Orion updates. Given the attacker's dedication to the whole process, I'd say someone else's servers probably ran the tests for them.
totalZero 6y ago
Alternatively, it is not unimaginable for a foreign entity to bribe or otherwise compromise a SolarWinds employee.
- somurzakov 6y ago
  it is much easier for highly skilled hacker to get fake identity and get employed at SW than to compromise an employee - although this is longer term play
  - cutemonster 6y ago
    > get fake identity Wouldn't Google, Apple etc do background checks that discovered fake identities? Or is it not so easy with background checks? Maybe SolarWinds would be less careful though?
    - slt2021 6y ago
      nation state spies can get fake identities issued legally by their own government and will be confirmed as usual during the background check

Stierlitz 6y ago

"So how do we guard against this type of attack?" Don't allow access to your 'secret' source code from the open Internet.

m4rtink 6y ago
Or actually do that and let everyone verify and build their own binaries. This is preatty much how Linux distros work, the multiple third party distro packagers packaging upstream code make it preatty hard to sneak in mallware unnoticed and almost impossible to affect multiple distros at once. In comparison proprietary software companies are a single point of failure where customers can't access the source and have no means to verify what ghe binaries they get actually contain.

exabrial 6y ago

At one end of the spectrum I need to make sure people aren't choosing dumb passwords and are applying software updates. On the other end, centralizing control makes a very juicy target for hackers.

wolverine876 6y ago

Similar attacks are going on right now, you just won't read about it until the year 202x, if ever. What are you doing about it?

dinny 6y ago

Do we know how the SUNSPOT malware got access to the build VMs?

ThinkBeat 6y ago

The scope of this thing is staggering to me. It must have taken a significant amount of time in prep and dev and then deploy and control. A serious investment in time and resources so money. But why? In order to burn so many nice tricks they had to be after something quite valuable in one way or another. What was the motherload they were after and did they get it? Will we ever know? Or was this harvesting of the intelligence and information there were needed for the real gold? They went wide, which might have been to obscure the real target, or they needed a lot of pieces from different sources

vondur 6y ago

I wonder is something as simple as two factor authentication for those who have access to the build servers may have helped prevent something like this attack?