11 ms·
Analyzing the compromised DLL file that started the Solorigate attack
- bassman9000 6y agoThis method is part of a class, which the attackers named OrionImprovementBusinessLayer to blend in with the rest of the code Chuckled at this one.
- MarekKnapek 6y agoDid SolarWinds' CA (certificate authority) reworked their code signing certificate?
- EvanAnderson 6y agoI do work for a SolarWinds Customer. SolarWinds told us on Thursday that the certificate was going to be revoked on the 21st. Then yesterday they told us the certificate wasn't being revoked until February 2021. This says to me that the certificate itself probably wasn't compromised. The attacker must have found a place in the CI pipeline where they could insert code and get it signed automatically.
- dboreham 6y agoI'd be surprised if signing was done automatically, that would be really bad. More likely it was done manually on a package that came out their build system, without anyone stroking their beard to wonder if that system had had its compiler replaced or its cache of dependencies poisoned.
- bob1029 6y agoI am curious how this code actually made it in, based upon the following: > The fact that the compromised file is digitally signed suggests the attackers were able to access the company’s software development or distribution pipeline. Evidence suggests that as early as October 2019, these attackers have been testing their ability to insert code by adding empty classes. Unless this a compromise of the build machine, it sounds suspiciously like a lack of code review standards to me. In our organization, the only way to get a line of code into master is through a process where a 2nd developer reviews and approves via GitHub. Branch protection rules are really nice for this kind of concern. Obviously, the attacker can hit right after cloning source, but it helps to know your foundations are clean regardless.
- tsimionescu 6y agoIf we're at the level where we think it's an inside job, it doesn't seem that difficult to have 2 people on the inside "reviewing" each other's malicious commits. For what it's worth, my org also has the same policy, but it's intended to catch mistakes, not to protect against malicious actors inside the company.
- Spooky23 6y agoThe vast majority of software shops don’t even consider insider threat in any meaningful way. Imo it’s would be trivial to compromise many. Most companies have soft underbelly units like offshore maintenance engineering, tools teams and patching teams who don’t get a lot of meaningful oversight and can bypass many controls.
- saagarjha 6y agoI mean, not even that. The cost to buy a software engineer and get them hired at the place you want to attack is really not that high. Once inside it’s generally possible to get things in (“the guild server was failing so I SSHed in and fixed it”).
- mehrdadn 6y ago
- mad_vill 6y ago> To have some minimal form of obfuscation from prying eyes, the strings in the backdoor are compressed and encoded in Base64, or their hashes are used instead. ah so base64 is valid encryption after all.
- yuribro 6y agoIt says encoded, not encrypted. base64 is an encoding. It also says obfuscation, which isn't encryption either. And finally, it talks about hashes (without even claiming cryptographic hash functions), and but it's not about the base64 strings...
- breck 6y agoAnyone else find it ironic that the country that is responsible for democratizing access to scientific research (via support of SciHub), the country protecting a whistleblower against government overreach (Snowden), the country pointing out how fundamentally insecure closed source, proprietary software is (SolarWinds), is...Russia? How did we get here?
- ipython 6y agoSo you're saying that Russia's interests are entirely altrustic? That's quite a stretch. I don't think they hacked SolarWinds to "prove how insecure closed source, proprietary software is". Why don't they prove how "insecure" Kaspersky AV is, in that case? Seems strange to pick a software package that nobody's heard of, but happens to be used by thousands of juicy industrial espionage targets of their primary political enemy.
- breck 6y agoI have no idea what Russia's interests are. I'm just genuinely curious and in the dark. I've never been to the country and grew up in the West during the Cold War, so when I think of Russia I was indoctrinated to think of "the bad guys". The country is an enigma to me—very good at math but from what I read a corrupt place with rule by power and not law. So yeah, I'm just genuinely asking, how are they the ones doing these things which make the world a better place, and why isn't it us leading the charge in these 3 issues? Maybe it is just a coincidence that those 3 things align with their selfish interests, and their is no altruism involved? Or maybe there's a group in Russia that loves the ideals of the USA, and is able to actually help implement those ideals from there, because if you tried to do those 3 things from here you would be thrown in prison due to some bad laws/people here?
- efdee 6y agoThis comment, along with the original comment of the same person, is laughable at best and downright Russian astroturfing in a less favorable reading. Obviously, Russia has a policy to support whatever the US is angry about (SciHub, Snowden, ...). To paint these as altruistic is just silly - there is no chance on earth they would support Snowden if his leaks were about Russia rather than about the US. To paint the SolarWinds hacking as "showing the world proprietary systems are dangerous" is intellectually dishonest at best. You are not genuinely curious. You are not genuinely asking. You do not genuinely wonder if there is altruism involved. GTFO with your doublespeak.
- mehrdadn 6y agoAny GitHub/GitLab/etc. employees here? I think you might be able to help mitigate some of these kinds of attacks: > To have some minimal form of obfuscation from prying eyes, the strings in the backdoor are compressed and encoded in Base64, or their hashes are used instead. There needs to be a quick tool that flags strings that appear to represent binary data before a merge, maybe even decoding them when possible and providing hints of what they might represent, especially inside source-code files. These shouldn't be common in checked code. And we should figure out a way to whitelist them in the repo that's both safe and convenient (I'm not sure how). Is this a feature code-hosting sites like GitHub can add?
- ronsor 6y agoThat's just going to cause annoying false positives, and it's not that difficult to get around such detection. I'll bet the only result would be confused developers.
- mehrdadn 6y agoI can imagine it failing or succeeding depending on how intrusive the UX is. It's worth trying out I think, especially in an opt-in fashion. They can improve it gradually and people can leave it disabled if they don't like it.
- katbyte 6y agoI presume it would be opt in, would make for a handy github action tbh
- mehrdadn 6y agoNote that it shouldn't be an "action" on GitHub Actions, since those are specified in the repo itself, and can hence be removed in the same commit. It needs to be an external thing.
- ThePadawan 6y agoThe relevant code was (allegedly) not part of source control, but inserted during the build process.
- eternalny1 6y agoInteresting tidbit at the bottom ... > In an interesting turn of events, the investigation of the whole SolarWinds compromise led to the discovery of an additional malware that also affects the SolarWinds Orion product but has been determined to be likely unrelated to this compromise and used by a different threat actor.
- infogulch 6y agoThis makes me think of a common refrain when dealing with parasite infestations: If you see one, there's way more than just one. Deterministic builds cannot come soon enough. And really, builds are not enough, we need to be able to extend confidence in the execution of the programs we write much deeper than just builds.
- Shank 6y ago> Deterministic builds cannot come soon enough. This doesn't do anything for people who buy SolarWinds Orion, which is a closed-source off-the-shelf tool that gets picked up everywhere because of a combination of good sales tactics, compliance checkboxes, and ability to remove work from all involved. Going back up the chain, a technical solution probably won't solve the issues inside SolarWinds either. Systemic organizational issues lead to RCE backdoors and implants distributed on official update servers, signed with authentic keys.
- toast0 6y agoDeterministic builds can be done with closed source too. It doesn't directly help the users, but if they had setup a second build machine and noticed the build output was different, they could have addressed this sooner. Of course, if following best practices, all build machines should be equally compromised. ;p
- pnutjam 6y agoMdt hashes and signing could have avoided this. Open source stuff always verifies, vote closed source doesn't have that habit.
- smspf 6y ago>In an interesting turn of events, the investigation of the whole SolarWinds compromise led to the discovery of an additional malware that also affects the SolarWinds Orion product but has been determined to be likely unrelated to this compromise and used by a different threat actor. Either that one was used to compromise the supply chain (in which case it makes little to no sense to keep it around and risk detection), or at least 2 different groups had the chance to target sensitive US infrastructure. Funny how media coverage of this issue misses no chance of mentioning Russia and nobody else, not even possible suspects. I wonder what happens if the attackers notice each other on the compromised system. Do they get along in exfiltrating data or do they fight quietly?
- Shank 6y ago> Funny how media coverage of this issue misses no chance of mentioning Russia and nobody else, not even possible suspects. There are parts of the intelligence community that know with confidence who the true attacker is. Even if they had no idea they were being exploited, there are many ways to perform post-mortem analysis when you're, e.g., the NSA. So, someone has 100% confidence, or close to it. In terms of what the media says: typically, they report on off-the-record remarks from officials and leaks. That's just how the game is played. It's an unfortunate byproduct of everyone wanting to tell, but nobody wanting to be caught telling. The value of Reuters and AP is that they typically do enough due diligence on their own sources to make sure that they're not just spouting nonsense. "Top of the food chain" sources like them are very regularly correct, but fallible.
- roywiggins 6y agoThe secretary of state has said as much, and pointed at Russia. Sure, he could be lying, but given the president's reflexive defense of Russia, that would be a weird lie to go with. If anything, it's an admission against interest, which strongly suggests to me that this is the assessment of the relevant security agencies.
- chris_wot 6y agoTrump said it was China
- otterley 6y agoI would very much like to see prevention advice tacked on to analyses like these. It's very interesting to see how the vulnerabilities were exploited, but I think it would be extremely valuable to understand how to prevent future attacks such as this. What were the root causes of the vulnerability, and how can the community prevent similar ones from being created in the future? (Ideally with some automated tooling, too.)
- jeffbee 6y agoThe root cause? It's that people buy snake oil from vendors to check boxes on meaningless compliance tests.
- meowface 6y agoIt can affect non-snake oil from good vendors who provide useful solutions for meaningful compliance tests, too, though. Or just any popular B2B software provider. If one of the big 3 superpowers really wants to backdoor your product, then even a top-notch company might fall victim. Hopefully this increased awareness will make it harder to pull off these subtle compromises without it getting caught sooner, though.
- robocat 6y agoGoogle banned Windows machines from their workforce a decade ago (due to getting hacked by the Chinese). Has Google relaxed that ban in any way? I.e. does Google believe that Windows 10 is now secure? What other major corporations or government departments have followed Google’s example since 2010?
- jeffbee 6y agoI think "banned" is an overstatement. There were/are people obligated to use Windows because they need professional EDA tools and suchlike software only available for that platform. And although it's by no means my speciality -- I'm a distributed systems developer, not a Windows sysop -- I was told a few years ago that the general belief was that Windows had the most sound security story of all the operating systems you could put on a laptop. The reputation of Windows is a combination of history and the fact that idiots are likely to use it, so it always looks like the OS people do stupid things with. But if your company culture isn't stupid you can make it work much better.
- danjc 6y agoI’d have expected that a catch block with nothing in it would have warranted some investigation
- chris_wot 6y agoI am curious what network detection could be done to detect this sort of thing. Clearly the code needed to make outbound connections to hosts. Some thoughts: if there is an encoded sub domain, flag this as suspicious. Any code that uses a function to decode a base64 encoded string should be a red flag. Any newly created thread code should be detected and checked most carefully. Any others people can think of?
- allyant 6y agoAny decent static code analysers should be able to detect things like this (catch all’s statements, base64 encoding etc), I am surprised none seem to be used for production code.
- derwiki 6y agoIs SCA often set up to run on the fully built end result running on the production machine? I’ve generally seen them as pre-merge-to-source-control.
- WarOnPrivacy 6y ago> what network detection could be done to detect this sort of thing. Clearly the code needed to make outbound connections to hosts. Renting space that shares a netblock with a trusted host could make that more difficult.
- er4hn 6y agoNOTE: The views I'm expressing here are solely my own and say nothing about my employers views. I think that the tech industry has a severe code supply chain issue. Supply chains are a super hard problem with physical products (Raise a hand if you (a) have tried tracing the supply chain of cocoa (b) Can tell a midnight factory run of luxury clothes from a legitimate one (c) remember the supermicro controversy ) but with software we have the ability to do a much better job on solving it. I find it really disappointing that we have failed to do so. Reading through the comments here I've seen discussions on deterministic builds, code signing, and other practices. I think that they are parts of a unified whole, but all the pieces need to be there and need to be correctly done. Below I outline where I think the industry should be. A complete, secure, code supply chain should do the following: * Validate signatures on all 3rd party dependencies * Ensure that all internally written code relies on signed commits * Have builds be reproducible * Sign the output of those builds Taken together all of these form a complete supply chain that applies to both closed and open source software. There is nothing technically infeasible about implementing much of it as well - to me it feels like a culture issue. The gap between where we seem to be and where we should be seems to be: * Validate signatures on all 3rd party dependencies ** Present Day: Many vendors cannot be bothered to properly sign the outputs of their builds. Microsoft updates, openssh releases, and things like that remain the exception rather than the rule. This problem becomes even more egregious when looking at enterprise to enterprise products such as drivers which are either massive sets of source code or precompiled blobs, both of which run with lots of privileges in the context of the product they are integrated into. Even Fedora provides lots of packages from their Koji build system, the majority of which are not signed. ** Where we could be: Normalize signing these, and normalize validating the signatures prior to any use in a build environment. This is one of the easiest places in the supply chain to insert malware due to the lack of verifications. * Ensure that all internally written code relies on signed commits ** Present Day: Outside of git, most VCS systems don't even support signed commits. Within git, signed commits are not popular. I personally blame the tooling. Signing is based on PGP keys which have all sorts of known issues with use, tooling, and a general disdain due to their initial use case for email being broken. Places like Github attempt features like mandatory signing, but that falls short. Keys are still sourced from unknown places, each developer is responsible for their own key, there is no support for validating prior commits once the signing key is rotated, and using the webui totally bypasses the signing requirements (https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/about-commit-signature-verification). ** Where we could be: Let's imagine a future where git is used as a VCS. Signing keys should be centrally controlled by an authority with developers issued code signing subkeys that are rotated and can be revoked by the central authority. By having a history of all code signing keys over time, the repository can also be audited at any point in time. Even if malicious insiders directly alter the VCS, it can be flagged! I lead a project to implement such a system at my work ( https://eos.arista.com/commit-signing-with-git-at-enterprise-scale/ ) which I am posting on every discussion here to try and normalize a discussion around how to do this at other companies. * Have builds be reproducible ** Present Day: This is probably the biggest gap in having a secure supply chain. Builds today are not reproducible nor are they deterministic. The best which I know of is NixOS which is around 99% reproducible ( https://r13y.com/ ). Debian appears 95% on a specific target ( https://isdebianreproducibleyet.com/ ). Most other products are much lower than that. ** Where we could be: The first step would be deterministic builds, where building with the same inputs always results in the same outputs. Once you have a way to store what those inputs are, you can then reproduce builds later. Securing build environments becomes much easier at that point. You can build in multiple places, at multiple security levels, and check the same output comes out each time. You can even build at a much later point in time since you should have your whole set of dependencies clearly documented and saved. Validating outputs is super easy later on since you can recreate exactly what it should have been. This is also great for build systems in general since it makes dependency graphs more accurate and reduces problems with building in different environments. With the existence of VMs and containers, this is also a problem that should be super solvable. The devil is in the details here, but there should not be any reason it cannot be solved other than a lack of proper investment. * Sign the output of those builds ** Where we are today: This is one of the items that is actually the most popular, since it is so easy to do. There are lots of methods to sign any sort of data and the tooling around them is pretty straightforward. By signing this data, it closes the loop on someone downstream validating that data as an input to their own system. ** Where we could be: Keeping up the good work and going further to normalize signing build system outputs!
- tester756 6y agoWhat's the thing about adding "-gate" to everything?
- krapp 6y agoit's been a common way to refer to a scandal since Watergate[0]. Don't know if that's actually the case here though. [0]https://en.wikipedia.org/wiki/Watergate_scandal https://en.wikipedia.org/wiki/Watergate_scandal
- tester756 6y agobut why? why cannot it just be "solarwinds hack"
- mhh__ 6y agoIf it fits, I prefer gate, but in this case the suffix is quite jarring so I agree
- loufe 6y agoIt's a tad conspiratorial, but I suspect there was a concerted effort by media to muddy down the impacts of future watergate-level scandals by highlighting a lot of minor "gates". Already I feel tempted to roll my eyes each time I see another HN post about X-gate, seeing many "weaker" stories using the same nomenclature certainly steals power away from the stories that hit hard.
- jjk166 6y agoI wouldn't refer to getting hacked as a scandal. Unless it comes out that there was some sort of coverup, it seems innapropriate to refer to it as a -gate.
- WarOnPrivacy 6y ago(not an answer but) It's less annoying than needlessly adding "Cyber" to criminal activity.
- jijji 6y agoI wonder how much different this would be had it been a linux application running under apparmor or in a container environment... One would expect from a security perspective that all of these remotely distributed applications would be running under some kind of chroot jail or container to prevent the kind of exposure that is obviously happening here. I think Microsoft is a little complicit in their lack of security in their OS platform allowing these types of issues to proliferate repeatedly year after year with no real changes happening in the ways that applications are locked down.
- dboreham 6y agoIsn't the whole point to this that the targeted software is supposed to run at high privileges and is also supposed to phone home? So it's the ideal vector for an exfil attack. The only way to avoid it would be to do like Hillary and run your own email server with none of this cool stuff installed.
- WarOnPrivacy 6y agoThe handling of this whole event is in stark contrast to just a few years ago - when details on malicious activity were a closely guarded secret and useful threat data was safely locked away from anyone who might protect the public with it.
- richardjennings 6y ago"Finally, the backdoor composes a JSON document into which it adds the unique user ID described earlier, a session ID, and a set of other non-relevant data fields. It then sends this JSON document to the C2 server." Is there any further explanation of how this was achieved? One might expect as "par for the course" that all external connections be blocked aside from explicitly designated ranges. I would expect that an attempt at external comms would set off alarms.
- nhoughto 6y agoI was wondering the same, if the compromise is of the Orion product which presumably isn't just sitting there with open access to the internet? Like this doesn't seem to be a very broad exploit that could start on a machine with outside world access (like a Windows exploit itself) and then pivot into more sensitive areas.
- Corrado 6y agoIt's my understanding that multiple, unexplained NXDOMAIN responses IS what exposed the compromised systems. Why it didn't happen earlier (or immediately) is a good question.
- metta2uall 6y agoI wonder if there will be a "post-mortem" regarding why Microsoft didn't detect this attack earlier..? (similar question for other security vendors)
- peter_d_sherman 6y agoObservation: "avsvmcloud.com" -- seems to be the one constant around which a whole bunch of other things, which are variables revolve... (oh sure, "appsync-api" also appears to be a constant -- but it exists at a far less important place in the URL). "avsvmcloud.com" is far, far more important -- because ALL of the communications go there... Now, it may be that "avsvmcloud.com" is a legitimate ISP, hosting provider or what-have-you... But, if I were an investigator on this case, I know I'd want to track each and every place that these requests flow through whoever owns the "avsvmcloud.com" network... I'd start with the idea that because a subdomain is being used, that the first thing that happens is that subdomains must be resolved by a DNS subdomain servers... so where exactly on whoever owns the "avsvmcloud.com" network, does that happen? I'd even go so far as to audit, completely dissasemble, the DNS software that is running on those servers... Give it to as many security researchers as possible... What does it do? Where does it point to? What's on the other end of those IP addresses that it resolves to? Are there any anomalies in that IP address resolution? Specifically, when/where and how do they manifest? Are there any patterns there? Who owns the machines on the other side of those IP addresses? Etc., etc. In fact... What would happen if someone were to run a machine learning algorithm on say a, let's be polite and call it a "challenged" DNS resolver? Would it find some DNS resolution anomalies? In fact, if I were an investigator, I'd go as far as to audit the whole chain of DNS resolvers / the DNS resolution process THOROUGHLY...