5 ms·
> So how do we guard against this type of attack? You can't ever prevent it, but you can raise the attack cost/complexity and make detection much more likely.
by bostik 6y ago
> So how do we guard against this type of attack?
You can't ever prevent it, but you can raise the attack cost/complexity and make detection much more likely.
Go for immutable infra and transient CI/CD systems. Provision the build nodes on demand, from trusted images, and wipe them out completely after just a few hours. The attacker will have to re-infect the nodes as they come online, and risk leaving more network traces. Anything they deploy on the systems goes away when the node goes away.
The attack against SolarWinds worked so well because the CI system was persistent and it was upgraded over time. For a reliable and secure environment, the correct amount of host configuration is zero: if you need to modify anything at all after a system has been launched, that's a bad smell. Don't upgrade. Provision a completely new system with the more recent versions.
This kind of architecture requires the attacker to compromise the CI image build and/or storage instead. (Or the upstream, directly.) It's an extra step for adversary, and a simpler point of control to the defender.
Recon. Compromise. Persist. Exfiltrate. -- As a defender you want to make every step as expensive and brittle as possible. You can't prevent a compromise, but you can make it less useful, and you can arrange things so that it must happen often and leave enough traces to increase the probability of getting caught.
- joe_the_user 6y agoAnother thing that might have made this hard - if Solar Winds were distributed as source code and each client built it themselves, with their own options (though the old back-doored c-compiler "thought experiment" may not be as much of a thought experiment anymore). Moreover, achieving the hack was likely costly given the effort and the benefit of the hack appeared once the Solar Winds binary was distributed. You can reduce the benefit of such a hack by not having information critical enterprises all running the same binary blob.
- SahAssar 6y ago> You can reduce the benefit of such a hack by not having information critical enterprises all running the same binary blob. That assumes that you actually inspect the source, right?
- joe_the_user 6y agoIf Solarwinds was distributed as source to hundreds of companies, maybe many would not bother diffing the source from the previous version but it seems plausible that a few would look at these at the least, especially given you are talking corporations who follow deployment procedures. The build process itself could spit out the diffs at the end, for example.
- Pokepokalypse 6y agowell, you still have a 'tragedy of the commons' situation if say 10,000 users are all counting on at least 1 user to inspect the source - but with no commercial incentive to do so, nobody does. How does a user of open source ever know or measure exactly how well the source code has been scrutinized?
- joe_the_user 6y agoI'm getting this "open source doesn't guaranteed there's no problem" claim. And I'd agree open source doesn't guarantee there's no problem. But it seems clear that routes which have allowed malware to be pushed automatically are fundamental, guarantee problems and malware in source code is a potential problem with an obvious answer.
- SahAssar 6y agoBesides all of these steps I think it's important to consider every convenience, every dependency, every package you add as another attack vector. This is even more relevant considering the product SolarWinds sold.
- twistedpair 6y agoPart of me wonders if they'd have been better with GitHub cloud CI/CD in Actions, with immutable build infra (e.g. Ubuntu base images). But, with the wild npm dependency community, where a small app can have 5K transitive dependencies, I feel we're going to be even more susceptible to these attacks going forward.
- f430 6y agoOne of the big reasons why some shops refuse to use Javascript on server side because of its unknown attack surface that is NPM. It's impossible to keep track of all of these libraries that can be written by anyone. Someone could hijack a github account and push malicious code to NPM.