4 ms·
Wow, your assessment of the impact here (or even possible impact) is way way way overblown. In reality, the vast majority of users don't pay attention to SSH h
by kelnos 4y ago
Wow, your assessment of the impact here (or even possible impact) is way way way overblown.
In reality, the vast majority of users don't pay attention to SSH host keys at all.
Even if an attacker got hold of this private host key, they'd have to be able to MitM the connections of their target.
Next, they have to decide what they want to do.
If they want to send malicious code to someone doing a 'git pull', they'd have to craft the payload specifically for the repo being pulled from. Not impossible, but difficult.
If they want to "steal" source code from someone doing 'git push' (perhaps to a private repo on GitHub), that's a bit easier, as they can just tell the client "I have no objects yet", and then the client will send the entire repo.
And, again, they'd have to have the ability to MitM some git user's connections. Regardless, there is no way that they could change any code on github.com; this key would not give them any access to GH's services that they don't already have.
So I think your anger here is a bit over the top and unnecessary.
I agree that it's pretty bad that some GH employee even had the ability to get hold of this private key (sure, probably should be in an HSM, but I'm not particularly surprised it's not) in order to accidentally leak it, but... shit happens. They admitted the mistake and rotated the key, even though it's likely that there was zero impact from all this.
- lrvick 4y agoEnd users do not pay attention but their clients pin the key after first use. Also everyone is using gitops these days and almost no one is using dns-over-tls. Imagine you control the right router on a company wifi, or any home wifi a production engineer works from and suddenly you can cause them to clone the wrong git submodule, the wrong go package, or the wrong terraform config. If you knew a CI/CD system blindly clones and deploys git repos to prod without signature checks, and that prod is a top 10 crypto exchange with 1b of liquidity in hot wallets, then suddenly a BGP attack to redirect DNS is a good investment. Myetherwallet got taken over for 15 minutes with a BGP so this is not hypothetical. Should that be the case? Of course not. But the reality is I find this in almost all of the security audits I do for fintech companies. Blind trust in Github host keys is industry standard all the way to prod.
- kelnos 4y agoSure I can imagine that. And in doing so, I imagine this attack is pretty unlikely. I mean, think of the confluence of things that have to line up for this to work for someone. Many stars have to align in order for someone to successfully exploit this leak. Is it impossible? No, of course not, and so GH was right to rotate the key here, even if their server request logs suggested that no one had accessed it. If people actually have this sort of attack in their threat model, there are ways to protect against it. Signing commits and verifying them. Pinning to a particular git SHA1 hash. Etc. If people are not doing these things, then it's possible they've made the decision not to worry about this sort of attack. Sure, you can disagree with, but I think you'd probably be in the minority. That's ok, though; you can certainly protect the things you're responsible for in stronger ways.
- lrvick 4y agoI have seen attacks along the lines I just outlined and well beyond in my industry many times. A wildcard TLS cert or an SSH host key in plaintext is a loaded weapon laying around and it will be used on a high value target. Sad fact is most of the companies that hold billions of dollars of customer assets do development exactly the same way a gaming company might. Those that even attempt security are unicorns. They bank everything on things like DNS, BGP, TLS certs, and ssh host keys working. This is like the medical industry before they learned hand washing was a thing. I teach every one of my clients how to do things like commit signing, but for every one I help improve there are 100 more on track to be hacked any day now. I can totally forgive a startup that cannot afford senior security engineers for a mistake like this, but Microsoft can afford that, or at least consultants, and yet they can not even enable optional signing for NPM, properly enforce git commit signing, or even protect an ssh host key trusted by millions in a TEE or HSM.
- lxgr 4y agoYes, but they will have to un-pin the (now compromised) key if they want to continue using Github. Any compromise would have to isolate them from the "real" Github hosts from today onwards, i.e. plausibly MITM them continuously, or they would just switch to the rotated key to be able to continue working. At least in OpenSSH, this means replacing the compromised trusted key, as there can only be one per host (or even IP in the default configuration). This is still very bad, but much less catastrophic than e.g. a world in which `.ssh/known_hosts` allows multiple entries, in which case you'd really have a sustained compromise of most clients.
- throwawaaarrgh 4y ago> Even if an attacker got hold of this private host key, they'd have to be able to MitM the connections This is not hard. If it were hard, we wouldn't need encrypted connections. If I were a nation state, it would be trivial to position an attack at the edge of a network peering point for a given service. How do I know? Our own government did it for 10+ years. Cyber criminals often have the same tactics and skills and can find significant monetary reasons to assume heightened risk in order to pull off a compromise. Random black hats enjoy the challenge of compromising high value targets and can find numerous ways to creatively infiltrate networks to perform additional attacks. Even without gaining access to a network, virtually anyone on the internet can simply push a bad BGP config and capture traffic for an arbitrary target. Weird routing routinely happens that nobody can definitely say isn't such an attack.