7 ms·
It never fails to amaze me how most incident mitigations seem completely oblivious to such security side effects. "We have no reason to believe that the expose
by p-e-w 4y ago
It never fails to amaze me how most incident mitigations seem completely oblivious to such security side effects.
"We have no reason to believe that the exposed key was abused, but out of an abundance of caution, we are going to expose 50 million users to a potential MITM attack unless they are extremely careful."
Not a single word in the post about whether this impact was even considered when making the decision to update the key. Just swap the key, and fuck the consequences. Same with the mass password resets after a compromise that some services have done in the past years. Each of those is any phishing operation's dream come true.
- BHSPitMonkey 4y agoHow is there an alternative here?
- ansible 4y agoThere isn't an alternative, really. The private key has been exposed, and presumably it is unknown if or how far it has spread. The SSH keys must be changed, and the sooner the better. All that can be done is to notify people after the change has occurred.
- p-e-w 4y ago> and presumably it is unknown if or how far it has spread Why would that be unknown? GitHub has HTTP and SSH access logs, right?
- tgsovlerkhgsel 4y agoSince the post doesn't mention anything like "we reviewed logs and confirm the key was not accessed", it is very likely that they either don't have logs that are reliable enough to rule it out (e.g. they may be sampled or otherwise incomplete), or that the key was accessed.
- renonce 4y agoKeeping a complete log of all GET requests to random files in a public repository in a reliable way would be insane.
- otoolep 4y agoNo, it wouldn’t be - assuming by “insane” you mean “silly to do”. I build systems at Google that do exactly that. Whether it’s worth the cost is a decision each company makes. Also, you don’t need to keep the log forever. Max of a few weeks retention would be common.
- pbhjpbhj 4y agoPresumably, keeping 'last remotely accessed' and 'last remotely modified' for every file (or other stats that are a digest of the logs) is sane for pretty much any system too. Having a handle on how much space one is dedicating to files that are never viewed and or never updated seems like something web companies that have public file access would all want?
- tgsovlerkhgsel 4y agoWhat guarantees do these systems provide? Are 100% of requests where data was served guaranteed to either end up in the log or at least create a detectable "logs may have been lost here" event? Or does it log all the requests all the time as long as everything goes well, but if the wrong server crashes at the wrong time, a couple hundred requests may get lost because in the end who cares?
- est31 4y agoIt's not just GET requests. Someone could have cloned/refreshed the repo using ssh. The repo might have been indexed by github's internal search daemon which might not use the public HTTP API but uses internal access ways however those might look like. You might have purged the database of that daemon but what about backups of it? What about people who have subscribed to public events happening in the github.com/github org via the API? You'd have to have logging set up for all of these services and it would have to work over your entire CDN... and what if a CDN node crashed before it was able to submit the log entry to the log collector? You'll never know.
- blitzar 4y agoOne H4X0R gave it to four friends, who in turn gave it to between 9 and 14 friends, who in turn gave it to between one and 6 friends. If train A leaves New York going 60 miles per hour with 167 people on board and train B leaves Chicago one hour later with 361 people on board going 85 miles per hour, how many people now have the key? The answer is 31337.
- p-e-w 4y agoSame as with any other decision: Do a cost/benefit analysis of whether the security risk created by rotating the key is actually outweighed by the security risk of doing nothing, taking into account logs that should tell you whether the exposed key was indeed accessed by unauthorized parties. To be 100% clear: Both courses of action come with associated security risks. The problem is not choosing one course of action over the other, the problem is thinking you can just skip the cost/benefit analysis because the answer is somehow 'obvious'. It's not obvious at all.
- johngalt_ 4y agoNo, you cannot keep using an exposed key. You must replace it. There is no cost/benefit analysis needed in this situation.
- p-e-w 4y agoWrong. A CBA is always needed. If the potential damage from MITM attacks made possible by rotating the key is greater than the potential damage from a rogue key multiplied by the likelihood that someone actually accessed the key, then it is wrong to rotate the key. It's that simple. The only way a CBA would be unnecessary is if rotating the key didn't have any security risks. But it does.
- eyelidlessness 4y agoHere I’ll do the CBA: - if they have evidence that the key was exposed to one person, even with zero usage of the key, failing to rotate the key is tantamount to knowingly accepting widespread compromise at a potential attacker’s whim. At GitHub’s scale, that’s untenable. - rotating the key is the only correct reaction to that - they should have better communications in place to help users mitigate MITM - there really isn’t an option, because they’re critical infrastructure; I’m glad they know that and acted accordingly - on principle this speculation makes sense, but understanding the threat makes it moot - you hopefully know that, and it’s good to insist on thoughtful security practices but it’s important to also understand the actual risk
- 4y ago
- rakoo 4y agoThat's why you need certificates and not just a key pair. Certificates make key rotation easier, and you want key rotation to be easy. I guess the proper way forward is a small utility that gets the latest signature through http+tls, and replaces the line in your known_hosts file, all in the background. Looking long term, maybe we need to get rid of all the security stuff in ssh and just pipe the rest of its functionalities inside a TLS pipe. Let the os do its certificate management, reuse security bricks that are way more studied, ...
- oleganza 4y agoCertificates just add more keys to worry about. The beauty of SSH is that it does not add hugely trusted parties in the name of convenience, while the UX of TOFU (trust on first use) is pretty decent. The real solution to break out of these UX/security tradeoffs is to put domain names on a blockchain: then you can simply rotate the key in your DNS record, while the blockchain model is such that you need to compromise many parties, instead of "one out of many parties", as with CAs. Tracking Bitcoin chain for DNS updates is lightweight enough that it can be built into OS alongside other modern components such as secure enclave, TCP/IP stack and WiFi/BT/5G radios.
- hnfong 4y ago> The beauty of SSH is that it does not add hugely trusted parties in the name of convenience Even with a certificate authority model, you don't have to trust any CAs if you don't want to. Not having the option to do so is more of a problem.
- datadeft 4y agoWe should use a separate system that could reliably verify which certs belong to which entity. Blockchain is a perfect solution to this. I wonder why it is not considered yet.
- jtsiskin 4y agohttps://certificate.transparency.dev https://certificate.transparency.dev it is
- ithkuil 4y agoThe alternative would be to use certificate authorities (ssh has CA support) which allow to effectively have private keys at different levels and allow you to keep the root private key in a physical vault and use it very rarely to issue other private keys
- terom 4y agoAnd then don't forget to setup key revocation as well, and make sure that an attacker in a position to MITM the connection cannot cause the revocation checks to fail-open. I hope you don't need that SSH connection to fix your broken CRL endpoint!
- datadeft 4y agoThis would just offload the problem to a separate entity. CAs can be (and have been) compromised.
- ithkuil 4y agoSure, but isn't it more likely that a key that has to be shared by who knows how many ssh load balancer machines at GitHub and can't be easily rotated because it's pinned by millions of users, isn't it more likely that that private key gets eventually compromised or thought to be at risk at being compromised? We need to compare the relative risks within the same context, namely within a company like GitHub So it's not relevant to bring up failures of other CAs
- michaelt 4y agoJump into a time machine, go back to the creation of SSH, and adopt SSL-style trusted third-party certificate authorities. Somehow get it adopted anyway, even though loads of people use SSH on internal networks where host-based authentication is difficult; SSH is how many headless machines are bootstrapped; and that you've got to do it 19 years before Lets Encrypt. Jump into a lesser time machine, go back to when Github were creating their SSH key, and put it into a hardware security module. Somehow share that hardware-backed security key to loads of servers over a network, without letting hackers do the same thing. Somehow get an HSM that isn't a closed-source black box from a huge defence contractor riddled with foreign spies. Somehow avoid vendor lock-in or the HSM becoming obsolete. Somehow do this when you're a scrappy startup with barely any free time.
- nibbleshifter 4y agoSsh certificate authorities are a thing that exists. We also have a way to put SSH host key fingerprints in DNS records already.
- Arch-TK 4y agoYes but the option to do verify host keys using ("VerifyHostKeyDNS") is not enabled by default.
- roblabla 4y agoDNS can trivially be mitm'd. DNS-stored fingerprints are strictly less secure than TOFU.
- tialaramex 4y agoIf you use DNSSEC (cue inevitable rant from Thomas) this just works. If you have DoH (and why wouldn't you?) and your trusted resolver uses DNSSEC (which popular ones do), you get the same benefits. https://en.wikipedia.org/wiki/SSHFP_record https://en.wikipedia.org/wiki/SSHFP_record
- datadeft 4y ago
- pid-1 4y agoClone repos using oauth2 with two factor enabled - both GitHub and GitLab support that though their CLIs.
- marcosdumay 4y agoWell, at least SSH could allow for signing a new key with the old one. So they could say it's signed, and people would know to accept only a different prompt. There is DNS verification, but people have been trained all their lives to accept insecure DNS information (and set their systems accordingly), and I really doubt the SSH client checks the DNSSEC data.
- dheera 4y agoHow would one stage a MITM attack without knowing the private key corresponding to the old key?
- edp 4y agoThe fact that users have to delete the old Github key from their systems and accept a new one is what could lead to a MITM attack. If your system doesn't know the public key of an SSH server, when you connect the first time, the SSH client will display a warning and ask you if you accept the server key. An attacker could be between you and Github and if you accept without checking it's the correct key, you would be toast.
- pbhjpbhj 4y agoWould it be more secure to access a https secured server to get the keyfile then?
- tialaramex 4y agoYes, GitHub's announcement provides the correct new public RSA key, and it also provides instructions for a curl invocation which does all the work if you don't trust yourself to copy-paste text or don't understand how.
- dheera 4y agoOnly if the https server cert wasn't compromised at the same time as the ssh key. For all we know, this entire announcement of "we have a new key" could be staged.
- stouset 4y agoThey don’t need it. Millions of users are going to blindly trust the “new” GitHub public SSH key they see the next time they connect without checking to see if it matches the published signature.
- p-e-w 4y agoBy pretending to be the host that the user is trying to connect to. You can then present the client with a key you generated yourself. Of course, SSH will warn the user that the fingerprint has changed, but they'll just think "Ah yes, GitHub changed their keys so it's probably fine." This is why updating the key creates a potential MITM risk, unless people actually bother to verify that the fingerprint is correct.
- faeriechangling 4y agoWhile their reaction is more likely to cause a security breach, consider the psychology. If the key was breached and Github just didn't know it, then a breach happened, then only Github would be to blame. If Github rotates its key, and somebody suffers a MITM attack, the blame is more diffuse. Why didn't they verify the key out of band?
- justeleblanc 4y agoI'm always amazed at this kind of posts. Did these 50 million users (surely none of them use git+https!) check the host key the first time they connected to github? Did you?
- arianvanp 4y agoIt doesn't matter because it didn't change! That's the beauty of TOFU.
- msm_ 4y agoThe point is, what if it you were MITMed from the beginning?
- emn13 4y agoSure, but the difference is that it's now both a plausible moment to go MITM (because they got that key), and furthermore the hypothetical attacker now has good reason to believe users won't be scared by a host-key-change warning, and the hypothetical attacker would know this opportunity exists for a large set of users simultaneously. If some malicious network operator were to try and exploit users, now would be a good moment - they'd likely catch many more people in the time it takes to be discovered than on an average day. The MITM-at-the-start risk is of course real, but I think this new everyone -restarts-simultaneously risk is qualitatively different enough to be worth at least considering.
- lxgr 4y agoMuch more concerningly, there is an activated-by-default OpenSSH extension (`UpdateHostKeys`) that allows the server to install new host keys into `.ssh/known_hosts` after every successful server authentication.
- tlb 4y agoThe bad guys would also have to have MITMed it every time I connected for the last 15 years, or I would have seen authentication failures when it connected to the real thing. MITMing someone once isn't that hard, but doing it consistently is.
- tomp 4y agoDon't trust corporate PR. They're obviously lying when they say "out of an abundance of caution". The private key was exposed in a public GitHub repo, it could literally be anywhere. So MITM for some of 50m users is strictly better than MITM for all of 50m users.
- the_other 4y agoIt could be, but also GH might be logging inbound requests long enough to see whether the file was requested.
- deleted 4y ago[deleted]
- theteapot 4y ago> The private key was exposed in a public GitHub repo. How do you know this? Github runs scanner for private keys in public and private repos and notifies owner (I did it once so I know ... ;)). So some Github engineer likely would have received such an email if what you say is true. Hilarious.
- 3np 4y agoFrom the OP: > This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository.
- Veen 4y agoIt says exactly that in the article: > This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository Hilarious.
- theteapot 4y agoI didn't read it fully before commenting. I'm sorry.
- 4y ago
- mihaaly 4y agoIs there a benefit (and practicality) in recording encrypted traffic by an adverse intermediary waiting for keys being exposed sometime? Like now?
- KAMSPioneer 4y agoNo, the risk of losing an SSH host key is less this (because of forward secrecy), rather impersonation of the server.