10 ms·

Please before replacing your local fingerprint with the new one, double check it is the expected value. This is an opportune time for man-in-the-middle attacker

• devenvdev 4y ago
  There should be a StackOverflow Streisand effect, at first, I peeked at the end of your comment to copy-paste the ssh-keygen string "solution".
• p-e-w 4y ago
  It never fails to amaze me how most incident mitigations seem completely oblivious to such security side effects. "We have no reason to believe that the exposed key was abused, but out of an abundance of caution, we are going to expose 50 million users to a potential MITM attack unless they are extremely careful." Not a single word in the post about whether this impact was even considered when making the decision to update the key. Just swap the key, and fuck the consequences. Same with the mass password resets after a compromise that some services have done in the past years. Each of those is any phishing operation's dream come true.
  • BHSPitMonkey 4y ago
    How is there an alternative here?
    • ansible 4y ago
      There isn't an alternative, really. The private key has been exposed, and presumably it is unknown if or how far it has spread. The SSH keys must be changed, and the sooner the better. All that can be done is to notify people after the change has occurred.
      • p-e-w 4y ago
        > and presumably it is unknown if or how far it has spread Why would that be unknown? GitHub has HTTP and SSH access logs, right?
        • tgsovlerkhgsel 4y ago
          Since the post doesn't mention anything like "we reviewed logs and confirm the key was not accessed", it is very likely that they either don't have logs that are reliable enough to rule it out (e.g. they may be sampled or otherwise incomplete), or that the key was accessed.
          • renonce 4y ago
            Keeping a complete log of all GET requests to random files in a public repository in a reliable way would be insane.
            • otoolep 4y ago
              No, it wouldn’t be - assuming by “insane” you mean “silly to do”. I build systems at Google that do exactly that. Whether it’s worth the cost is a decision each company makes. Also, you don’t need to keep the log forever. Max of a few weeks retention would be common.
              • pbhjpbhj 4y ago
                Presumably, keeping 'last remotely accessed' and 'last remotely modified' for every file (or other stats that are a digest of the logs) is sane for pretty much any system too. Having a handle on how much space one is dedicating to files that are never viewed and or never updated seems like something web companies that have public file access would all want?
              • tgsovlerkhgsel 4y ago
                What guarantees do these systems provide? Are 100% of requests where data was served guaranteed to either end up in the log or at least create a detectable "logs may have been lost here" event? Or does it log all the requests all the time as long as everything goes well, but if the wrong server crashes at the wrong time, a couple hundred requests may get lost because in the end who cares?
    • p-e-w 4y ago
      Same as with any other decision: Do a cost/benefit analysis of whether the security risk created by rotating the key is actually outweighed by the security risk of doing nothing, taking into account logs that should tell you whether the exposed key was indeed accessed by unauthorized parties. To be 100% clear: Both courses of action come with associated security risks. The problem is not choosing one course of action over the other, the problem is thinking you can just skip the cost/benefit analysis because the answer is somehow 'obvious'. It's not obvious at all.
      • johngalt_ 4y ago
        No, you cannot keep using an exposed key. You must replace it. There is no cost/benefit analysis needed in this situation.
        • p-e-w 4y ago
          Wrong. A CBA is always needed. If the potential damage from MITM attacks made possible by rotating the key is greater than the potential damage from a rogue key multiplied by the likelihood that someone actually accessed the key, then it is wrong to rotate the key. It's that simple. The only way a CBA would be unnecessary is if rotating the key didn't have any security risks. But it does.
          • eyelidlessness 4y ago
            Here I’ll do the CBA: - if they have evidence that the key was exposed to one person, even with zero usage of the key, failing to rotate the key is tantamount to knowingly accepting widespread compromise at a potential attacker’s whim. At GitHub’s scale, that’s untenable. - rotating the key is the only correct reaction to that - they should have better communications in place to help users mitigate MITM - there really isn’t an option, because they’re critical infrastructure; I’m glad they know that and acted accordingly - on principle this speculation makes sense, but understanding the threat makes it moot - you hopefully know that, and it’s good to insist on thoughtful security practices but it’s important to also understand the actual risk
          • 4y ago
    • rakoo 4y ago
      That's why you need certificates and not just a key pair. Certificates make key rotation easier, and you want key rotation to be easy. I guess the proper way forward is a small utility that gets the latest signature through http+tls, and replaces the line in your known_hosts file, all in the background. Looking long term, maybe we need to get rid of all the security stuff in ssh and just pipe the rest of its functionalities inside a TLS pipe. Let the os do its certificate management, reuse security bricks that are way more studied, ...
      • oleganza 4y ago
        Certificates just add more keys to worry about. The beauty of SSH is that it does not add hugely trusted parties in the name of convenience, while the UX of TOFU (trust on first use) is pretty decent. The real solution to break out of these UX/security tradeoffs is to put domain names on a blockchain: then you can simply rotate the key in your DNS record, while the blockchain model is such that you need to compromise many parties, instead of "one out of many parties", as with CAs. Tracking Bitcoin chain for DNS updates is lightweight enough that it can be built into OS alongside other modern components such as secure enclave, TCP/IP stack and WiFi/BT/5G radios.
        • hnfong 4y ago
          > The beauty of SSH is that it does not add hugely trusted parties in the name of convenience Even with a certificate authority model, you don't have to trust any CAs if you don't want to. Not having the option to do so is more of a problem.
        • datadeft 4y ago
          We should use a separate system that could reliably verify which certs belong to which entity. Blockchain is a perfect solution to this. I wonder why it is not considered yet.
          • jtsiskin 4y ago
            https://certificate.transparency.dev https://certificate.transparency.dev it is
    • ithkuil 4y ago
      The alternative would be to use certificate authorities (ssh has CA support) which allow to effectively have private keys at different levels and allow you to keep the root private key in a physical vault and use it very rarely to issue other private keys
      • terom 4y ago
        And then don't forget to setup key revocation as well, and make sure that an attacker in a position to MITM the connection cannot cause the revocation checks to fail-open. I hope you don't need that SSH connection to fix your broken CRL endpoint!
      • datadeft 4y ago
        This would just offload the problem to a separate entity. CAs can be (and have been) compromised.
        • ithkuil 4y ago
          Sure, but isn't it more likely that a key that has to be shared by who knows how many ssh load balancer machines at GitHub and can't be easily rotated because it's pinned by millions of users, isn't it more likely that that private key gets eventually compromised or thought to be at risk at being compromised? We need to compare the relative risks within the same context, namely within a company like GitHub So it's not relevant to bring up failures of other CAs
    • michaelt 4y ago
      Jump into a time machine, go back to the creation of SSH, and adopt SSL-style trusted third-party certificate authorities. Somehow get it adopted anyway, even though loads of people use SSH on internal networks where host-based authentication is difficult; SSH is how many headless machines are bootstrapped; and that you've got to do it 19 years before Lets Encrypt. Jump into a lesser time machine, go back to when Github were creating their SSH key, and put it into a hardware security module. Somehow share that hardware-backed security key to loads of servers over a network, without letting hackers do the same thing. Somehow get an HSM that isn't a closed-source black box from a huge defence contractor riddled with foreign spies. Somehow avoid vendor lock-in or the HSM becoming obsolete. Somehow do this when you're a scrappy startup with barely any free time.
      • nibbleshifter 4y ago
        Ssh certificate authorities are a thing that exists. We also have a way to put SSH host key fingerprints in DNS records already.
        • Arch-TK 4y ago
          Yes but the option to do verify host keys using ("VerifyHostKeyDNS") is not enabled by default.
        • roblabla 4y ago
          DNS can trivially be mitm'd. DNS-stored fingerprints are strictly less secure than TOFU.
          • tialaramex 4y ago
            If you use DNSSEC (cue inevitable rant from Thomas) this just works. If you have DoH (and why wouldn't you?) and your trusted resolver uses DNSSEC (which popular ones do), you get the same benefits. https://en.wikipedia.org/wiki/SSHFP_record https://en.wikipedia.org/wiki/SSHFP_record
        • datadeft 4y ago
    • pid-1 4y ago
      Clone repos using oauth2 with two factor enabled - both GitHub and GitLab support that though their CLIs.
    • marcosdumay 4y ago
      Well, at least SSH could allow for signing a new key with the old one. So they could say it's signed, and people would know to accept only a different prompt. There is DNS verification, but people have been trained all their lives to accept insecure DNS information (and set their systems accordingly), and I really doubt the SSH client checks the DNSSEC data.
  • dheera 4y ago
    How would one stage a MITM attack without knowing the private key corresponding to the old key?
    • edp 4y ago
      The fact that users have to delete the old Github key from their systems and accept a new one is what could lead to a MITM attack. If your system doesn't know the public key of an SSH server, when you connect the first time, the SSH client will display a warning and ask you if you accept the server key. An attacker could be between you and Github and if you accept without checking it's the correct key, you would be toast.
      • pbhjpbhj 4y ago
        Would it be more secure to access a https secured server to get the keyfile then?
        • tialaramex 4y ago
          Yes, GitHub's announcement provides the correct new public RSA key, and it also provides instructions for a curl invocation which does all the work if you don't trust yourself to copy-paste text or don't understand how.
        • dheera 4y ago
          Only if the https server cert wasn't compromised at the same time as the ssh key. For all we know, this entire announcement of "we have a new key" could be staged.
    • stouset 4y ago
      They don’t need it. Millions of users are going to blindly trust the “new” GitHub public SSH key they see the next time they connect without checking to see if it matches the published signature.
    • p-e-w 4y ago
      By pretending to be the host that the user is trying to connect to. You can then present the client with a key you generated yourself. Of course, SSH will warn the user that the fingerprint has changed, but they'll just think "Ah yes, GitHub changed their keys so it's probably fine." This is why updating the key creates a potential MITM risk, unless people actually bother to verify that the fingerprint is correct.
  • faeriechangling 4y ago
    While their reaction is more likely to cause a security breach, consider the psychology. If the key was breached and Github just didn't know it, then a breach happened, then only Github would be to blame. If Github rotates its key, and somebody suffers a MITM attack, the blame is more diffuse. Why didn't they verify the key out of band?
  • justeleblanc 4y ago
    I'm always amazed at this kind of posts. Did these 50 million users (surely none of them use git+https!) check the host key the first time they connected to github? Did you?
    • arianvanp 4y ago
      It doesn't matter because it didn't change! That's the beauty of TOFU.
      • msm_ 4y ago
        The point is, what if it you were MITMed from the beginning?
        • emn13 4y ago
          Sure, but the difference is that it's now both a plausible moment to go MITM (because they got that key), and furthermore the hypothetical attacker now has good reason to believe users won't be scared by a host-key-change warning, and the hypothetical attacker would know this opportunity exists for a large set of users simultaneously. If some malicious network operator were to try and exploit users, now would be a good moment - they'd likely catch many more people in the time it takes to be discovered than on an average day. The MITM-at-the-start risk is of course real, but I think this new everyone -restarts-simultaneously risk is qualitatively different enough to be worth at least considering.
          • lxgr 4y ago
            Much more concerningly, there is an activated-by-default OpenSSH extension (`UpdateHostKeys`) that allows the server to install new host keys into `.ssh/known_hosts` after every successful server authentication.
        • tlb 4y ago
          The bad guys would also have to have MITMed it every time I connected for the last 15 years, or I would have seen authentication failures when it connected to the real thing. MITMing someone once isn't that hard, but doing it consistently is.
  • tomp 4y ago
    Don't trust corporate PR. They're obviously lying when they say "out of an abundance of caution". The private key was exposed in a public GitHub repo, it could literally be anywhere. So MITM for some of 50m users is strictly better than MITM for all of 50m users.
    • the_other 4y ago
      It could be, but also GH might be logging inbound requests long enough to see whether the file was requested.
    • deleted 4y ago
      [deleted]
    • theteapot 4y ago
      > The private key was exposed in a public GitHub repo. How do you know this? Github runs scanner for private keys in public and private repos and notifies owner (I did it once so I know ... ;)). So some Github engineer likely would have received such an email if what you say is true. Hilarious.
      • 3np 4y ago
        From the OP: > This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository.
      • Veen 4y ago
        It says exactly that in the article: > This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository Hilarious.
        • theteapot 4y ago
          I didn't read it fully before commenting. I'm sorry.
          • 4y ago
  • mihaaly 4y ago
    Is there a benefit (and practicality) in recording encrypted traffic by an adverse intermediary waiting for keys being exposed sometime? Like now?
    • KAMSPioneer 4y ago
      No, the risk of losing an SSH host key is less this (because of forward secrecy), rather impersonation of the server.
• tgsovlerkhgsel 4y ago
  They provide convenient commands to import the correct keys. It would probably be better to only include the block that contains both the -R and the update command, but at least they do provide them.
• brabel 4y ago
  I've updated the key in known_hosts, then was able to connect successfully. What do I have to do to ensure I connected to the right server?? I thought just making sure the correct RSA key was in known_hosts would be enough?
  • snorremd 4y ago
    That is enough, given that you've fetched or compared the key from a trusted GitHub.com server.
  • nirimda 4y ago
    It depends on how you found out what the new key value is. By the sounds of your description, you're fine. But in principle there's more than one way people could proceed from here. If you read the blog post on a verified domain and saw the new key and updated manually, or you deleted the known key and verified the key fingerprint when it warned you about an unknown key, you should be good to go. Here, you trust the people who issue TLS certificates and you trust github to be in control of their domain name, so you can be reasonably confident that the key they advertised on their website is the correct key. If your internet connection was compromised, you would have got an error message when you connected to https://github.blog https://github.blog (because they wouldn't have a certificate from a trusted certificate issuer) or when you connected to the git server (because they wouldn't have the key you just trusted). If you saw the blog post and then removed the old key and told ssh to save the new key it's receiving without checking it matches the value on the webpage, you might have a problem. The connection to github's ssl could have been compromised, and if you just accepted whatever it told you, you have no trusted intermediate to verify that the key is trustworthy. All you know is that each time you connect to github's server hereafter, you're either connecting to a server with the same key (no error), or you're connecting to one that doesn't have the same key (error message). But whether you can trust that key? You don't know that. You just know it's the same key. But even if you did the latter, all is not lost. You can look at the known_hosts file (on Linux and MacOS it's ~/.ssh/known_hosts) and check the fingerprint. If it's what they advertise, then you're good to go. If it's different, you should fix it and find people who can help you deal with a security incident. The reason people are raising a flag is that today, lots of people will be rotating their key. That means if you're looking to target someone, today is the day to do it. Even if 90% of people do it the proper way, by manually verifying the key, that still means there's going to be a lot of people who could be victimised today.
• defanor 4y ago
  Here are the expected fingerprints (since they don't publish those via SSHFP RRs): https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints https://docs.github.com/en/authentication/keeping-your-accou... SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s (RSA) SHA256:br9IjFspm1vxR3iA35FWE+4VTyz1hYVLIE2t1/CeyWQ (DSA - deprecated) SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM (ECDSA) SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU (Ed25519)
  • cwillu 4y ago
    Note the MITM here :) We humans really aren't cut out for this, are we.
    • darthrupert 4y ago
      What MITM? What are you talking about?
      • fulafel 4y ago
        The poster of the fingerprints is in the middle, you are not getting them from GH if you use them instead of going to the linked page.
      • vxNsr 4y ago
        This is literally a man in the middle between you and GitHub.
      • nicky0 4y ago
        If someone wanted to trick HN users into trusting a phoney key, one way to do that would be to post the phoney fingerprint on HN claiming it to be the real one.
        • iso1631 4y ago
          And within a few seconds someone will have called this out in a reply
• yosito 4y ago
  > double check it is the expected value Not all of us are familiar enough with the SSH protocol to understand how to "double check the expected value"? Where can I determine what the expected value should be?
  • Gravyness 4y ago
    Run "ssh -T git@github.com" command. It should error like this: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s. Please contact your system administrator. Note that the SHA256 present there matches perfectly the one github send. If you don't remember the very first time you connected to github you also had to accept the key. The warning above shows up because the server is saved as a different RSA, for the SSH client it seems that someone setup a server for github but has a different key, which could mean someone is trying to trick you into connecting into the wrong server. This could also mean that github changed their RSA key which is why they published this article.
    • pbhjpbhj 4y ago
      The fingerprint is a hash of the key, so in theory -- say with a quantum computer -- I could create a key that's different and provides a hash-collision. Is that right? It would just take many ages of the universe, at present, to calculate a collision, right?
      • 8organicbits 4y ago
        There's a narrow window for that attack. The fingerprint is only used on the first connection, for manual verification. Any later connections would check the ~/.ssh/known_hosts which has the full public key. If you somehow can MITM an SSH connection on the first connection, you can probably use any key. Most people don't check the fingerprint. But you are correct, computing an SSH key with a collisionwis expected to take an infeasible amount of time/energy with current understanding of crypto and available computers.
• deleted 4y ago
  [deleted]
• datadeft 4y ago
  Certificate pinning check built in when? We should have a blockchain for certificates btw. That would be such an amazing solution to this problem. You could advertise ahead of the time that you are changing certificates and we could verify that it was in fact you.
• rpigab 4y ago
  Double-check with what source? The one mentionned in docs.github.com? I assume it's safe because the SSL cert for docs.github.com is probably not compromised, so it's giving us the right key, and compromising docs.github.com would be extra effort and is unlikely to happen. However, I wonder what kind of steps an MITM attack would have to perform, I assume one of the easiest would be compromising my local DNS server, since regular DNS does not offer a high level of security, then github.com resolves to the attacker's IP and the attack works. Do you have examples of such attacks that don't involve a virus already active on the end user's PC? Maybe if someone owns an IP previously owned by Github that is still somehow advertised as being Github by some DNS lagging behing?
  • 8organicbits 4y ago
    This is always a concern with SSH as it uses trust on first use. The first time you connect it shows you the fingerprint for out of band verification. That manual step is on you to perform, but most people skip it. Future visits check against the saved fingerprint. The best practice is to verify the fingerprint out of band using a secure channel. In this case, that's HTTPS and docs.github.com. If (hypothetically) docs.github.com was also compromised, then you don't have a secure channel. https://en.m.wikipedia.org/wiki/Man-in-the-middle_attack https://en.m.wikipedia.org/wiki/Man-in-the-middle_attack has some MITM examples.
• bityard 4y ago
  SSH host certs would make this a non-issue, and I've often wondered why GitHub doesn't use them.
• aidenn0 4y ago
  Or just use the posted command to fetch the fingerprint over ssh and automatically add it to your known hosts?