17 ms·
We updated our RSA SSH host key
- jupp0r 4y ago> out of an abundance of caution > This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository. No further questions for this witness Your Honor
- nightpool 4y agoAny idea which public repository it got exposed in? What happened, and how did they notice?
- yababa_y 4y agomy guess is the copilot X AI is threatening escalation and was forced into demonstrating its capabilities. ;)
- yetanotherjosh 4y agoPlease before replacing your local fingerprint with the new one, double check it is the expected value. This is an opportune time for man-in-the-middle attackers to strike, knowing everyone has to replace their stored signatures, and that some will be lazy about it with a blind "ssh-keygen -R github.com" command.
- devenvdev 4y agoThere should be a StackOverflow Streisand effect, at first, I peeked at the end of your comment to copy-paste the ssh-keygen string "solution".
- p-e-w 4y agoIt never fails to amaze me how most incident mitigations seem completely oblivious to such security side effects. "We have no reason to believe that the exposed key was abused, but out of an abundance of caution, we are going to expose 50 million users to a potential MITM attack unless they are extremely careful." Not a single word in the post about whether this impact was even considered when making the decision to update the key. Just swap the key, and fuck the consequences. Same with the mass password resets after a compromise that some services have done in the past years. Each of those is any phishing operation's dream come true.
- BHSPitMonkey 4y agoHow is there an alternative here?
- ansible 4y agoThere isn't an alternative, really. The private key has been exposed, and presumably it is unknown if or how far it has spread. The SSH keys must be changed, and the sooner the better. All that can be done is to notify people after the change has occurred.
- p-e-w 4y ago> and presumably it is unknown if or how far it has spread Why would that be unknown? GitHub has HTTP and SSH access logs, right?
- tgsovlerkhgsel 4y agoSince the post doesn't mention anything like "we reviewed logs and confirm the key was not accessed", it is very likely that they either don't have logs that are reliable enough to rule it out (e.g. they may be sampled or otherwise incomplete), or that the key was accessed.
- renonce 4y agoKeeping a complete log of all GET requests to random files in a public repository in a reliable way would be insane.
- otoolep 4y agoNo, it wouldn’t be - assuming by “insane” you mean “silly to do”. I build systems at Google that do exactly that. Whether it’s worth the cost is a decision each company makes. Also, you don’t need to keep the log forever. Max of a few weeks retention would be common.
- tgsovlerkhgsel 4y agoThey provide convenient commands to import the correct keys. It would probably be better to only include the block that contains both the -R and the update command, but at least they do provide them.
- brabel 4y agoI've updated the key in known_hosts, then was able to connect successfully. What do I have to do to ensure I connected to the right server?? I thought just making sure the correct RSA key was in known_hosts would be enough?
- snorremd 4y agoThat is enough, given that you've fetched or compared the key from a trusted GitHub.com server.
- nirimda 4y agoIt depends on how you found out what the new key value is. By the sounds of your description, you're fine. But in principle there's more than one way people could proceed from here. If you read the blog post on a verified domain and saw the new key and updated manually, or you deleted the known key and verified the key fingerprint when it warned you about an unknown key, you should be good to go. Here, you trust the people who issue TLS certificates and you trust github to be in control of their domain name, so you can be reasonably confident that the key they advertised on their website is the correct key. If your internet connection was compromised, you would have got an error message when you connected to https://github.blog https://github.blog (because they wouldn't have a certificate from a trusted certificate issuer) or when you connected to the git server (because they wouldn't have the key you just trusted). If you saw the blog post and then removed the old key and told ssh to save the new key it's receiving without checking it matches the value on the webpage, you might have a problem. The connection to github's ssl could have been compromised, and if you just accepted whatever it told you, you have no trusted intermediate to verify that the key is trustworthy. All you know is that each time you connect to github's server hereafter, you're either connecting to a server with the same key (no error), or you're connecting to one that doesn't have the same key (error message). But whether you can trust that key? You don't know that. You just know it's the same key. But even if you did the latter, all is not lost. You can look at the known_hosts file (on Linux and MacOS it's ~/.ssh/known_hosts) and check the fingerprint. If it's what they advertise, then you're good to go. If it's different, you should fix it and find people who can help you deal with a security incident. The reason people are raising a flag is that today, lots of people will be rotating their key. That means if you're looking to target someone, today is the day to do it. Even if 90% of people do it the proper way, by manually verifying the key, that still means there's going to be a lot of people who could be victimised today.
- defanor 4y agoHere are the expected fingerprints (since they don't publish those via SSHFP RRs): https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints https://docs.github.com/en/authentication/keeping-your-accou... SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s (RSA) SHA256:br9IjFspm1vxR3iA35FWE+4VTyz1hYVLIE2t1/CeyWQ (DSA - deprecated) SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM (ECDSA) SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU (Ed25519)
- cwillu 4y agoNote the MITM here :) We humans really aren't cut out for this, are we.
- darthrupert 4y agoWhat MITM? What are you talking about?
- fulafel 4y agoThe poster of the fingerprints is in the middle, you are not getting them from GH if you use them instead of going to the linked page.
- vxNsr 4y agoThis is literally a man in the middle between you and GitHub.
- nicky0 4y agoIf someone wanted to trick HN users into trusting a phoney key, one way to do that would be to post the phoney fingerprint on HN claiming it to be the real one.
- iso1631 4y agoAnd within a few seconds someone will have called this out in a reply
- yosito 4y ago> double check it is the expected value Not all of us are familiar enough with the SSH protocol to understand how to "double check the expected value"? Where can I determine what the expected value should be?
- Gravyness 4y agoRun "ssh -T git@github.com" command. It should error like this: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s. Please contact your system administrator. Note that the SHA256 present there matches perfectly the one github send. If you don't remember the very first time you connected to github you also had to accept the key. The warning above shows up because the server is saved as a different RSA, for the SSH client it seems that someone setup a server for github but has a different key, which could mean someone is trying to trick you into connecting into the wrong server. This could also mean that github changed their RSA key which is why they published this article.
- pbhjpbhj 4y agoThe fingerprint is a hash of the key, so in theory -- say with a quantum computer -- I could create a key that's different and provides a hash-collision. Is that right? It would just take many ages of the universe, at present, to calculate a collision, right?
- 8organicbits 4y agoThere's a narrow window for that attack. The fingerprint is only used on the first connection, for manual verification. Any later connections would check the ~/.ssh/known_hosts which has the full public key. If you somehow can MITM an SSH connection on the first connection, you can probably use any key. Most people don't check the fingerprint. But you are correct, computing an SSH key with a collisionwis expected to take an infeasible amount of time/energy with current understanding of crypto and available computers.
- deleted 4y ago[deleted]
- datadeft 4y agoCertificate pinning check built in when? We should have a blockchain for certificates btw. That would be such an amazing solution to this problem. You could advertise ahead of the time that you are changing certificates and we could verify that it was in fact you.
- rpigab 4y agoDouble-check with what source? The one mentionned in docs.github.com? I assume it's safe because the SSL cert for docs.github.com is probably not compromised, so it's giving us the right key, and compromising docs.github.com would be extra effort and is unlikely to happen. However, I wonder what kind of steps an MITM attack would have to perform, I assume one of the easiest would be compromising my local DNS server, since regular DNS does not offer a high level of security, then github.com resolves to the attacker's IP and the attack works. Do you have examples of such attacks that don't involve a virus already active on the end user's PC? Maybe if someone owns an IP previously owned by Github that is still somehow advertised as being Github by some DNS lagging behing?
- 8organicbits 4y agoThis is always a concern with SSH as it uses trust on first use. The first time you connect it shows you the fingerprint for out of band verification. That manual step is on you to perform, but most people skip it. Future visits check against the saved fingerprint. The best practice is to verify the fingerprint out of band using a secure channel. In this case, that's HTTPS and docs.github.com. If (hypothetically) docs.github.com was also compromised, then you don't have a secure channel. https://en.m.wikipedia.org/wiki/Man-in-the-middle_attack https://en.m.wikipedia.org/wiki/Man-in-the-middle_attack has some MITM examples.
- bityard 4y agoSSH host certs would make this a non-issue, and I've often wondered why GitHub doesn't use them.
- aidenn0 4y agoOr just use the posted command to fetch the fingerprint over ssh and automatically add it to your known hosts?
- coderintherye 4y agoIf you are seeing "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!" in respect to Github.com then this is likely why.
- encryptluks2 4y agoGood that they changed it but this is certain to break a lot of automation pipelines. I'd rather be safe than sorry though.
- rurban 4y agoI was just going to submit this thread, thinking how in the ** could the github host key being changed. Leading to their blog post explaining it. Their exposed their private RSK key! ssh-keygen -f ~/.ssh/known_hosts -R github.com Now it's a ED25519 with +DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU (they don't list this on their blogpost, just their renewed RSA key)
- yetanotherjosh 4y agoThat key didn't change and is listed here https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints https://docs.github.com/en/authentication/keeping-your-accou...
- pquerna 4y agoWould it of been possible for Github to use Host-key rotation instead of hard breaking it? https://lwn.net/Articles/637156/ https://lwn.net/Articles/637156/ I'm honestly not familiar with anyone actually using host-key rotation?
- dlgeek 4y agoIf an attacker has access to the private key, they could use the Host-key rotation feature to migrate you to an attacker-controlled key instead, as the old key is trusted. So, GitHub needs everyone to forcibly untrust the old (exposed) key.
- pquerna 4y agoYeah, but... shouldn't Github of rotated their keys over the last decade? I mean it seems like its clearly a key that wasn't in an HSM.. and over the lifetime, hundreds? Thousands of Github employees could of accessed it?
- p1mrx 4y agoCongrats, you just used "would of", "should of", and "could of" in a single thread.
- kelnos 4y agoThe problem with rotating this particular private key is that it's incredibly disruptive. Everyone who uses GH will see a big scary message from ssh saying the host key changed and something malicious might be going on. A majority of those people probably won't have seen a blog post announcing the change beforehand. Anyone who's baked the host key in the known_hosts file that gets shipped on their CI systems would start to see jobs failing, and have to manually fix it up with the new host key. These things are just annoying enough that I think it's perfectly understandable that GH doesn't want to regularly rotate this private key.
- computerfriend 4y ago
- archgoon 4y agoOnce took a security course where the prof accidentally opened his rsa key file while streaming.
- benatkin 4y agoIn this case I think "security" belongs in quotes.
- archgoon 4y ago[dead]
- xaphod 4y agoI saw once a support person from $COMPANY_I_WORK_IN ask a client to send their private key in email, so they could troubleshoot something. I pressed "Reply all" and called it out, and was yelled at later.
- lisper 4y agoThis is exactly what an attacker impersonating github would say.
- yababa_y 4y agoand yet my response is identical… bizarre!
- quickthrower2 4y agoAt the end of the day human power structures and trust are heavily important. We are trusting their site didn't get hacked, and if it did that we would find out about it. I mean if you can't trust anything at all, the modern internet is unusable.
- cavisne 4y agoIs it weird that this wasn’t in a TPM somewhere? I don’t know how host keys work, are they rotated based on some root cert regularly or is it just the same keypair since the creation of GitHub?
- iforgotpassword 4y agoYes, with ssh you don't use CAs, or certificates at all; just raw asymmetric keys. It does support certificates with all their bells and whistles though, just nobody does really...
- tgsovlerkhgsel 4y agoNot really weird, no. The host key needs to be used every time someone establishes a SSH connection to Github. HSMs can't handle that volume of requests. Given that the key is extremely long lived, this has unfortunate implications: If any of these servers is compromised, or decides to go randomly spewing memory content because of a bitflip, or screws up the nonce on a DSA/ECDSA operation, the key can be compromised. This is hard to exploit if you're a random person, but for a global adversary that collects internet traffic at scale (e.g. NSA), it's feasible and I would be surprised if they weren't exploiting such issues. They were collecting HTTPS handshakes for a reason.
- rocqua 4y agoSsh keys are much simpler than the PKI cert infrastructure. This has advantages for development, as well as bug resistance. But it means ssh expects every server behind the same hostname to use the same SSH key. You can't get two (well functioning) TPMs to have the same key. They come with their own, un-extractable* and unchangeable keys built in. *TPMs claim this, it is probably not impossible to extract keys just incredibly difficult and requiring specialist knowledge.
- execveat 4y agoThis is factually wrong. All major HSM vendors offer ways to backup and replicate keys, it's usually done using key-wrapping keys. Here's an example: https://www.ibm.com/docs/en/datapower-gateway/2018.4?topic=module-cloning-hsm-key-wrapping-keys https://www.ibm.com/docs/en/datapower-gateway/2018.4?topic=m... All cloud vendors offer the same functionality, if you think about it, so it's not an obscure feature.
- raggi 4y agoAre they confident that the private key was not seen by anyone, or did it actually leak? If it leaked, this may be more of an issue than how it appears to be presented.
- renewiltord 4y agoIf you ever push anything to a public Github repository it is immediately mirrored by a host of bots. If it was there for any amount of time, it's there. That said, if I'm honest I think almost no one checks the SSH host key. I would bet that for 9 out of 10 engineers / devops / security people, they will ignore the SSH Host Key and might even set StrictHostKeyChecking=no. I know I know. All you have to do is to just MITM someone's DNS or IP traffic or whatever and it's all over. Well, I will guarantee you that millions of bitcoin are sitting there waiting for you to take it. It's all yours. Go take it. You could be a millionaire tomorrow. And if you do it right, they'll let you keep some of it if you send most of it back.
- raggi 4y ago> If you ever push anything to a public Github repository it is immediately mirrored by a host of bots. I'd love to get a copy of the key, so I can setup a demo to encourage education. > That said, if I'm honest I think almost no one checks the SSH host key. I would bet that for 9 out of 10 engineers / devops / security people, they will ignore the SSH Host Key and might even set StrictHostKeyChecking=no. I believe you, and I agree this is common. I have been carrying the github host keys in my dot files for more than a decade. I do this somewhat ergonomically by having a Host config that adjusts the known_hosts path to permanent_known_hosts for things I care about. Setting StrictHostKeyChecking=no is negligence, despite being common there really isn't two ways about that, it's a trivial vector with only a very moderate need for additional information to turn this into an RCE in most cases. There are big problems with ssh's host keys, I understand and agree, but there are also big problems with passwords and we've made a dent in getting people to use password managers and 2FA. We should not wave this away just because it's a little harder, otherwise we're just relegating ssh to the same status as gpg.
- herewulf 4y ago> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository. Seriously? How that happened is deeply concerning. And why weren't the other keys exposed?
- a2800276 4y agoSomeone at GH was using Copilot DevOps and prompted "How would you go about compromising the integrity of all open source software?", obviously.
- sneak 4y agoMost people do not clone using SSH.
- computerfriend 4y agoThe Git HTTPS authentication UX is quite bad, so I don't understand how this can be true.
- GauntletWizard 4y agoMost clones happen without auth on public repos. Most development work, however, uses SSH
- comprev 4y agoWe use GitHub Enterprise at work and only HTTPS is permitted for authentication. The "insteadOf" git config is added to workstations and runners to convert attempted SSH connections over to HTTPS.
- computerfriend 4y agoWhy is SSH not permitted?
- paxys 4y agoBig oof from an ops standpoint. How is a key like this ever accessible to an operator in unencrypted text form? Funny enough, given that every person (and every bit of tooling) I know blindly approves SSH public key verification anyways, they will likely not even notice this switch.
- iforgotpassword 4y ago> blindly approves SSH public key verification anyways, they will likely not even notice this. Maybe on initial connect, but who TF ignores that "key has changed danger danger, high voltage" warning? You at least look around and ask a colleague?
- __turbobrew__ 4y agoCan’t an attacker now man in the middle anyone who hasn’t connected to github since the host key was rolled?
- paxys 4y agoUnless they can also hijack the github.com domain, no.
- aaomidi 4y agoThat’s not that hard tbh. So many attacks.
- __turbobrew__ 4y agoBut if you are doing git operations using SSH instead of HTTPS then you aren’t checking domain certs?
- paxys 4y agoIt's not about the certs. To execute a man-in-the-middle attack the attacker has to literally put themselves in the middle of the route your packet takes to get to github's servers and intercept it.
- __turbobrew__ 4y agoSure, there are many ways an attacker can do that. Not trusting your IP transit is kind of the whole reason for encryption in the first place. 1. Various DNS hijacking and cache poisoning attacks 2. three letter agencies in meet me rooms 3. Exploited/hacked routers 4. Wifi hot spots
- deleted 4y ago[deleted]
- nextaccountic 4y agoYes, and that's how malware ends up in npm and other package managers
- dskloet 4y ago> out of an abundance of caution, we replaced our RSA SSH host [...] > GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository What the... That's not "an abundance of caution". That's the only possible course of action.
- irjustin 4y ago> That's the only possible course of action. Only _reasonable_ course of action. Possible is to do nothing =)
- awill 4y agoYou're absolutely right. It's absurd to frame it this way. Do they expect people to think "Wow, Github leaked a key, but even without knowing if anyone snagged it, they're still replacing it. Wow, they go above and beyond." It's so ridiculous.
- Xenoamorphous 4y agoNot playing devil’s advocate but guess they at least have some confidence that no one checked out/pulled the repo while the key was there? After all it’s them hosting and serving the requests for that (and every other) repo.
- diggan 4y agoThere is a literal stream of all public data on GitHub. I don't think they can 100% know if it was accessed or not.
- colonwqbang 4y ago"We have no reason to believe" => We don't actually know
- execveat 4y agoCharitable explanation is that they rotated they key without waiting for an analysis.
- dajonker 4y agoMaybe their new copilot made this rookie mistake
- speedgoose 4y agoIs it a rookie mistake though? It happens to everyone in my experience.
- dajonker 4y agoLeaking something like a private key? You do that once, then you learn. If you don't learn, you remain a rookie. Or just reckless.
- retrocryptid 4y agoSo... uh... why was their private key in a form that could be stored in a Git repo? In the old days, we used to put private keys in hardware co-processors that would never reveal an un-encrypted private key. I guess we don't do that anymore.
- 0x0 4y agoImagine the amount of ssh connections that github must handle every second, I don't think it's so easy to hold the private key in a HSM while maintaining the necessary performance and availability.
- StopHammoTime 4y agoYeah but surely there’s a happy medium ground between HSM and storing it on some old mates desktop (which would have to be what’s going on here). It still shouldn’t have been easily accessible to anyone except the instances running the SSH service.
- parhamn 4y agoLoading it from HSM to memory/keychain is probably fine too. It's certainly odd it found its way to a repo and makes you wonder how that could have even possibly happened. And what that indicates about their security practices in general. Github is host to a large percent of US tech IP. Pretty concerning if you extrapolate.
- 0x0 4y agoThe whole point of a HSM is to never reveal the private key, but instead to have the HSM perform the necessary crypto operation for each request. This wouldn't scale to the amount of ssh traffic I would imagine github sees. I think it's completely reasonable that they have the ssh private key in some sort of configuration management repository, because they need to be able to deploy that key to all their public facing ssh servers. You would hope they would have more than 1 ssh server instance world wide for availability and resiliency.
- paxys 4y agoConveniently missing from the announcement: - When exactly was the key leaked? Yesterday? A month ago? - How long was it visible for? And no, "briefly" doesn't cut it. - Did any internet traffic reach the page while it was exposed? We know you log everything, so it is a yes or no answer. If any of these answers were pretty, I imagine they would have explicitly included them in the post.
- CGamesPlay 4y agoIn fairness, it's better that they rotate the key immediately before even looking at the logs. But we can demand better answers from them from this point.
- paxys 4y agoDid they rotate it immediately? The only reference to a specific time is: > This week, we discovered... So they found out about it sometime between Monday and Thursday, and rotated it Thursday evening.
- remus 4y agoThis seems a reasonable timeframe to me given the scale of the impact. Security issues are obviously very important, but equally I would be worried if they started acting impulsively immediately after discovering the issue, because there would not have been time to understand the problem and consider the options. If you jump on a solution without giving it a bit of thought first it's too easy to exacerbate the situation.
- retrocryptid 4y agoOh. I guess this is why the ops guys at work prefer GitLab. (And honestly, GitLab has gotten MUCH better over the last couple years.)
- parhamn 4y agoWell this is certainly going to break quit a few CI/builds/systems that didn't account for this.
- stn_za 4y agoIMHO, It should break the CI systems. If you don't pre-bake the known-hosts, then you'd allow each new ephemeral run to use whatever github tells you it's key is. It did briefly break ours, as we pre-bake the known_hosts file into our CI image for convenience and security. Convenience due to CI not having TTY's, so various tools would get stuck on prompt Y when connecting to github for the first time. (Which is every run, if you are ephemeral CI) And security, as now everything broke due to github's key changing, which is the desired outcome actually. We bumped the known_host key entry, merged and all is well again...
- computerfriend 4y agoYou could query GitHub's meta API for the host keys, which is trusting GitHub's HTTPS certificate instead.
- execveat 4y agoYou shouldn't start with a blank state, instead you should be querying https://api.github.com/meta https://api.github.com/meta . But there are so many repos on Github itself which hardcode the host keys in Github Actions, etc.
- EMIRELADERO 4y agoWhat were the implications of this? Could anyone with that key have cloned any GitHub repo, private or not?
- progval 4y agoThis is the host key, so allows impersonating the server to clients, not impersonating clients to the server.
- 8organicbits 4y agoImpersonating the server allows code theft. > Could anyone with that key have cloned any GitHub repo, private or not? If you try to push to GitHub but someone is MITMing you with the leaked key, they can say "I'm an empty repo" and the client will helpfully push all the commits to the attacker. Until everyone updates their .ssh/known_hosts file, a MITM attack can still steal code in this way.
- lrvick 4y agoGithub had one RSA ssh host key, the most widely supported key format. It was trusted to clone code into the infrastructure of hundreds of thousands of organizations. People pin it everywhere. With this key someone could infect the infrastructure of fintech companies and steal billions of dollars. I know this well because I run a security consulting company focusing mostly on that industry. Mainly this is possible because almost no companies check for git commit signing, and Github does not enforce it anyway, and I digress. This key held enough power over value that some might have used violence to acquire it. With that context of course they chose to place this key in a hardware security module controlled with an m-of-n quorum of engineers to ensure no single human can steal it, expose it, or leak it. Right? ... right? Nope. They admit they just stuck it in a git repo in plain text where any engineer could have copied it to a compromised workstation, or been bribed for it, for who knows how many years. Worse, it was not even some separate employee only intranet git repo, but their own regular public production infra and someone had the power to accidentally make it public. I have no words for this level of gross negligence. Maybe, just maybe, centralizing all trust and security for most of the worlds software development to a proprietary for-profit company with an abysmal security reputation was not the best plan. I will just leave this here: https://sfconservancy.org/blog/2022/jun/30/give-up-github-launch/ https://sfconservancy.org/blog/2022/jun/30/give-up-github-la...
- CGamesPlay 4y ago> With that context of course they chose to place this key in a hardware security module controlled with an m-of-n quorum of engineers to ensure no single human can steal it, expose it, or leak it. Right? ... right? This is unfortunately not how SSH works. It needs to be unlocked for every incoming connection. You raise valid hypotheticals about the security of the service... but fixing it involves removing SSH host key verification from Github; better OpsSec would not fully resolve this issue.
- p-e-w 4y agoHardware security modules can perform key operations without allowing anyone to access the key data. Key material being (accidentally or deliberately) leaked has been a solved problem for a long, long time.
- CGamesPlay 4y agoReading the comments here, I can see several things GitHub could have done to improve the security around their host key. But pragmatically speaking, any change involves changing the host key, which is an extremely disruptive operation. Now that they've bit the bullet and forced everyone to rotate, are they going to use this time to actually generate a new, secured key that never touches persistent memory unencrypted?
- lrvick 4y agoOf course not. This is Microsoft we are talking about. If they did something that expensive for security they would be bragging about it from their blog to earn any possible trust they could at a time when they could really use it. If you are reading this Microsoft, make me look silly. Please. I will even give you the enclave designs for free.
- protoman3000 4y agoHow does the process to write this PR-bullshit go exactly? Do the PR people have technical knowledge or does some tech person write “I accidentally screwed up and put the key public so we have to replace” and the PR glosses over it and rewrites “Out of an abundance of caution…”?
- mjg59 4y agoIn an ideal world this could be avoided by using SSH certificates - an assertion by a trusted party that the key presented by an SSH server is legitimate. The signer of those certificates could be in an HSM, ensuring that the private keys can never be leaked, and the certificates could be sufficiently short-lived that there'd be no need to deal with revocation. Unfortunately there's no way to make this easy. To associate a CA key with a specific domain you need to add something like: @cert-authority github.com (key) to known_hosts, and this (as far as I know) can't currently be managed in the same way that keys are generally added to known_hosts on first use when you ssh to something. So instead we're left with every github.com SSH frontend using the same private keys, and those can't be in an HSM because it simply wouldn't scale to the required load. Which means the private key exists in a form where this kind of thing can occur.
- londons_explore 4y ago> every github.com SSH frontend using the same private keys HTTPS has the same problem. Compromise any frontend, and you can MITM traffic to any other front-end till the expiry of the certificate (usually 90 days). I would really like to see some kind of almost realtime delegation - so that for example, every second a new sub certificate could be generated and signed by a HSM, valid for only 2 seconds.
- mjg59 4y agoClock skew unfortunately makes that scale a little impractical, but yeah it would be great to have mechanisms to delegate endpoint certificate issuance to much shorter timescales
- 8organicbits 4y agoLots of bad things happen if your front-end is compromised, that's different and it's a high bar. They can persist access with a backdoor. They can exfil historical data, password hashes. They can corrupt and modify data. With HTTPS certs you usually have at most 90 days of impact when a key is leaked (less if you revoke and software is checking CRL). GitHub used the same RSA key for over a decade, they may have continued using this key for quite some time more had they not noticed the leak this week.
- rasengan 4y agoI literally freaked out and came to Hacker News looking for an answer. Bullseye. Here's the list of keys if anyone needs: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints https://docs.github.com/en/authentication/keeping-your-accou... > Host key for github.com has changed and you have requested strict checking. > Host key verification failed.
- panda_2134 4y agoAt the same time, most newbie programmers just remove the old key and 'git push' again without checking the authenticity of the new key. IMHO those users will never benefit from host keys.
- janto 4y agoIf this happened to a smaller company they would be dead. Edit: where can I get good Mercurial hosting? F this.
- chrismorgan 4y agoGiven their secrets scanning functionality, I’m mildly curious if they discovered the leak by a notification from that system.
- buildbot 4y agoI don’t understand how this is possible - do they not pre-scan commits for secrets? Our Azure devops repos reject any push with anything that looks like a credential.
- sltkr 4y agoThe key was probably encoded in some way that made it hard to detect, for example, as a VM disk snapshot or a .tar.gz archive of a home directory. Something like that must have happened anyway, since it's highly unlikely a private key is just lying around as a plain text file on an engineers workstation to be accidentally included in a push.
- buildbot 4y agoThat would make more sense, though cred scanning should at least attempt to unzip/untar compressed files IMO.
- fomine3 4y agoThey must put a banner on every page on Github to notify this.
- fareesh 4y agoyikes
- gassius 4y agoActually I am not seeing the fingerprint they announce in the blogpost When I tried to pull from my repo I got the warning message, right I removed the old keys with ssh-keygen -R github.com Then, trying with `ssh -T git@github.comp` I see this The authenticity of host 'github.com (140.82.121.3)' can't be established. ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM. So, first thing of course, is that the fingerprint does not match the one in the blogpost which is SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s Second problem, is that ip 140.82.121.3 seems to be reported as HIGH RISK[0] So basically, how should I proceed? I am not security expert but I would say I am not an illiterate on this, and I have no idea. I guess the majority of users would you accept the new key, but is this the right move? I would need to do it if I want to do some work, that is for sure EDIT: Formatting [0]https://www.ipqualityscore.com/free-ip-lookup-proxy-vpn-test/lookup/140.82.121.3 https://www.ipqualityscore.com/free-ip-lookup-proxy-vpn-test...
- DangitBobby 4y agoTry adding all the keys shown in the docs here to your known_hosts. 1. https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints https://docs.github.com/en/authentication/keeping-your-accou...
- nneonneo 4y agoYou’re looking at the fingerprint of the ECDSA key; only the RSA key was replaced (and only the new RSA key’s fingerprint is in the blogpost). Check https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints https://docs.github.com/en/authentication/keeping-your-accou... for the full list: the key you’re seeing is listed, so you should be fine.
- gassius 4y agoThank you. It makes sense
- deleted 4y ago[deleted]
- groestl 4y agoNot directly related, but SSH certificates are one of the most underutilized features of SSH.
- DangitBobby 4y agoHa. About 30 minutes ago I was trying to figure out if I was being MITMd and decided to manually update known_hosts to list all public keys published in GitHub's docs. At least I know what happened now.
- INTPenis 4y ago>This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository. Let's remove the words briefly exposed and public. Why is a private key in a Git repository? These days I don't see any reason for this.
- contradictioned 4y agoOf course, there is no reason. But humans make errors, and humans deal with those errors.
- robbat2 4y agoStart protecting yourself from a potential MITM better, mark the key as revoked. Hopefully distributions & OpenSSH upstream can start shipping this by default. (sorry, the comments are mangling this, clean version at https://gist.github.com/robbat2/b456f09b7799f4dafe24115095b8b29c https://gist.github.com/robbat2/b456f09b7799f4dafe24115095b8...) ``` # You might need to insert this in a slightly different place cat >>/etc/ssh/ssh_config <<EOF Host * RevokedHostKeys /etc/ssh/ssh_revoked_hosts EOF cat >>/etc/ssh/ssh_revoked_hosts <<EOF # https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/ https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-k... ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ== EOF ```
- LeonM 4y agoYou can do verbatim formatting on HN by placing 4 spaces. See https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc Thanks for the gist though, seems helpful!
- e12e 4y agoThank you - TIL about ssh key revocation (I was aware of them, but haven't really used them). I expanded on your gist: https://gist.github.com/e12e/0c1868479c0b8d0a52914d44be66d769 https://gist.github.com/e12e/0c1868479c0b8d0a52914d44be66d76...
- ollien 4y agoJust in case anyone is paranoid that this comment has the right key, you can generate a fingerprint with $ ssh-keygen -lf github.old.pub 2048 SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8 no comment (RSA) and you'll notice that fingerprint is on this archived page https://web.archive.org/web/20230320230907/https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints https://web.archive.org/web/20230320230907/https://docs.gith... (please check my work on your own machines and don't take my attestation on faith!)
- dave4420 4y agoNot mentioned: whether this also affects gist.github.com. They seem to have been using the same key.
- creamyhorror 4y agoUseful comment about git using the ECDSA key instead and running into a mismatch issue (from the Github discussion): ===== After I followed the instructions to remove the old RSA key, `git pull` started using the ECDSA key, and now shows this warning: Warning: the ECDSA host key for 'github.com' differs from the key for the IP address '20.205.243.166' Offending key for IP in /home/forge/.ssh/known_hosts:87 Matching host key in /home/forge/.ssh/known_hosts:88 Are you sure you want to continue connecting (yes/no)? In this case, I think the old ECDSA key for the github.com IP needs to be removed from `known_hosts`. This can be done with: ssh-keygen -f ~/.ssh/known_hosts -R 20.205.243.166 It worked for me, hope this helps. I'm not sure why the ECDSA key of the github.com IP has supposedly changed - maybe someone can clarify this?
- hddqsb 4y agoThe ECDSA key hasn't changed. As pointed out elsewhere in this thread, the official command `ssh-keygen -R github.com` is incomplete -- it doesn't remove IP address entries, they need to be removed manually. What you got was a conflict between the ECDSA key and the old RSA key left over in ~/.ssh/known_hosts (the error message was unclear). Since GitHub's IP address is not stable, I suggest disabling the IP-checking feature using Host github.com CheckHostIP no The CheckHostIP feature is pretty useless anyway, it just gives a warning when the IP changes and gives a better diagnostic message if the key and IP both change at the same time (references: https://serverfault.com/questions/1040512/ssh-how-does-the-option-checkhostip-yes-really-help-me https://serverfault.com/questions/1040512/ssh-how-does-the-o..., https://unix.stackexchange.com/questions/285520/why-does-ssh-add-a-known-host-entry-for-an-ip-address/285551#285551 https://unix.stackexchange.com/questions/285520/why-does-ssh...).
- guluarte 4y agoHave a fun weekend, all my DevOps fellows!
- stockhorn 4y agoHow likely is it, some bad actor can pull off a MitM on someone using a resident/coporate connection (and not a public hotel wifi...) ? How would one even pull this off? (Except for ISPs or security agencies)
- uglygoblin 4y agoI think it's bizarre they just casually use jq in their "automatic" fix commands. Is it really that ubiquitous?
- nessex 4y agoIt's not mentioned in the blog post or keys page, but the _old_ value[1] you'll find in known_hosts is: github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ== You can search for this in your codebases, hosts etc. to see if there are any areas that need updating. The new value is linked from the blog post, you can find it here: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints https://docs.github.com/en/authentication/keeping-your-accou... [1] https://github.blog/changelog/2022-01-18-githubs-ssh-host-keys-are-now-published-in-the-api/ https://github.blog/changelog/2022-01-18-githubs-ssh-host-ke...
- Timothee 4y agoGood callout! I looked at my `~/.ssh/known_hosts` file and that key is associated with a few IP addresses in addition to github.com. Those lines stayed after I ran `ssh-keygen -R github.com`. I imagine that I also need to remove those other lines manually, but isn't that something that GitHub should have mentioned? I'm not sure in which circumstances these got added either…
- bravetraveler 4y agoSame here, found the two following IPs with the same hostkey: - 192.30.253.112 - 140.82.114.3 WHOIS shows github ownership, just not sure when/how/why I got these
- bxparks 4y agoMy ~/.ssh/known_hosts used to look like that, but now it looks like the host part is encrypted? It looks like: |1|{base64-encoded-string?}|{another-base64-encoded} ssh-rsa AAA{long-string} This is on Ubuntu 22.04 (Linux Mint).
- banana_giraffe 4y ago
- gehaxelt 4y agoIt would be great if Github and other large providers could start using SSHFP records [0]. [0] https://www.rfc-editor.org/rfc/rfc4255.html https://www.rfc-editor.org/rfc/rfc4255.html
- LeonM 4y agoAFAIK openssh does not use SSHFP (VerifyHostKeyDNS) by default, for good reason. Also, SSHFP requires DNSSEC, which Github of course does not support. (amongst other shiny new technologies, such as IPv6... /r). And even if GH would deploy DNSSEC, it'll still be opening you up to a host of other attack vectors that come with DNS based trust anchors.
- vbezhenar 4y agoIt continues to amaze me that ssh still does not use https PKI and relies on developer manually checking fingerprint (which he supposedly does in a hard-to-google location and realistically he does not check anything). So much care and work went to implement web security and developers still basically live in a self-signed world. At least put your key to https://github.com/.well-known/ssh/ed25519.pub https://github.com/.well-known/ssh/ed25519.pub so I don't need to Google it... And may be some day ssh will support it natively. Someone need to act first.
- franky47 4y agoIf anyone from GitHub is reading, the docs page for testing SSH keys needs an update on the RSA key fingerprint: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/testing-your-ssh-connection https://docs.github.com/en/authentication/connecting-to-gith... (it would actually be best to update it to the Ed25519 public key, as it's the recommended algorithm when generating a new SSH key).
- hddqsb 4y agoGood point, but a clarification re Ed25519: what matters here is the host key algorithm with highest preference, not the default/recommended algorithm for the identity key. The highest preference appears to be Ed25519 or ECDSA depending on the OpenSSH version; see HostKeyAlgorithms in ssh_config(1).
- SURA 4y agoThey forced me not to use a password to pull code before and only informed me of a password leak several months later. There must have been something serious happened this time as well.
- abraham 4y agoGitHub has had a password breaching the past?
- Aeolun 4y agoWell, I’m sure glad there was a reason for me trashing the previous key without giving it a second thought. Commits need to commit!
- acatton 4y ago$ ssh -o VerifyHostKeyDNS=yes github.com The authenticity of host 'github.com (140.82.121.3)' can't be established. ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU. No matching host key fingerprint found in DNS. They didn't update their DNS SSHFP records? https://en.wikipedia.org/wiki/SSHFP_record https://en.wikipedia.org/wiki/SSHFP_record What a security nightmare…
- esclear 4y agoThat is the ED25519 key's fingerprint, which shouldn't have changed. But that is moot, as GitHub doesn't publish fingerprints as SSHFP records.
- tome 4y agoTheir instructions could stand to be more thorough. If you have hashed hostnames in your known_hosts, you don't just have to ssh-keygen -R github.com you also have to ssh-keygen -R 140.82.121.3 which is the IP address for that host. Otherwise you'll get something like Warning: the RSA host key for 'github.com' differs from the key for the IP address '140.82.121.3' Offending key for IP in /home/tom/.ssh/known_hosts:77 Matching host key in /home/tom/.ssh/known_hosts:102
- jasongill 4y agoThere are numerous IP's that are reported for github.com. The easiest way to remove all of the offending known_host entries appears to be: sed -i.github-removed '/AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa/d' ~/.ssh/known_hosts
- St-Clock 4y agoThank you so much for this shortcut, I was trying a way to find all of the bad known_host entries and this did the trick!
- 5mv2 4y agoCorporate PR lessons from this post: - Push security breach details down so fewer people read them. Nothing until line 11. Or line 13 if you count the TLDR. - Litter your text with understatements. After publicly exposing keys, you replace them "out of an abundance of caution" to replace them. And yet this is one of the best security breach posts I've seen in a while. When will companies start to respect developers?
- cprecioso 4y agoOne-liner to update your `known_hosts`, directly from github (if you have `jq`) ssh-keygen -R github.com && curl 'https://api.github.com/meta' | jq -r '"github.com " + .ssh_keys[]' >> ~/.ssh/known_hosts
- masklinn 4y agoSeems unnecessarily complicated, you can just `ssh-keygen -R` then `ssh git@github.com`, it should ask if you want to add the new key to the known hosts. Also you should first check: 1. If you’re using GitHub’s rsa key at all 2. That you don’t have the other keys In which case nuking all the keys is completely unnecessary, just remove the RSA one.
- jwilk 4y ago> it should ask if you want to add the new key to the known hosts But how do you know it's the right key?
- masklinn 4y agoBy comparing the fingerprint to a trusted source, which the snippet above doesn't do. And if you trust that the https source will lead you to the correct github, you can trivially check the fingerprint against what github publishes.
- cprecioso 4y agoYou’d be pressed to find someone who doesn’t trust the https source. So… you’d do manually what the snippet does?
- Phemist 4y ago> If you are using our ECDSA or Ed25519 keys, you will not notice any change and no action is needed. How is this true? AFAIK, A MITM with the github priv key will be able to do an SSH downgrade attack...
- hddqsb 4y agoThe SSH client verifies that the host key sent by the server matches the one stored in ~/.ssh/known_hosts. If the key material or algorithm is different, the client refuses to connect. So the server can't force the client to switch from ECDSA / Ed25519 to RSA.
- hddqsb 4y agoEdit: The above is true if known_hosts contains only the ECDSA / Ed25519 host key. But as mentioned elsewhere in this thread, newer OpenSSH versions automatically add alternate host keys (https://lwn.net/Articles/637156/ https://lwn.net/Articles/637156/), in which case the old RSA key would have been added to known_hosts even if only ECDSA / Ed25519 were used. In that case, the user will be vunlerable until the next time they ssh into (non-MITM) github.com, since that will update the RSA key in known_hosts.
- deleted 4y ago[deleted]
- amrb 4y agolet he who has never committed credentials to a repo, cast a stone
- amrb 4y agoI will give you the keys of the kingdom of heaven; whatever you bind on earth will be bound in heaven, and whatever you loose on earth will be loosed in heaven.
- deleted 4y ago[deleted]
- pavlov 4y ago"Hey Copilot, complete the following statement: let GITHUB_COM_PRIVATE_KEY = "
- daneel_w 4y agoWhy 3072 bits for the key instead of 4096? I understand that they see a huge amount of connections to their systems every second, but this isn't the year 2008. They have the processing power. Is it a decision based on "good enough"? Or some compatibility nod towards old abandonware still in use?
- barryfam 4y agoIn case anyone wants another example of how to set up revoked host keys: # Add this to the top of ~/.ssh/config RevokedHostKeys /home/username/.ssh/revoked_host_keys then cd ~/.ssh mkdir revoked_host_keys.d echo 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==' > revoked_host_keys.d/github-leak-2023.03.23 ssh-keygen -k -f revoked_host_keys revoked_host_keys.d/* the last command combines all keys in the subdirectory into the one properly-formatted binary file. So, you can add more keys into the subdirectory later (but you do have to remember to rerun it -- personally I saved it into a one-line script at ~/.ssh/revoked_host_keys.sh so I don't forget)
- LeonM 4y agoThe fact that this key was apparently not stored in an HSM, and that GH employees had access to this private key (allowing them to accidentally push it) means that effectively all communication with GH since the founding of the company has to be considered compromised. This basically means that, depending on your level of paranoia, you will have to review all code that has ever interacted with Github repositories, and any code pushed or pulled from private repositories can no longer be considered private. Github's customers trust GH/MS with their code, which, for most businesses, is a high value asset. It wouldn't surprise me if this all results in a massive lawsuit. Not just against GH as a company, but also to those involved (like the CISO). Also, how on earth was it never discovered during an audit that the SSH private key was plain text and accessible? How has GH ever been able to become ISO certified last year [0], when they didn't even place their keys in a HSM? Obviously, as a paying customer I am quite angry with GH right now. So I might be overreacting when I write this, but IMO the responsible persons (not talking about the poor dev that pushed the key, but the managers, CISO, auditors, etc.) should be fined, and/or lose their license. [0] https://github.blog/2022-05-16-github-achieves-iso-iec-270012013-certification/ https://github.blog/2022-05-16-github-achieves-iso-iec-27001...
- djbusby 4y agoWhat license?
- LeonM 4y agoAuditors require a license/accreditation to do certain certifications.
- brightball 4y agoI wonder if they found it by turning on their own secret detection system?
- Veliladon 4y ago> The fact that this key was apparently not stored in an HSM, and that GH employees had access to this private key (allowing them to accidentally push it) means that effectively all communication with GH since the founding of the company has to be considered compromised. For a host key? Like I get that being able to impersonate Github isn't great as far as state level actors having the ability to do this but you do know the actual transport layer keys are ephemeral and aren't derived at all from the host key, right?
- reuven 4y agoWhy did I, a paying GitHub customer, discover this when trying to do "git pull" on a repo I own? My immediate assumption wasn't that GitHub had changed keys. I thought that my computer was having some sort of problem. Only after searching around for a bit did I find out that yes, GitHub changed their private key. Couldn't they have e-mailed all of their users and/or customers, to tell us what had happened?
- tabbott 4y agoEmailing every GitHub user may take a while, but it seems wrong that they didn't post it on https://githubstatus.com https://githubstatus.com; this is effectively an outage in their service -- customers should not need to google for their blog post to find out what happened.
- m4lvin 4y agoWell, that would be something to show in the "Latest changes" in the top right corner on github.com after login, right? But no, I have to go here to read about it. It is also not mentioned on https://www.githubstatus.com/ https://www.githubstatus.com/ aka status.github.com.
- hk1337 4y agoI'd switch to Bitbucket but they don't support ed25519-sk and ecdsa-sk, among other things.
- fmajid 4y agoThis would have been a good opportunity for them to issue SSH certificates (backed by a HSM) instead of just keys.
- amrb 4y agoA number of large companies you use everyday don't have key management, if you ask them for a architecture demo/diagram they will squirm!
- nwellnhof 4y ago> If you see the above message, you’ll need to remove the old key by running this command: $ ssh-keygen -R github.com That's bad advice. Everyone should remove the compromised key, not only people seeing the message. If you don't see the message and everything still works (while using RSA), you've been MitMed.
- xign 4y agoAlso, from what I can tell, the old key somehow seems sprinkled among a lot of Github-owned IP addresses in my known_hosts file as well, not just github.com. I'm not sure how that happened though.
- Timwi 4y agoInfo to everyone using Windows and as confused as I was: • When you run a git command and git tells you about the mismatching key, it will prompt you to enter `y` or `n` but not actually let you do so. • You need to run `plink github.com` to get the same prompt. You can then enter `y` to accept it. • You should only accept it if it says the new fingerprint is d5:2c:63:d9:bc:75:9d:de:b1:4e:36:28:9f:7a:9c:39. If it is any different, you may be targeted by an attacker. • The new key is stored in the file `C:\Users\<your name>\.ssh\known_hosts`. The new key at the time of this writing is `github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=`. • In the future, if the key ever changes again, you can look it up on GitHub’s SSH key fingerprints page [0]. [0] https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints https://docs.github.com/en/authentication/keeping-your-accou...
- omoikane 4y agoplink is part of PuTTY right? In which case I think the host keys are in the registry, under HKEY_CURRENT_USER\SOFTWARE\SimonTatham\PuTTY\SshHostKeys
- pornel 4y agoNo worries, I wasn't checking it anyway: https://nvd.nist.gov/vuln/detail/CVE-2023-22742 https://nvd.nist.gov/vuln/detail/CVE-2023-22742
- aftbit 4y agoJust in case anyone is curious, you can get the current github host keys from their HTTP API[1]. Our deploys fetch the latest and add them to a known_hosts file as part of the process. We set the `GIT_SSH_COMMAND` environment variable to use a temporary known hosts file with just the needed keys on each deploy. We do it in fabric with Python, but here's similar bash: curl -s https://api.github.com/meta | jq -r '.ssh_keys | join("\n")' | sed 's/^/github.com /' > /etc/github_known_hosts GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/etc/github_known_hosts" git pull 1: https://api.github.com/meta https://api.github.com/meta
- fjni 4y ago> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository > ... out of an abundance of caution, we replaced our RSA SSH host key used to secure Git operations for GitHub.com Yeah, that's not an "abundance of caution." That's the bare minimum response at that point. What's the "not cautious approach?" Make the repo private and go on your merry way?
- neilv 4y agoI had just composed a comment with the exact same two quotes before I saw yours. I suppose "abundance of caution" might apply, if they determined that the only ways it could've leaked were from requests that were logged, and they've removed all the ways and checked all the logs. But if I had to guess, even a brief exposure can be picked up by bots (and perhaps they can already see log entries for this). Even if no one at all picked it up, there'd still be the question of whether traces of it are still left behind on various infrastructure, in storage and caches (even ML training?) not intended for key safety.
- fjni 4y agoYou are right of course. There’s a scenario where they have exhausted all possible ways in which it could have leaked, determined that it wasn’t and decided despite that conclusion to rotate the key. That would be doing so purely out of abundance of caution.
- pxc 4y agoDevelopers can't be trusted with SSH private keys. I'm stunned by how many times I've asked a developer to send their SSH public key only to see them dump their private keys on Slack, in a Pastebin, or in a Git commit or wherever. It would be so much better if standard practice was to generate and store the private key on a smartcard or the TPM, so that the only file a clueless/careless developer could upload would be a stub.
- Reitet00 4y ago> It would be so much better if standard practice was to generate and store the private key on a smartcard or the TPM, so that the only file a clueless/careless developer could upload would be a stub. Yep. Especially given that basically all modern laptops (and some PCs) ship with TPMs and ssh can use it via the TPM PKCS#11 lib. I'm using that daily on multiple machines and it's working great.
- 8organicbits 4y agoIt's too late to fix now, but that sounds like a file naming UX issue. If you type `cat ~/.ssh/id_` and hit tab, you'll see `id_rsa` first, which is the wrong one. It really should be named `id_rsa.private` to help a busy developer realize they have the wrong file.
- lopkeny12ko 4y agoFor these major changes that break builds across the entire industry, why does Github insist on rolling out the change first and then publishing a blog post notice about it? Is it really so hard to give a week of advance notice? It wasn't even that long ago that Github changed the checksums of git tarballs which also broke builds everywhere. All we got was a "oh btw we changed the archive behavior" blog post after the change went live [0]. [0] https://github.blog/changelog/2023-01-30-git-archive-checksums-may-change/ https://github.blog/changelog/2023-01-30-git-archive-checksu...
- ihattendorf 4y agoI think changing a key because it was compromised is a little different compared to changing archive behavior, and can be afforded a little leeway with advance notice.
- 8organicbits 4y agobroken builds << migrating off a leaked key I'm upset they haven't sent out an email to every user. This affects their entire usebase and basically everyone needs to take manual action to continue working. If you haven't seen the news then you'll see failed builds/git commands.
- spyremeown 4y agoJesus Christ, what a bunch of incompetent morons, really. I honestly hope they get sued to oblivion with this. M$ and GitHub are having their way with everything lately, this is the first of many mishaps that are bound to happen we put so much trust in one company.
- deleted 4y ago[deleted]
- AtNightWeCode 4y agoSo what actions am I supposed to take to verify that I am on the safe side? The blog post did more to downplay the problem than actually being helpful for the end users.
- fierro 4y agoYou could literally go grab a Wifi Pineapple, sit in a Starbucks in SF, and MITM Github for anyone waking up and blindly running `ssh-keygen -R github.com`. Very opportune time for hackers to try this, since few are probably verifying the fingerprint.
- screcth 4y agoDoes this mean that someone could decrypt a old dump of traffic between a client and GitHub? If so, this means that every repo on GitHub must be assumed to have been leaked?
- Madhouse61 4y agoIf I was a three letter agency, which I'm not. I'd be interested in conducting a covert operation to glean data on Github repositories, sort of like a mass-code surveillance tool. The likes of this idea sparks my curiosity - but who I am kidding, this could never happen. GitHub is a secure and reliable corporation owned by Microsoft.
- twothamendment 4y agoI'd love to see the sum of time lost around the world with this one little change!
- throwawaaarrgh 4y agoEveryone should standardize on HTTPS to pull repositories instead of plain (non-certificate) SSH. A username and password (often called an API token) are used instead of an SSH key. - An SSH key can be revoked on the server, but the client won't register it immediately, and all clients must manually verify the new key signature and update their local configuration. This doesn't happen with HTTPS; the client just works with the server's new cert, no user intervention is required. - A username can be used in the HTTPS URI to tell both your client and the server what credential to use. SSH method requires the user to load the correct SSH key first. - Most servers like GitHub allow more fine grained access control for HTTPS tokens than SSH keys. - HTTPS access method works on all HTTP proxies, whereas SSH is often blocked. - HTTPS can be faster than SSH. - HTTPS allows a client and server to use specific ciphers to address regulatory and other requirements. - HTTPS tokens are supported in a wider range of password managers/keychains than SSH keys. - Most users don't password protect their SSH keys, but they often have a password manager with a master password that they can keep HTTPS token in. - SSH key management is much more complex on the user end than HTTPS token management. The former requires ssh-agent, a key generator, and instructions for use, as well as specific filesystem permissions for keys. - The user often doesn't understand the idea that they shouldn't share their private key. But everyone basically gets they shouldn't share their password. - Virtually no one ever checks the fingerprint of ssh server keys. Often entire companies have configuration that disables host key checking, completely eliminating the security SSH is supposed to provide. With HTTPS the user is implicitly protected with no actions necessary. - Nobody ever commits a private TLS key to a GitHub repo, but apparently they do with SSH private keys...
- organman91 4y ago> Nobody ever commits a private TLS key to a GitHub repo Oh how I wish this were true...
- xign 4y ago> - Most users don't password protect their SSH keys, but they often have a password manager with a master password that they can keep HTTPS token in. Are there password managers that actually seamlessly do that? GitHub's official docs (https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token#using-a-personal-access-token-on-the-command-line https://docs.github.com/en/authentication/keeping-your-accou...) for example recommends using the GitHub CLI to cache your authentication token for you and as far as I can tell the token is stored unencrypted (https://github.com/cli/cli/issues/1773 https://github.com/cli/cli/issues/1773), unlike SSH keys which should be encrypted behind a password. If you don't do this you have to enter your password every single time you use you Git, which is not good even if you have a password manager (it adds steps, and passwords aren't great to begin with because you send them in plain text to the server and it is why Apple etc are moving away from password-based authentication for websites). > - Nobody ever commits a private TLS key to a GitHub repo, but apparently they do with SSH private keys... I think the key issue here is this really shouldn't have happened. There is no reason why GitHub couldn't secure their SSH private key just like their TLS key. --- Edit: Thinking more about it, I think the situation would have been much worse if GitHub leaked a TLS key instead. With SSH, an attacker who stole GitHub's private SSH key actually *cannot* MITM the user. They can pretend to be GitHub to the user, but they cannot pretend to be the user to GitHub, because with SSH both sides have a private/public key known to the other side. In a normal usage scenario, it would actually take some social engineering to do a real attack, because a simple Git fetch from the attacker would fail when trying to fetch a private repo (since the attacker does not have the permission to fetch it from GitHub). With HTTPS, under current setups of username/password or an auth token, the user has no real way to protect themselves from a MITM attack. The token isn't a private key as it's shared directly to the server. That means the attack could actually MITM the server/user connection and do much much more damage.
- foresto 4y agoFor anyone who removed the old key and accepted the new one without comparing it to the published list[1], you can review the fingerprint you accepted with this command: ssh-keygen -l -F github.com It works for IP addresses as well, which are also published[2]. [1] https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints https://docs.github.com/en/authentication/keeping-your-accou... [2] https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/about-githubs-ip-addresses https://docs.github.com/en/authentication/keeping-your-accou...
- pashadee 4y agoOur keys are documented here*. Hyperlink with broken certificates. Fun.
- CoachRufus87 4y agossl cert expired on https://gist.githubusercontent.com/ https://gist.githubusercontent.com/; #hugops
- madsbuch 4y agoMaybe this is why I start getting the following error error pulling image configuration: download failed after attempts=6: x509: certificate is valid for *.githubassets.com, githubassets.com, not pkg-containers.githubusercontent.com
- deleted 4y ago[deleted]
- pyuser583 4y agoI had a very unpleasant 30 seconds this morning.
- jason_s 4y agoHow come github.blog doesn't have the "GitHub, Inc." organization as part of its certificate, the way github.com does? $ openssl s_client -showcerts -connect github.blog:443 CONNECTED(00000006) depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = R3 verify return:1 depth=0 CN = github.blog verify return:1 --- Certificate chain 0 s:CN = github.blog i:C = US, O = Let's Encrypt, CN = R3 a:PKEY: id-ecPublicKey, 256 (bit); sigalg: RSA-SHA256 v:NotBefore: Feb 3 17:19:36 2023 GMT; NotAfter: May 4 17:19:35 2023 GMT ...
- bananapub 4y agobecause it's a purely domain-verified LetsEncrypt certificate vs the EV company-paperwork-verified cert for github.com
- dolmen 4y agoUsers of github-keygen [1] might be affected. Please check the public key recorded in ~/.ssh/known_host_github. Note that github-keygen stores the ssh-ed25519 key since release 1.306 in June 2022. You can check by yourself: cat ~/.ssh/known_hosts_github If you are affected, just run github-keygen again to regenerate your SSH configuration: curl --silent https://raw.githubusercontent.com/dolmen/github-keygen/release/github-keygen | perl Disclaimer: I am the author and maintainer of github-keygen [1]: https://github.com/dolmen/github-keygen/ https://github.com/dolmen/github-keygen/
- dolmen 4y agoNote that the use of curl for the upgrade is on purpose: if the user had a old copy of github-keygen made with `git clone` and the remote is configured using SSH, a `git remote update && git rebase` would not work because of the revoked SSH key. So, in this case, I consider that curl over https is safer as long as your curl version is recent and your local repository of TLS certificiates is up to date.