9 ms·
SSH Key Audit on Github (required)
- memset 15y agoHere is the command you use to obtain your fingerprint for this audit: `ssh-keygen -lf ~/.ssh/id_rsa.pub`
- bobbyi 15y agoThank you. They should include that information on the page because otherwise it is really unclear how you can tell if the listed key is correct.
- zdw 15y agoSome people might be running DSA keys - this covers that case: `ssh-keygen -lf ~/.ssh/*.pub`
- masklinn 15y ago> Some people might be running DSA keys Yep, or use a non-default key name.
- amirmc 15y agoFair point but if someone is using non-default names then they presumably had to deal with ssh config files too. I'd expect such users to have just enough familiarity with ssh keys to know how to check them (though I could be wrong). Edit: Also, GitHub's setup guide explicitly leads to rsa keys so anyone using dsa keys consciously made that choice http://help.github.com/mac-set-up-git/ http://help.github.com/mac-set-up-git/
- cschmidt 15y agoThanks for this. I found it kind of weird that the email they sent didn't include any instructions on what to do. The linked webpage didn't have any guidance either. I guess they assume we all know what to do.
- charliepark 15y agoThis. In fact, it creates an even more dangerous situation, as users could go to the site, see their keys, and say "I dunno. Looks fine?" and approve all of the keys, without actually confirming that the keys are legitimate. Not giving instructions on the page on how to verify the info was weak. Github people, if you're reading this, please update that page.
- amirmc 15y agoThis is a good point. I was really impressed with their GitHub Bootcamp [1], which made setting up the keys and getting started a breeze. It's a shame that hasn't carried through to the key audit page but there is (now) a link on how to verify keys (http://help.github.com/verify-ssh-redirect http://help.github.com/verify-ssh-redirect) [1] http://help.github.com/set-up-git-redirect http://help.github.com/set-up-git-redirect
- brazzy 15y agoHow is that "more dangerous" than having the users never look at the keys. Most users will not have many keys (probably just one), and will be alerted if they see more.
- julian37 15y agoI very much agree they should have added instructions to the page. However when I went through the process there was a prominent note saying that when in doubt, you should reject keys and upload new ones. So the "I dunno. Looks fine?" case seems like it would be a problem only for the careless.
- mikeash 15y agoSort of like how the Rails default settings only caused problems for the careless.... I bet a lot of people would have verified their keys with instructions but didn't bother without.
- bricestacey 15y ago
- js2 15y agoYou can also use 'ssh-add -l' to see all the keys you've added to ssh-agent.
- k33l0r 15y agoYou can also iterate over all the public keys in ~/.ssh: for f in ~/.ssh/*.pub; do ssh-keygen -lf $f; done
- tonyarkles 15y agoI had the same problem. I emailed them to invite them to add that exact command to the audit page. Glad to see I wasn't the only person who had trouble.
- graywh 15y agoThere's a help page now: http://help.github.com/linux-verify-ssh/ http://help.github.com/linux-verify-ssh/
- deleted 15y ago[deleted]
- finnh 15y agoSadly the link in the email isn't direct (it's a tracking link through "news.github.com"), so Thunderbird flags it as a possible phishing attempt =( Edit: github send out an email with a link to the ssh audit page; that's the email to which I refer
- spullara 15y agoI guess this answers my questions about how long this vulnerability existed (a long time) and whether or not they could verify no other accounts were compromised (no).
- zby 15y agoI've just seen it and I headed to Hacker News to verify if it was legit :)
- oelmekki 15y agoSame here, wasn't sure seeing no reference to github.com in mails' header, even if the link pointed directly on their domain. i guess that's the precise moment when I should feel too much paranoid.
- Ecio78 15y agoI've just registered yesterday on Github (it's suggested for Coursera's Saas Class i'm attending) but they've sent it to me too, even though the vulnerability has already been resolved before my account was created. Maybe they've not checked account age..
- sophacles 15y agoI suspect that since they only closed the hole late Sunday or early Monday, that they decided the number of new accounts was so small, it wasn't worth introducing the complexity (and potential bug/insecurity sources) to handle such a small percentage of the user-base. A reasonable tradeoff since getting this audit done in a timely manner is more important than waiting to handle edge cases such as yours.
- jgrahamc 15y agoIt would be interesting to know the details of the vulnerability. Given that they've patched it, it would be good to see what the error was in case others are affected. Was this Rails-related and what was it?
- chimeracoder 15y agoThere are a number of articles that surfaced on Sunday/Monday, but in short, yes - it was a Rails mass-assignment vulnerability.
- oscardelben 15y agoI'd like to point out that this is not a rails vulnerability, but a mistake Github engineers made, which happens to the best of us. Mass assignment is a feature and I guarantee the problem has been know for years and Github engineers were probably well aware of it.
- lucian1900 15y agoIt most certainly is a vulnerability in rails, by encouraging bad practice by design. Mass assignment should work by defaulting attributes to protected, if it should exist at all.
- anthonyb 15y agoAnd yet they still fell prey to it. Insecure by default is not secure, even if you can fix the insecurities.
- jarek 15y agoThe same way magic_quotes was a "feature".
- technoweenie 15y agoIt was a mass assignment vulnerability in our code.
- deleted 15y ago[deleted]
- andrewjshults 15y agoThey also did a notification when you tried to push: ERROR: Hi andrewjshults, it's GitHub. We're doing an SSH key audit. Please visit https://github.com/settings/ssh/audit/<removed> https://github.com/settings/ssh/audit/<removed> to approve this key so we know it's safe. Fingerprint: <removed> fatal: The remote end hung up unexpectedly A little weird to see when you're doing a push but good that they put it in there. Their email got flagged as bulk in gmail so until I saw this I didn't know they were doing the audit.
- kapitalx 15y ago> "good that they put it there" I think this was handled very badly. This broke alot of automated jobs for many people. The email actually arrived after some of our automated jobs had stopped running. There should have been an advanced warning that such an audit would take place. I'm glad we had monitoring on our automations, but many people won't have those and might still be unaware that they are not running until they authorize their keys.
- deleted 15y ago[deleted]
- brazzy 15y agoAND when you tried to pull, which broke my current project's build. First time I've seen current news directly affect my work...
- pilif 15y agoWhat makes me the most happy about this is that they ask for the password in order to add a key now. I was always very afraid of XSS attacks (I know - there shouldn't be any - but there could and were, though not for this) that would add another key, so I always hoped they would add that additional bit of protection. As such: Another huge thanks to @homakov for forcing the issue.
- SaltwaterC 15y agoThe API token is still there, in the "plain": https://github.com/settings/admin https://github.com/settings/admin Fetching it via XSS should be fairly trivial. Via a simple script in that page is straight forward. Still have to see if I can get it via XHR :).
- natrius 15y agoThis change is orthogonal to @homakov's attack. Silently changing the user id on a key wouldn't generate an email.
- ajross 15y agoThe change is good and was quite clearly precipitated by Homakov. Thus it seems entirely reasonable to thank him. So I don't see what your point is beyond irrelevant pedantry.
- orblivion 15y agoI don't think it's pedantry. I think it's good to point out that while this change is good, and is in the same realm, if you give the impression that it would prevent an attack like the one this weekend, it's security theater.
- pilif 15y agoIt doesn't and I really hoped that I was clear enough in my initial comment. I know that this has nothing to do with the attack on Sunday. But ever since that XSS vulnerability was shown maybe a year ago, I was scared that there could be another one of them as they are very, very easy to miss, especially when you have to rely on filtering instead of escaping. Github has to because you can freely use HTML in Markdown, which is the main markup language on github. So there we are: Relying on filtering out bad stuff from user content, instead of blindly escaping it, at least one XSS vulnerability already happened and the public key interface still allows adding keys without confirming the password. It would really suck to be hit by a XSS attack that silently adds a key to my account, not only giving the attacker the possibility to impersonate me in commits, but, even worse, giving access to my private repositories. Regardless of the fix for the problem on Sunday, I have always hoped Github would add the check for the password. Not to mitigate the mass-assignment problem, but to prevent a possible XSS attack from being used to deal much worse damage. I see this package of changes that we see now happening at Github as a direct result of the Sunday hack, so while we got the mass-assignment fix (of course that had to be fixed), we also finally got the password recheck which we likely would not have gotten without the hack. Hence my "thank you" to @homakov. Not just for uncovering the mass-assignment thing (which is a simple code fix), but also for forcing a change of policy (with negative usability repercussions and thus probably not universally accepted between github internals) elsewhere.
- niels_olson 15y agoUm, is anybody else having the experience that their keys really do seem to be different?
- rmc 15y agoYou should probably tell GitHub
- niels_olson 15y agoI thought I deleted this comment. Sorry. Turns out I had an old key still in my .ssh folder. Two current and valid keys and no others. My bad.
- jdludlow 15y agoEach fingerprint listed on my audit page was exactly what I expected it to be.
- brown9-2 15y agoSure you aren't confusing the fingerprint with the actual key contents?
- tclem 15y agoDid you contact support@github? We definitely want to know about that.
- alastairpat 15y agoI'm missing a whole bunch of keys – I previously had about 4 keys registered, now it's just one.
- spicyj 15y agoThe accompanying email: A security vulnerability was recently discovered that made it possible for an attacker to add new SSH keys to arbitrary GitHub user accounts. This would have provided an attacker with clone/pull access to repositories with read permissions, and clone/pull/push access to repositories with write permissions. As of 5:53 PM UTC on Sunday, March 4th the vulnerability no longer exists. While no known malicious activity has been reported, we are taking additional precautions by forcing an audit of all existing SSH keys. # Required Action Since you have one or more SSH keys associated with your GitHub account you must visit https://github.com/settings/ssh/audit to approve each valid SSH key. Until you have approved your SSH keys, you will be unable to clone/pull/push your repositories over SSH. # Status We take security seriously and recognize this never should have happened. In addition to a full code audit, we have taken the following measures to enhance the security of your account: - We are forcing an audit of all existing SSH keys - Adding a new SSH key will now prompt for your password - We will now email you any time a new SSH key is added to your account - You now have access to a log of account changes in your Account Settings page Sincerely, The GitHub Team --- https://github.com support@github.com
- rwmj 15y agohttp://unix.stackexchange.com/questions/2116/given-keys-in-ssh-authorized-keys-format-can-you-determine-key-strength-easi/2146#2146 http://unix.stackexchange.com/questions/2116/given-keys-in-s... This script is very useful when doing this audit, because you can turn your .ssh/authorized_keys file into a list of key names and fingerprints to check against what github is showing you.
- homakov 15y agoyou got balls guys. It is hard to force everyone to do something but you did it. Kudos also, if we go back few years ago this way would be a bit secure to handle keys @key.body = params.. @key.title = params.. I am sure update_attributes is good choice when you got 5+ fields and update database scheme pretty frequently. Just my 2 cents
- technoweenie 15y agoWe sent someone back in time to make this change. Somewhere there's an alternate universe with a secure GitHub by default. Thanks!
- deleted 15y ago[deleted]
- rdl 15y agoWhy are ONLY keys at risk, which this implies? Presumably someone could have added a key, done evil, then removed the key. Evil includes all sorts of interesting things, like checking in code under the name of an existing contributor. This could potentially be really subtle and would be difficult to find in an audit later. (Remember the stink over OpenBSD potentially having backdoors in the IPsec stack, revealed in late 2010? http://blogs.csoonline.com/1296/an_fbi_backdoor_in_openbsd http://blogs.csoonline.com/1296/an_fbi_backdoor_in_openbsd)
- ivix 15y agoMy guess is the last thing github want to do is make its users aware of that distinct possibility.
- gcr 15y agoI disagree. It is github's duty to provide a realistic view of the situation. Such an attack is unlikely, yes, but still possible.
- Volpe 15y agoNo, it's not possible... how do you delete a key with a mass-assignment hack?
- rdl 15y agoDeleting the keys afterward isn't really a key (heh) part of the attack, just a nice way to cover your tracks afterward. Enough accounts have multiple expired or forgotten keys to make the mere presence of multiple keys and potentially some unknown keys not always an absolute indicator of compromise, too.
- bricestacey 15y agoIt's likely that your ssh key only authenticates your account anyway. If that were the case, you could change your key to a different account, make a change, and then change it back and no one would be the wiser (though any logs demonstrating an identical key across users would identify it, but unlikely they exist and in a form worth pursuing). Security is hard.
- pak 15y agoAs an interesting side effect, they will have pretty exact stats on how many active users they have; might help them sunset old accounts or move them to the slowest servers. (Because of the offline nature of most git actions and different habits on pushing/pulling, it's probably hard to otherwise estimate how much a user cares about their github.)
- arohner 15y agoWhy is this more effective than say "number of push / pulls in the last month"?
- pak 15y agoSome people try to pool local commits into larger, less frequent pushes and pulls so the number of push/pulls is perhaps less relevant than their cumulative size. But pushes/pulls will never correspond well with user involvement because people use github for all kinds of scenarios. For instance, I might be developing branches that I don't want to push or pull to github yet--maybe I don't even intend to ever make them public. However, I may still want people to clone from my github repo and report issues to me. The amount of time between someone getting an email from github and re-activating their account is probably the best metric github will ever have on users' attachment to their accounts.
- deleted 15y ago[deleted]
- icefox 15y agoFrom the way their servers are laid out in the presentation I saw they wouldn't want to move them all to a slow server, but would want them evenly distributed to help decrease load on the servers.
- technoweenie 15y agoWe have stats on this from other sources, actually. But it did provide some interesting graphs this morning :)
- levigross 15y agoThey also added a audit log so you will be able to track and address any future issues.. https://github.com/settings/security https://github.com/settings/security
- deleted 15y ago[deleted]
- avar 15y agoCorrect me if I'm wrong but the nature of the vulnerability was that someone who's not you had to submit a page with certain POST variables they could have determined after the fact to be malicious while logged in. So the fact that they're sending out this E-Mail tells us that they either don't keep logs on requests + POST contents, or that they haven't had the time or inclination to analyze this data if they have it.
- chrisguitarguy 15y agoOr they realize that psychology is just as important to security (and customer confidence) as logs, analysis, and good code. Every person that got this email now feels more secure about Github. They audited their own private keys. They were reminded that they can remove keys at will. And they know Github has improved its code and given users more power (email alerts, etc) to be in control of content.
- avar 15y agoI for one am feeling way less confident in them with every announcement they make. They clearly have no idea if and how they were exploited, and their communication with their users only asks their users to check for one attack vector, while in actuality the attack was not limited to just adding ssh keys.
- maratd 15y ago> So the fact that they're sending out this E-Mail tells us that they either don't keep logs on requests + POST contents, or that they haven't had the time or inclination to analyze this data if they have it. No. Github is primarily a B2B company. They're not making their big bucks off of individuals. Businesses understand that problems arise. What they want to see is immediate action taken to rectify the problems. Business 101. Even if the problem can be easily fixed by flipping a switch on your end that the customer never has to know about, always show the client "you did something to fix the problem". This is an in-your-face-we-are-taking-charge action. Even though it is completely unnecessary from a security standpoint, it is necessary from a business one. They get it.
- 15y ago
- homakov 15y agoIs it a good idea to check created_at != updated_at ? People update public keys very rarely. I would even say NEVER. Just make an sql against your table to see what are the most possibly are malicious keys. (i see no reason to update timestamps doing 'the trick'. I believe attackers didn't)
- aparadja 15y agoThe vulnerability probably also allowed the attacker to edit the created_at and updated_at columns in the ssh key table.
- benatkin 15y agoIt was easier for me to just delete all of the keys. I had some I didn't need anymore. I also didn't pick great names for the keys I had. It's easy to add a key so instead of checking the fingerprints I can just create a new key.
- joshklein 15y agoSeveral comments below praise the Github team response to this vulnerability. I agree. But it should also be mentioned that the first email I sent to my company this morning read, "should [our product] source code be in the cloud?"
- rdl 15y agoAt the very least, I think it is irresponsible for your product's source code to ONLY be in the cloud. Luckily git provides an easy way to keep a mirror (it's kind of automatic), but some kind of regular off-github backup, signing, etc. would make a lot of sense.
- joshklein 15y agoThis is a vulnerability where an attacker would be able to add his SSH key to a private repository and pull proprietary source code he was not authorized to see. This is why we pay for Github, not use a free account. We don't want people to be able to walk off with the intellectual property of our company. That being said, we don't have the resources to deploy a more secure alternative without hamstringing our development capabilities (e.g. no internet connectivity).
- rdl 15y agoOh, I agree. I'm just saying the baseline for everyone should be backups and integrity checking. A lot of companies don't have great value in their code remaining confidential, a lot do, so that has to be factored in. Other infrastructure (including runtime environments/hosting) need to be factored in, too, and there's confidentiality plus a lot of other concerns like availability. Why isn't Github Enterprise an option for you? Too expensive? (plus of course you have to run it; if you don't have a good VPN or premises network, sysadmin resources to run it, etc., it's entirely possible a self-hosted thing could be less secure than a SaaS solution) (The irony of my running a cloud tech startup and not trusting "the cloud" for our source control, email, file storage, compute, ... is not lost on me. It definitely adds costs, but I think this is an appropriate level of paranoia. The providers of business services need to provide convincing arguments why their services are secure enough to use, at least for b2b.)
- ricardobeat 15y agoDid this change just disable re-use of deploy keys across multiple repos?
- my8bird 15y agowhile this was a good response to their security issue a little heads up would have been good. they broke all of our auto builds and by the time we figured it out the guy who's key was used for the builds was gone on his vacation. luckily, we got ahold of him prior to him turning his phone off.
- skrebbel 15y agoDamn I envy the GitHub guys. They can send a mail to their users about SSH Keys and nearly all users simply understand it and get it over with. In any other business, the result of a similar mail would be an overloaded helpdesk, a significant reputation hit and a massive bucketload of competitor FUD.
- tomjen3 15y agoWhy are you guys praising GitHub? They basically screwed up thrice: first by not catching such an obvious flaw (granted it should have been changed in Rails, but still), second by breaking half the scripts that rely on their service and finally by sending such an obnoxious email (really required action? Who the hell to do you think you are?). Anyway it is pretty moot at this point since I have long ago forgotten my password and changing the orgion to somebody else is pretty easy. That said, can anybody recommend alternatives? I know Bitbucket and they seem pretty great, especially as they allow private repositories, but it seems the consensus here doesn't like them for some reason?
- unoti 15y agoI love Bitbucket and have been using them exclusively for a long time. I have many private repos histed there for free. I have several private repos there shared between a few users, also free. I'm unnerved by how they havent asked me for money, but grateful. Free jira, wiki, and hipchat, too. I'm a Mercurial guy, but they do git too.