13 ms·
> Here’s how the hack went down: Two attackers accessed a private GitHub coding site used by Uber software engineers and then used login credentials they obtain
by rickcnagy 9y ago
> Here’s how the hack went down: Two attackers accessed a private GitHub coding site used by Uber software engineers and then used login credentials they obtained there to access data stored on an Amazon Web Services account that handled computing tasks for the company. From there, the hackers discovered an archive of rider and driver information. Later, they emailed Uber asking for money, according to the company.
Don't check secrets into VCS, folks!
- claudiulodro 9y agoI'm surprised Uber doesn't have their engineers set up 2FA for GitHub. Super simple to implement and require organization-wide[1] and would have prevented this. Then again, not storing credentials in GitHub would also have prevented this . . . [1] https://help.github.com/articles/requiring-two-factor-authentication-in-your-organization/ https://help.github.com/articles/requiring-two-factor-authen...
- lhorie 9y agoGithub 2FA has been part of the first-day training/laptop setup for a while now (I joined in may) and there's security-related training in place as well. I was told there are also scanners in place now that check repos, gists, etc for secrets for exactly this type of mistake. One snippet of the email the article didn't mention was that Sullivan's firing happened pretty much right after Dara learned of the breach and an investigation was conducted. It definitely inspires more confidence in leadership seeing that the CEO will not tolerate unethical behavior.
- perfmode 9y agoits possible and even likely that this happened post hack.
- lhorie 9y agoTrue, I just wanted to shed some light into the current state of affairs in here.
- msh 9y agoUber will not tolerate unethical behavior, you got to be joking!?!?
- brianpan 9y agoI think the commenter meant the new CEO will not tolerate unethical behavior.
- ibmthrowaway218 9y agoThe new CEO will not tolerate new unethical behaviour. Hopefully he will also slowly eradicate the existing unethical behaviour.
- katastic 9y agoThe new CEO will fix everything just like the last 3 GM CEO's changed their corporate culture and stopped them from making cars that kill teenagers... ... crap. My kids won't be buying a GM car.
- pc86 9y agoThe downvotes are likely because you're taking an Uber thread veering it off to GM's management and your children, neither of which have any relevance here.
- katastic 9y agoExcept for the CEO being changed and having a toxic corporate culture that didn't change and produced the same deadly car across CEO's after promising change but did nothing different--including not stopping production of a deadly vehicle. I probably should have spoonfed the readers more. They grew up in a world that doesn't need critical thinking anymore so it's probably too much to ask for their brains to activate while reading on a website and have them put distinct ideas together to form a grander one. Must. Downvote. Comments full of facts but from people I dislike. Must.... errooorrrrroooorrrrrr. 505. It's okay. Every time I see downvotes here, I know I said something great but I just pissed someone in power off. I'm used to being a minority oppressed by a majority in power. It's no big deal. The system just builds people like that these days.
- sAbakumoff 9y agoAre you using Github Enterprise? Is it available from outside of the uber network?
- lhorie 9y agoWe primarily use private phabricator and gitolite instances for internal stuff, but we also have OSS things in regular public GH repos. We do have a few private GH repos, but AFAIK, you're not supposed to version control internal stuff on GH, and there's no real reason to use a private GH repo, except for legal review prior to open sourcing. I don't have any context on why someone would have put production secrets in a GH repo. If it had happened in my team, I would definitely have sounded the alarm at code review.
- johnhenry 9y agoJust curious, are you speaking as an Uber employee
- lhorie 9y agoWell, I am one, but the things I say here are my individual opinions and observations. I just think that as an insider I get some insights that you'd normally not get from the media, and I figured I'd share them.
- ggggtez 9y agoI'll believe it when they stop having stories like this every few months. They had over a year to report the breach, and they paid hush money instead. Typical Uber
- lhorie 9y ago> They had over a year to report the breach, and they paid hush money instead. Yeah, I'm totally with you there. Not cool :(
- igorgue 9y agoHah, setting the example himself I remember him yelling / and cursing at an Uber driver in NYC, very ethical. Good luck and I hope you're doing it for the money, cause nobody should buy the "Uber is an ethical company" bs.
- slackingoff2017 9y agoFound the newest marketing hire...
- dmoy 9y agoWorking at another large tech company, this does not surprise me. Edit: I mean it would surprise me if it wasn't recommended practice, but it would also surprise me if it was somehow strictly enforced.
- komali2 9y agoThis is so gob-smackingly uncommon I started asking "do you require 2fa for your github accounts" as part of my interview questions when I was looking for jobs (i.e. I'd ask my interviewers). I don't know how to feel knowing that there is even one software-focused company out there that doesn't enforce 2fa on its github accounts. Like... how?! Why?!
- 21 9y ago2fa is just another hurdle. Good to have, but by no means a silver bullet. Just one of the many ways to bypass it in this case: hack a developer machine and look at the local checkout.
- giobox 9y agoI really don't think using 2FA and the direct hacking of an individual developer's machine are all that comparable here. Who cares about access to individual dev's machines if the credentials to access code on github are obtained - 2FA at least offers some degree of protection in this scenario. The scope for attack is extremely different.
- 21 9y agoThe hackers wanted access to the code to look for Amazon keys. For them it doesn't matter if they get the code from the internal GitHub or from a developer machine. If you have an ultra-secure door, the thiefs will just enter through your regular window.
- IncRnd 9y agoHow do you know they "wanted" access to look for Amazon keys? Do you know it wasn't from a blanket scan of github? Sure, there are only 13 projects on https://uber.github.io/ https://uber.github.io/, but there are 169 on https://github.com/uber https://github.com/uber, and it only takes a short while to scan for access keys. There are plenty of open tools that will scan github for keys. This may not have been targeted at Uber but a net for all of github with Uber being just one company that was hit up for cash. Unless you're saying that you know the motivations of the attackers.
- dboreham 9y agoUnless 2fa was bypassed with the token you get from GitHub in order to use the git client via https.
- flylib 9y agoI mean they don't say how they accessed the GitHub repo or whether there was a vulnerability in Github itself that allowed access
- claudiulodro 9y agoI assume it was password reuse from one of their engineers or something similar. If you could compromise GitHub itself there would probably be higher value targets (source code for upcoming AAA games, Coinbase, government organizations, etc.)
- flylib 9y agoI mean 100k is a lot of money and there is no saying they didn't hit those guys also
- thaumasiotes 9y ago> If you could compromise GitHub itself there would probably be higher value targets (source code for upcoming AAA games I'm intrigued. Why would that be a higher-value target?
- noarchy 9y agoThe most I've ever personally seen a company do is require a VPN for their privately-hosted repos. For others using GitHub or Bitbucket? Never anything beyond a standard login.
- tmh79 9y agouber engineer here, we have 2fa set up for everything. Starting my day takes about 5 different 2fa checks (ssh access, aws, phabricator, team chat, etc)
- claudiulodro 9y agoI know Uber has a strong engineering culture, which is why I was so surprised. I think philsnow's assessment that organization-wide required 2FA wasn't available for GitHub Enterprise at the time of the hack is probably correct.
- deleted 9y ago[deleted]
- HaoZeke 9y agoThat sounds really inefficient
- giosch 9y agoThat sounds reasonably secure and quite common for a big tech company.
- davb 9y agoAlthough more and more applications support SAML for SSO, much of the SaaS world is disparate and siloed. There's definitely something to be said for centralised user management on a homogeneous system. User leaves your organisation? Just retire them in LDAP.
- philsnow 9y agoYou couldn't enforce 2FA on GHE for the longest time. GHE version 2.8.0 lists [0] "Enforce two-factor authentication" as a feature. 2.8.0 was released November 2016. According to the article, > Kalanick, Uber’s co-founder and former CEO, learned of the hack in November 2016, a month after it took place, the company said. I don't know if they were using GHE. If they were, at the time it did not come with a good way for them to enforce 2FA for users. [0] https://enterprise.github.com/releases/2.8.0 https://enterprise.github.com/releases/2.8.0
- kelnos 9y agoIf they are/were using GHE, I would expect (hope?) that they require some sort of VPN to get access to it, so my guess would be this was stored on github.com.
- chimeracoder 9y ago> I don't know if they were using GHE. If they were, at the time it did not come with a good way for them to enforce 2FA for users. Well, sort of - at the application level, that's true, but GHE is typically run behind a VPN. Certainly that should be the case for a company the size of Uber. Even before GHE added 2FA, it shouldn't have been possible for a leaked set of login credentials to be used to access GHE, without some other sort of compromise (VPN cert, physical compromise of hardware, etc.).
- uxp 9y agoAt my company (mostly a Windows and Microsoft shop), my domain credentials are used to log into the VPN, and TFS, and Octopus. Compromising just that one set of credentials could effectively "own" our company. And I'm just a senior-ish developer. Lateral movement by an attacker is a real thing. And while credential reuse is something most security focused web companies are trying to mitigate, a push for "sso"-like account management is seemingly undoing most of that effort inside the network if not done properly (specifically, auditing and monitoring of behavior).
- deathanatos 9y ago
- benwilber0 9y agothat doesn't protect you from GitHub employees snooping around.
- mintplant 9y agoOr anyone who manages to breach GitHub's defenses.
- colejohnson66 9y agoCouldn't you say the same thing about any commercial web platform? Like AWS?
- benwilber0 9y agoyes of course
- ctennis1 9y agoYep, but think of all of the private keys and tokens used in automation servers (think CI) for pulling down source. Those don't have 2FA - because they don't login - but they have full access to most source. In an organization of about 200 engineers across various products, 1000+ github repos, and 10 or so different CI systems. We enforce 2FA at github. I can still easily see how someone could easily gain access to source code with secrets in it.
- amazingman 9y agoThis is almost certainly what actually happened.
- ryandrake 9y ago> In an organization of about 200 engineers across various products, 1000+ github repos Wait, what? That's 5+ repos per engineer. What on earth would warrant that level of granularity? I've only worked once in my career in a place that used more than 2-3 repositories total, and that was a "MegaTechGiant" with thousands of engineers.
- gjjrfcbugxbhf 9y agoIt depends upon the culture. Some places favour a project repo others a repo per microservice/job.
- gutnor 9y agoCould also be a company using clone - pull request workflow. 10-20 project repo an then each developer has a bunch of projects clones, including a few shared one - like the common infrastructure stuff, ... I can see that with a company that has grown day 1 around Github, especially during early startup stages with a variety of contributors but no formalised "organization".
- 6d6b73 9y agoThere are just three of us in my company and after 10 years I worked on close to 80 projects for 30 different clients. Each project has its own repo. So +3 per engineer is really not that much;)
- xkcd-sucks 9y ago2FA doesn't help if they used SSH access
- rockinghigh 9y agoIt’s also required for SSH access to Uber’s servers.
- aaronduino 9y agoMaybe it's just me, could "private GitHub coding site" have meant a private GitHub repo with GitHub pages turned on? If that were the case, there would be no authentication whatsoever to access the closed-source site; the hacker would have just needed to guess the right url.
- Buge 9y agoDo we know how the attackers accessed the github repo? If it was via malware on the employee's machine, or cookie theft then 2fa wouldn't have helped.
- gejjaxxita 9y ago2FA wouldn't have necessarily solved this, if the hackers had access to an engineer's ssh keypair (e.g stolen laptop) they could clone repos as they pleased. 2FA isn't a silver bullet.
- ihattendorf 9y agoCould use a Yubikey (or similar) for SSH access.
- lawnchair_larry 9y agoNo, 2FA would not have prevented disclosure of credentials in GitHub. The fix for that is to not check credentials in to GitHub. Nothing else.
- gideon_b 9y agoTwo factor won't protect you from a spear-fishing attack. The attacker can submit your info to GitHub the moment you submit to the malicious site. You receive the token via SMS as expected, enter it on the second page of the malicious site, granting them access.
- Toast_25 9y agoFrom what I hear it's pretty common...
- shallot_router 9y agoIt's very common, but there are lots of ways of addressing it.
- missing_cipher 9y agoJust pigging-backing on your comment. If you did, here's a guide from Github on how to remove it: https://help.github.com/articles/removing-sensitive-data-from-a-repository/ https://help.github.com/articles/removing-sensitive-data-fro...
- PeterisP 9y agoThey key part is "Warning: Once you have pushed a commit to GitHub, you should consider any data it contains to be compromised. If you committed a password, change it! If you committed a key, generate a new one." Removing the secrets from the repository is nice to have, but not that necessary - what is mandatory is to ensure that the compromised secrets are no longer useful, since they aren't secret any more and won't be ever again.
- munk-a 9y agoI am rather disappointed in github for publishing this guide. The portion at the top stating > Warning: Once you have pushed a commit to GitHub, you should consider any data it contains to be compromised. If you committed a password, change it! If you committed a key, generate a new one. Is a good argument as to why you shouldn't let users erase this data from history, it's already out there so no matter how painful or convoluted your process is for regenerating auth credentials is, you need to do it if you've published them into your SCM. If the process is painful you might want to simplify it because you'll probably need to do it sometime in the future again... yes even you large corporate workers who have no control over credential regeneration, an arduous process leads to credential sharing between projects which is another horrible thing.
- tedivm 9y agoThey are doing the right thing by letting the users control their own data, and at most they can make it more complicated to do but not impossible. There are cases- such as complying with court orders- where removing the data is appropriate (even if a bit futile in the long run).
- matt4077 9y ago
- nikcub 9y agogit-secrets is a pre-commit hook that regexp's out secrets and blocks commits https://github.com/awslabs/git-secrets https://github.com/awslabs/git-secrets
- deleted 9y ago[deleted]
- kfrzcode 9y agoThings like this make me feel much less concerned about the confidence gap.
- Garbage 9y agoYou can use tools like Talisman which registers a Git hook to check if you are checking in anything that looks like secret. https://github.com/thoughtworks/talisman https://github.com/thoughtworks/talisman
- anaphylactic 9y agoIt's not foolproof but this tool needs to be more widely-known - it would've saved me on countless occasions.
- lhinds 9y agoWe use a tool under a Linux Foundation project called anteater https://github.com/opnfv/releng-anteater https://github.com/opnfv/releng-anteater, which does the same thing (but is for a jenkins / gerrit workflow). A key difference from looking at talisman, is anteater uses standard RegEx rather then code to seek out strings, so anyone can add their own strings / file names easily into a simple yaml file. Like wise they can use regex to provide a waiver, should something be incorrectly reported. I am thinking now would be a good time to port it to working with webhooks as well. The tool would have blocked the aws credentials from being checked in: https://github.com/opnfv/releng-anteater/blob/master/master_list.yaml#L55 https://github.com/opnfv/releng-anteater/blob/master/master_...
- ringaroundthetx 9y agoI'm not even mad, thats a good bug bounty
- navinsylvester 9y agoTotally agree. I am moving secrets onto consul/vault - would like to hear what others use for the same.
- celim307 9y agoDumb question: What's the best practice to share authentication credentials across the team for services that don't have an IAM feature?
- T-Winsnes 9y agoThere are a few SaaS offerings that will let you do that. LastPass or onepassword are two commonly used. One you can use something like keypass to store a database in a shared location if you don't trust the SaaS offerings. If you are looking at storing credentials for automation purposes, and don't have a secret store built in, you could look at something like Hashicorp Vault to help provide this for you
- dzhiurgis 9y agoLastPass has a terrible track record in security, that was nicely edited out from wikipedia by a fresh user: https://en.wikipedia.org/w/index.php?title=LastPass&action=history https://en.wikipedia.org/w/index.php?title=LastPass&action=h... The user in question has some specific interest in editing LogMeIn, parent of LastPass, pages: https://en.wikipedia.org/w/index.php?limit=50&title=Special%3AContributions&contribs=user&target=FrankTursetta&namespace=&tagfilter=&start=&end= https://en.wikipedia.org/w/index.php?limit=50&title=Special%...
- vor0nwe 9y agoWe use 1password for teams.
- M4v3R 9y agoWe're using Keepass / MacPass password protected vault shared with the team using Dropbox. It's really good and essentially free to use if you use a free Dropbox account.
- hollander 9y agoThen make sure you use 2FA on the Dropbox account. And you should use a key + password to unlock keepass.
- Cthulhu_ 9y agoBut you have to put them somewhere; how is idk, AWS credential management secured?
- tskaiser 9y agoStore credential information where it is used. It is not used by the repository, so it is an improper location for it. If someone gains access to a system that uses the credentials, then there is, in principle, no difference between puppeteering that system versus stealing its credentials.
- davidumoh 9y agoReally surprising to see that sensitive credentials were checked in to VCS. Apart from peer code review, how can a company avoid developers checking in sensitive data to VCS?
- rplnt 9y agoYou could have a git hook (even remote) that would check for pre-configured patterns and reject the push if it contains them. Quick google yielded this https://github.com/awslabs/git-secrets https://github.com/awslabs/git-secrets
- selvakn 9y agoPlug: https://github.com/thoughtworks/talisman https://github.com/thoughtworks/talisman
- ransom1538 9y ago"Don't check secrets into VCS, folks! " I suppose? But at this point they have your code base. You are so owned at that point.
- diggan 9y agoYeah, but hopefully they can't do much if they just have your code base. If the secrecy of your code is the only thing stopping hackers from exploiting you, you're missing some gaping holes in your infrastructure. With that said, nothing wrong with using secrecy as a additional barrier, but shouldn't be the only, and if it's not the only, you're not "so owned at that point".
- ransom1538 9y ago"If the secrecy of your code is the only thing stopping hackers from exploiting you" I hate these types of arguments. Yeah no one said that ever. Losing your code base is terrible. I view it as losing a journal. What your company tries, tests you run, funny comments, or funny mistakes. I mean they post it on the net, blackmail team members, imposter team members, forge for leaks, sell it, pushes to prod from compromised accounts, CI systems, -- seems bad to me. Sure don't have aws keys in there.
- diggan 9y agoGlad to be talking with you too! :) I didn't mean to imply you said something you didn't, only that I would consider access keys to various services be of much more importance the code base itself. I read you comment as "Doesn't matter about the access keys, if they have your source code, you're screwed no matter what", which in that case would seem a bit strong. Also "pushes to prod from compromised accounts, CI systems" seems more related to access keys and account security rather than the actual code base. But hey, in the end I'm no security expert so what do I know.
- toyg 9y ago“Just” leaking full source could be enough to destroy a lot of IP-based companies. A lot of companies stay wealthy because their IP is so huge than nobody can afford to develop competitive alternatives anymore (Adobe, Microsoft Office, Salesforce etc). Some of them have actual “secret sauce” that they cannot afford to share (suggestion engines, biotech processes etc). Even a service like Github, which relies on others entrusting their work to them, would take a humongous reputation hit from a leak like that.
- notyourday 9y ago> Don't check secrets into VCS, folks! Ok, how do you handle a bootstrap problem?
- timsayshey 9y agoI really wish AWS would stop enabling master API keys by default. As soon as you create an AWS account you are given API keys which basically have SUDO permissions to your entire account. That is super dangerous and is probably the same key set that these hackers got ahold of. AWS needs to disable these full access API keys by default and instead should encourage users to generate keys for specific access to limit what they can do.