12 ms·
GitHub mirror compromise incident report
- Belphemur 8y agoMirror is the important word here. I'm pretty sure it didn't impact anybody.
- Operyl 8y agoReally it was just a major inconvenience to the contributors preferring to use GitHub to merge their work in. There was also a repository that was only on GitHub, but I think one of their action items is to change that. Overall, nothing overly bad indeed.
- anemic 8y agoAlso the type of the malicious content (rm -fr /*) is "easy" to spot in which I mean there are no rootkits or remote exploitable flaws added. Not very nice to be a victim of that but at least there is no doubt whether you're vulnerable or not. The timeline also suggests that the malicious content was made after the break-in and not planned beforehand?
- akerl_ 8y agoFrom the incident report it sounds like the gentoo/systemd GitHub repo is not a mirror: the official repo is on GitHub and they're now looking to mirror it onto Gentoo infrastructure. Does anybody know why this is the case?
- noobermin 8y agoIt probably just happened that way. To note, systemd isn't as popular/required for Gentoo as most people I know still use openrc. There is even a set of hacks (that I very thankfully use!) to use Gnome 3 on a non-systemd system[0]. [0] https://github.com/dantrell/gentoo-project-gnome-without-systemd https://github.com/dantrell/gentoo-project-gnome-without-sys...
- Semaphor 8y ago> The main Gentoo repositories are kept on Gentoo hosted infrastructure and Gentoo mirrors to GitHub in order to "be where the contributors are." Doesn't sound to me as if it's not a mirror. Though I guess you could say as their CI depended on this mirror, it had a higher status than normal mirrors.
- akerl_ 8y agoI'm specifically talking about https://github.com/gentoo/systemd https://github.com/gentoo/systemd From their incident report (under https://wiki.gentoo.org/wiki/Github/2018-06-28#What_went_badly https://wiki.gentoo.org/wiki/Github/2018-06-28#What_went_bad... ): "The systemd repo is not mirrored from Gentoo, but is stored directly on GitHub." And from their Action Items: "mirror systemd repo on git.gentoo.org" I'm curious why they seem to treat this repo differently from the others, using GitHub as authoritative and adding a mirror to git.gentoo.org rather than making git.gentoo.org authoritative and mirroring to GitHub.
- deleted 8y ago[deleted]
- LeoPanthera 8y agoActual link to full report: https://wiki.gentoo.org/wiki/Github/2018-06-28 https://wiki.gentoo.org/wiki/Github/2018-06-28
- sctb 8y agoThanks, we've updated the link from https://lwn.net/Articles/759046/ https://lwn.net/Articles/759046/ which points here.
- akerl_ 8y agoSomething to note here: "Evidence collected suggests a password scheme where disclosure on one site made it easy to guess passwords for unrelated webpages." For any folks out there who've been using or promoting formula-based passwords, this is the potential impact: a leak on one site can be leveraged by an attacker towards other sites you use.
- simias 8y agoI suppose it's only as good as the formula itself and at least it protects you from automated attacks. Still, I wouldn't advise anybody from going with formula-based passwords, just use whichever password manager you prefer and use unique strong passwords per-website. Maybe one day the web will get its shit together and let us login everywhere with a certificate that we'll be able to securely store on a HSM and easily revoke and update if necessary but in the meantime it's the best we can do.
- akerl_ 8y agoMost of the day-to-day issues with certificate auth aren't actually problems with "the web", so much as they're broader user workflow or endpoint support issues. Here's a couple: * How does a user log in from a new device? * Especially if they've lost / broken their original device? * What happens if users want to log in from a shared computer (think public library)? * Do all the OSes/browsers that users are using actually support certificate management and auth?
- philjohn 8y agoHence using an HSM, e.g. something like a Yubikey. Of course, they can be lost or stolen, which is why certificate revocation is necessary, and they would need broad adoption among OS and browser vendors. Plus you still have the issue of proving your identity if it's lost/stolen.
- akerl_ 8y agoHardware tokens don't really solve any of the problems I listed. You still have to deal w/ devices that don't do cert auth from a Yubikey (iOS being a great example), so as a site operator you don't get to avoid supporting new device / temp device workflows. Hardware tokens can allow individual power users to solve issues around multiple devices, lost devices, etc themselves, but unless you're suggesting porting 100% of users to hardware tokens, it doesn't change the workflows a site must support.
- rich-and-poor 8y agoHe should have sat on the password. He should have watched for PRs, and started pushing updates immediately after they had received approval, and then merging. Instead, he panicked and kicked out all the maintainers, who realized the intrusion only 10 minutes after he gained access. And all he did was add `rm -rf /*` to build scripts, and the N word to the readme. The malicious commits: https://github.com/gentoo/gentoo/commit/e6db0eb4 https://github.com/gentoo/gentoo/commit/e6db0eb4 https://github.com/gentoo/gentoo/commit/afcdc03b https://github.com/gentoo/gentoo/commit/afcdc03b https://github.com/gentoo/gentoo/commit/49464b73 https://github.com/gentoo/gentoo/commit/49464b73 https://github.com/gentoo/gentoo/commit/fdd8da2e https://github.com/gentoo/gentoo/commit/fdd8da2e https://github.com/gentoo/gentoo/commit/e6db0eb4 https://github.com/gentoo/gentoo/commit/e6db0eb4 https://github.com/gentoo/gentoo/commit/c46d8bbf https://github.com/gentoo/gentoo/commit/c46d8bbf https://github.com/gentoo/gentoo/commit/50e3544d https://github.com/gentoo/gentoo/commit/50e3544d
- noobermin 8y agoHe was likely not a person trying to do serious damage more than trying to have fun.
- justinclift 8y agoPutting "rm -rf /*" in every ebuild seems like a pretty clear indication of malicious intent. Can't really picture anyone doing that as a "just trying to have fun" thing.
- giancarlostoro 8y agoforgot --no-preserve-root for it to really take effect properly.
- ryan-c 8y agoI believe rm -rf /* will work without --no-preserve-root
- driverdan 8y agoAlways require 2FA for something as important as your code base.
- noobermin 8y agoThis wasn't the code base (save the systemd stuff which increasingly is not being used by gentoo users), it was merely a github mirror.
- akerl_ 8y agoIt was the codebase. It's mirrors, but if the attacker had been less subtle the code have pushed smaller changes instead of force-pushing, anybody using the mirror would have pulled those changes down. Suggesting that this is "merely" a mirror downplays how seriously a more sneaky attacker could have harmed users of the project. Would similar statements downplaying the attack be made if this had been https://mirrors.kernel.org/ https://mirrors.kernel.org/ ?
- Hello71 8y agoactually it would be much less bad if it was. Gentoo GPG verification still doesn't work quite right, but most users of the kernel for critical purposes do GPG verification.
- swsieber 8y agoThe wouldn't have even had to push smaller changes - there are ways in git to do a commit that basically overwrites the current state of the branch, but doesn't require a force push. We use it where I work when promoting branches (e.g. to production branch) so that we never ever have to worry about merge conflicts or force pushing. It's a merge strategy that basically says "When merging, just use parent 1, and pretend parent 2 doesn't exist when calculating files, etc."
- deleted 8y ago[deleted]
- berkes 8y ago
- badrabbit 8y agoAre there any generic incident response plans or playbooks for foss projects? Is gentoo's documented anywhere? Gotta say,a bit impressed by the response.
- tetha 8y agoIt's pretty hard because that's very generic. The security workflow we have for a couple of customers essentially has these steps: 1. Identify a security incident. This is the hardest one usually, unless they are blunt and noisy like in this case. 2. Shutdown and firewall everything involved in the security incident and shutdown and firewall everything that has a high probability to be immediately involved or targeted. - For example, if you have a cluster of several identical application servers and one of these application servers is compromised, all application servers of the same kind in the same cluster must be shutdown and firewalled to the VPN. - identical application servers in other clusters are handled as per 2.1, but with less hesitation to react strongly. - If you have a database only available inside a private network and you have identified suspicious activity, it is reasonable to shutdown + firewall everything with access to this database. 2.1 Increase attention similar systems or connected systems in case of lateral movement. - For example, our productive clusters have access to our monitoring setup. If a cluster is accessed in a creative way by a user, we'll revoke the credentials and certificates of this cluster. However, an attacker might have obtained information and utilize that to pivot his attack onto the monitoring cluster or the management infrastructure, so we need to pay attention to that. 3. Communicate to customer support and management about the incident. - Yup. This is intentionally after axing productive systems. If senior persons are sure about a security incident, we don't want to wait for management to kill the malicious access. We want the senior guys to shutdown the security incident and ask for forgiveness later. Fast isolation beats bureaucratic process. 4. Identify vector of entry and eliminate it. - This is normal post-work. What did they do, how did they come in, patch or report. This tends to be split into immediate mitigation (in gentoos case, change of passwords and implementation of 2FA, or disabling software features in an incident we had some time ago) and further mitigation down the road, such as the audit logs they are planning. 5. Resume operation of service coordinated with customers. - This is something we learned some time ago. Some of our customers have their own dedicated systems, and from there they have a word how to properly resume operation after an incident. In one case, one department of a customer had an unannounced penetration test against a system without coordinating with the security department of the customer and without coordinating with us. All of that went to hell and back. We were forced to leave the system down for almost a month due to contractual bindings and catfights between teamleads of that customer on our bug tracker. That was fun.
- noobermin 8y agoSmall piece of annoyance, the clickbait press had a field day with this. From the positive which I appreciate[0] to claiming putting `rm /*` in the wrong place so that it didn't even work is "file wiping malware."[1] For the most part, this appeared to the work of a teenage skiddy (given the addition of a readme with a racial slur as the text), and not any actual sophisticated attack. [0] https://www.techrepublic.com/article/gentoo-stops-github-repo-hack-in-an-hour-setting-standard-for-security-response/ https://www.techrepublic.com/article/gentoo-stops-github-rep... [1] https://www.bleepingcomputer.com/news/linux/file-wiping-malware-placed-inside-gentoo-linux-code-after-github-account-hack/ https://www.bleepingcomputer.com/news/linux/file-wiping-malw...
- petters 8y agoGithub supports U2F (two-factor auth). Please use it.
- fulafel 8y agoGithub's SMS or TOTP requirement for u2f use is a shame - why would you use u2f if you have already set up these? And if you want to use u2f because you don't trust your phone as a security device (hello Android users) - not good. Or maybe you don't want to mix your personal phone with work it infra, it's a bad equation there too. I guess you could still use a software TOTP implementation on your workstation to fool Github, but then you are not getting the additional security from u2f because the totp codes are a substitute for the u2f token.
- leni536 8y agoWhat if you just throw away the TOTP secret right after you get it and verify it on Github? You don't have to use your phone either. Just because you used TOTP to set up u2f on Github doesn't mean you have to store the TOTP secret indefinitely.
- fulafel 8y agoYes, I guess you could navigate it this way. But clearly this is not something Github wants users to do, so I wonder if this way is bad somehow. Maybe there is no backup mechanism to recover from u2f token loss other than the old 2fa mechanisms, for example.
- Ajedi32 8y ago> - action-item: review 2FA requirements for GitHub org > - done: Gentoo GitHub Organization currently requires 2FA to join.
- snowwolf 8y agoAs they have now done, they could have reduced the threat exposure by requiring 2FA to join the organisation, and using a password pattern scheme does expose you to a targeted attack so ideally encouraging password manager generated passwords is probably also recommended. But Github could probably have helped by detecting logins from unusual IPs for that user - i.e. a login attempt from an IP they haven't logged in from before, and required something like email verification too. Although if they were using an easy to guess pattern, then likely the admins email could have been compromised too. Edit: GitHub could also warn people whose accounts have admin access to organisations if they don't have 2FA enabled.
- gkoberger 8y agoTo be clear, GitHub itself was NOT compromised. The password of a Gentoo contributor, who presumably wasn't using 2FA, was. The title should probably clarify it's a "Gentoo mirror on GitHub compromise incident report"
- lucb1e 8y agoI too thought the title was about one of Github's CDN sites or something.
- keithnz 8y agoTitle is super misleading.. should be something like Incident Report for a Gentoo GitHub contributor account being compromised
- SquareWheel 8y agoGlad I clicked through. When I scanned the Hacker News titles earlier, I'd assumed Github had been compromised. Very misleading.
- PretzelFisch 8y agowhat mitigations do you use to protect systems that automatically updated after every git push?
- yjftsjthsd-h 8y agoWould GPG signatures suffice?
- badrabbit 8y agoDon't auto-update production like that? If a change will affect all users,manually review it and as another commenter said,gpg signatures. Have every prod commiter use mfa too.
- PretzelFisch 8y agoI have seen a lot of people describe their continuous delivery where the code is checked in and moves into production after passing tests. I wasn't sure if they had a way to deal with this type of issue or I am misunderstanding their process somehow. I
- Thaxll 8y agoNot enabling MFA is just unacceptable at that level...
- deleted 8y ago[deleted]
- taf2 8y agoGithub supports two factor authentication, maybe make it mandatory for all admins?
- lousyd 8y agoWhaddya expect from a Microsoft product? Just saying.
- mattaco 8y agoGitHub was not at fault. Someone gained access to an account using a disclosed password scheme. Also, the Microsoft acquisition of GitHub hasn't happened - they have only agreed to acquire GitHub.
- jrochkind1 8y ago> Clearer guidelines that even if users got bad copies of data with malicious commits, that the malicious commits would not execute. Eh, that sees unwise. To suggest based on what you think you are "sure" about security protections, that users should not worry about having a copy with malicious code.
- zenexer 8y agoI'd sleep better at night if every organization were this transparent.
- imdsm 8y agoNo 2FA?