12 ms·
PHP's Git server compromised, moving to GitHub
- jsmith45 6y agoNote: I did not keep the original title (Changes to Git commit workflow), because it sounds mundane, rather than being related to an apparent compromise of the git server of a high profile open source project.
- jnwatson 6y agoThis looks like the work of either a talented but neophyte black hat, or an intentional burning of an exploit.
- EvanKnowles 6y agoMy PHP is poor, but it looks like it was being changed to check for a string coming in via a "typo'd" HTTP_USER_AGENTT header, that started with zerodium, and then eval the string without the zerodium prefix on the server. Reminds me of that Linux compromise years ago where they picked it up because the change was directly on the build box, except at least that one was hidden behind an apparent `=` vs `==` "bug" - this one has a bit less style.
- oaiey 6y agoIt is C not PHP in the linked diff :)
- djxfade 6y agoIts not actual PHP code, it's C, which PHP is written in
- EvanKnowles 6y agoAh - that explains a lot, I was expecting to understand a lot less of it ;)
- kasperni 6y agoRun your own infrastructure. It will be fun, they said.
- obayesshelton 6y agoHaHa I think some point like the challenge until...
- megous 6y agoMoving to github doesn't change anything fundamentally. It just moves infrastructure hosting to someone else (even though it may or may not be a more secure hosting). Something like signing commits, and requiring all commits to be signed properly with a known key (and checking this on push and before release) would be an independent verification layer, and a more fundamental response to the compromise.
- williamdclt 6y agoIt moves infrastructure to someone else _whose core job it is to manage this infrastructure_. I definitely trust GH more than myself in managing a git infrastructure securely
- bogwog 6y agoThat's assuming the self-hosted git server itself was compromised, and it wasn't just a simple matter of the attacker stealing some committer's private keys. If that's what happened, then moving to Github solves nothing. > I definitely trust GH more than myself in managing a git infrastructure securely That's FUD, and appropriate too considering who owns Github now.
- lovedswain 6y agoTrying to sneak that diff in was a hugely wasted opportunity, it sticks out like a sore thumb. If they were going to commit anything, it should have looked like a legitimate change. Another option might have been to wait until just before a release and fiddle with the tags or the newly opened QA branch (if any). Check out https://freedom-to-tinker.com/2013/10/09/the-linux-backdoor-attempt-of-2003/ https://freedom-to-tinker.com/2013/10/09/the-linux-backdoor-... for another attempt, that one at least requires careful review to notice the problem
- thomond 6y agoThey were presumably testing the waters to see if they were noticed first.
- carlio 6y agoIt does beg the question if this was only the first time they were noticed.
- mvolfik 6y agobut why would you test that so obviously ('fix typo' but add code)? and they made it clear that they likely had access to the git server (by pushing 'from' different accounts), which (obviously, if that's what is actually happening) can kill their access
- Ndymium 6y agoIn the diff, could they be hinting that they've had access since 2017? [1] Maybe they just wanted to burn it now for some reason. [1] https://github.com/php/php-src/commit/c730aa26bd52829a49f2ad284b181b7e82a68d7d#diff-a35f2ee9e1d2d3983a3270ee10ec70bf86349c53febdeabdf104f88cb2167961R370 https://github.com/php/php-src/commit/c730aa26bd52829a49f2ad...
- stordoff 6y agoIt does make me wonder if something else was changed and this was what they wanted you to notice (as a diversion).
- aargh_aargh 6y ago> If you are not part of the organization yet, or don't have access to a repository you should have access to, contact me at nikic@php.net with your php.net and GitHub account names, as well as the permissions you're currently missing. Migrating accounts of PHP core developers by asking them for their username over unencrypted email? What could go wrong? A security incident already occurred, so we should be good on that front for a few years, right?
- upbeat_general 6y agoAre github usernames especially sensitive?
- Sebb767 6y ago"I'm xy and had push access to the php core git server, but I do not yet have access to the PHP organization. Could you add @xyfake to it? Thanks!" Combine that with a spoofed/typo'd sender and you're good to go.
- Denvercoder9 6y agoYou can mitigate that pretty easily by checking whether recent commits from this person are linked to that GitHub account.
- Sebb767 6y agoIf the person had a GitHub account before, it could be newly created by the attacker. Or they could have had different e-mails for php.net and GitHub, i.e. something like php@xy.com and github@xy.com. In a perfect world, this would be fine, yes. But when adding work on top of a probably already very stressful workday, the likelihood of this creating additional problems is there.
- Denvercoder9 6y ago> If the person had a GitHub account before, it could be newly created by the attacker Doesn't GitHub require e-mail validation before it associates commits with that e-mail address? > Or they could have had different e-mails for php.net and GitHub, i.e. something like php@xy.com and github@xy.com. Then require them to add their php.net mail address to their GitHub account. > the likelihood of this creating additional problems is there. That I agree with.
- killingtime74 6y agoWow PHP still going huh
- _joel 6y agoOf course, there's a vast ecosystem still running shopping carts, forums and whatnot. It's much more prevalent that you seem to think. Just because it's not cool anymore (or well ever) doesn't mean it's dead.
- jw1224 6y agoThat's a very dated view of PHP. Modern PHP is far "cooler" than it was a few years ago. Just look at the entire https://laravel.com https://laravel.com ecosystem — the most starred web framework on all of GitHub, across all languages, is Laravel.
- fogihujy 6y agoAlso, WordPress. WordPress is pretty much everywhere.
- rocketpastsix 6y agoand there are reasons for that. As bad as it is for a developer experience, the ui for non-tech people just can't be beat.
- fogihujy 6y agoOh, definitely. As a blogging engine it's quite usable and very easy to get used to even for people who are IT-illiterate. I'm not convinced it's the best for building e-commerce sites and whatever people are piggy-backing on top of WordPress nowadays.
- zoid_ 6y agoDiscovering Laravel a couple of years ago was perhaps one of the most significant things that made PHP interesting again after 20+ years of working with PHP. It's refreshing (as far as PHP goes) and worth checking out with an open mind. Performance on the other hand, that take a little more work.
- iBotPeaches 6y agoI wonder if the signed features of GitHub would have helped here. Probably can't ask all the 3rd party contributors to setup signing commits, but it would help to spot commits not signed.
- trulyrandom 6y agoIf the attackers would have used their own name/email linked to a GitHub account and signed the backdoor commits, GitHub would display a green "Signed" label, and everything would look OK to an outsider.
- megous 6y agoThat's why it's just a gimmick. Signing only works as protection against hosting compromise if the signing/verification is separate from the hosting itself. Someone responsible for release would have to manually keep a list of keys of authorized comiters and check the repository against this list at the very least prior to a release.
- kenmacd 6y agoIt's not about the green label. A set if commits signed by some unknown person is a lot easier to spot and clean up. Looking at these commits they say mid-2017. Without signatures any previous commit could be by the same author.
- 6y ago
- Sebb767 6y agoZerodium[0], the magic string searched for in the fake user agent, is a zero-day acquisition platform, in case anyone else is wondering. The third parameter to `zend_eval_string` in the commit [1] is the "filename" [2] of the executed code, which is probably used for stack traces. Which is quite a strange choice, because seeing that in the logs would probably ring alarm bells immediately. Overall, this commit seems strange in its structure IMO - not in a "good" covert way, but in a "why does it not try to hide better" way. EDIT: According to zerodiums charts, this would have been a payout of up to 250k$. However, I'm not quite sure what the attackers plan on cashing out on that would have been, given that the exploit would have quite obviously referenced zerodium and therefore would make the submitter the primary suspect of being the author. Maybe some APT used this "simple" format to disguise the actual power? But then again, it would be in their interest for the exploit to actually go through. [0] https://www.zerodium.com/ https://www.zerodium.com/ [1] https://github.com/php/php-src/commit/c730aa26bd52829a49f2ad284b181b7e82a68d7d#diff-a35f2ee9e1d2d3983a3270ee10ec70bf86349c53febdeabdf104f88cb2167961R370 https://github.com/php/php-src/commit/c730aa26bd52829a49f2ad... [2] https://bugs.php.net/bug.php?id=78224 https://bugs.php.net/bug.php?id=78224 (only mention I found with a documented use of the parameter) [3] https://zerodium.com/program.html https://zerodium.com/program.html
- Denvercoder9 6y agoZerodium say they have nothing to do with this: https://twitter.com/cBekrar/status/1376469666084757506 https://twitter.com/cBekrar/status/1376469666084757506
- oaiey 6y agoQuick migration of toolchain. Curious how that will work out. If it does, my deepest respect to their engineering.
- TazeTSchnitzel 6y agoPHP already uses GitHub and pull requests there, it's just not been the canonical git repository until now. It's not really a big change for most; if anything it's easier.
- rurban 6y agoPerfect time to get your php commit bit now. Just ask:)
- megous 6y agoThis was so dumb. Even mentioning zerodium and skip-ci and sloppy commit message.
- vntok 6y agoThis was not dumb, most probably this was marketing. They chose to burn -very publicly- the two accounts they had access to in order to showcase that they have an in that they can use somewhere else.
- megous 6y agoDunno, they just provoked a hardening and possible review of the remaining infrastructure.
- kenmacd 6y agoIt seems likely it was meant to be seen and fixed.
- cccat 6y agoWas PHP's Git server using a self hosted Gitlab instance? I can't find the past threads, but I thought there was an announcement a while back that they were moving to Gitlab self-hosted. Maybe I'm misrembering which project that was?
- hannob 6y agoIt seems they use some combination of gitolite and their own stuff. The post mentions karma, there's some code here: https://github.com/php/karma/ https://github.com/php/karma/
- cccat 6y agoThanks. That's a bit of a relief, given that we use self-hosted GitLab instances.
- INTPenis 6y agoWell it's still vulnerable. At least in Gitlab you can use fancy features like requiring 2FA for members of a group. But you still need to secure its infrastructure. I think the most common attacks on Github repos and groups have been on individual accounts. Which 2FA would mitigate.
- ynezz 6y ago> Which 2FA would mitigate. Or Rust, right? </sarcasm> https://github.com/php/karma/search?q=exec https://github.com/php/karma/search?q=exec
- T3RMINATED 6y agoGitLab is better.
- zelon88 6y agoMaybe this was someone close to Zerodium burning the compromise against PHPs Git server to bring attention to other zerodium commits dating back to 2017?
- jujodi 6y agoBold move
- synergy20 6y agoAfter learning Django for a few months I decided to switch back to php/wordpress yesterday, php7 and php8 look solid and php remains to be the best language for the web in my opinion, Php8 added JIT which might extend itself outside of web applications. Hope moving to github will free the team to do even more on Php development.
- xutopia 6y agoThere is a reason why some services are worth it... for one you don't have to worry about all those issues that could pop up. It's someone else's problem.
- pabs3 6y agoI wonder if GitHub will ever support requiring signed pushes. It supports requiring signed commits, but that isn't always feasible, since you could want to merge commits signed by an external contributor without rebasing those commits in order to sign them yourself.
- kenmacd 6y agoThat does seem like a really good idea. If I make a protected branch that requires signed commits I really only care the signatures on the commits you'd see with `git log --first-parent`, as those are the ones by people I trust.
- arp242 6y agoI wish git would support signing with tools other than GPG. I hate using it and tools like minisign/signify are a huge improvement. Hell, the entire thing is simple enough to just be integrated in git.
- Corrado 6y agoIt would have been interesting if GitHub had purchased Keybase instead of Zoom. I'm still a bit bitter over that sell out.
- imcotton 6y agocommits history have following: - 8d743d5 Revert "Revert "Revert "[skip-ci] Fix typo""" - 2b0f239 Revert "Revert "[skip-ci] Fix typo"" - 046827a Revert "[skip-ci] Fix typo" - c730aa2 [skip-ci] Fix typo
- tyingq 6y agoPoking around on https://git.php.net https://git.php.net, it appears to be using "gitweb": <!-- git web interface version 2.11.0, (C) 2005-2006, Kay Sievers <kay.sievers@vrfy.org>, Christian Gierke --> "'Gitweb' is a Git web interface. It is written in Perl and can be used as a CGI script, or as a mod_perl legacy script (run by ModPerl::Registry handler). It allows browsing a git repository (or a set of git repositories) using a web browser." - https://git.wiki.kernel.org/index.php/Gitweb https://git.wiki.kernel.org/index.php/Gitweb The last commit looks to be Dec 2020: https://github.com/git/git/tree/master/gitweb https://github.com/git/git/tree/master/gitweb Though I don't know that this was how they got into the git server.
- lucideer 6y agoDisclaimer: wild speculation below There's a strong possibility the vulnerability lay in their hosting setup rather than in the software application itself. There's an argument to be made that self-hosted software that are non-trivial to configure for security have "dev ux" type security bugs, but I doubt that'd constitute a CVE. It could also be unrelated to the git-web hosting config.
- tyingq 6y agoLooks like Softlayer / IBM Cloud. There are some high numbered ports listening and accepting connections, including git/9418, which is interesting.
- ChrisArchitect 6y agoFeel for the maintainers, tough to keep own stuff together -- but also first reaction is that another massively influential part of the web/opensource being stored on GitHub. I'm not anti-MS by any means but just reminded of the dominance of so much of our global codestores being on there
- pas 6y agoIt's git. That part is easy to migrate. Exporting issues and discussions might take a bit of work, but if/when GitHub starts going downhill there'll be a script to migrate to GitLab (or whatever will be the preferred option by then) (Yes, there's already such a script for GitLab.)
- pferde 6y agoUnless the first step in MS Github's downhill journey will be to cut off APIs needed for those exports.