9 ms·
How I gained commit access to Homebrew in 30 minutes
- raesene9 8y agoThis is an interesting example of a long-standing problem which is that in general we make use of huge amounts of software and trust our security with it, whilst knowing very little about the security practices of the people developing it. The article makes a good point that it's very hard for small projects, like the team running Homebrew, to fund their security, yet they are likely to be a target for quite high end attackers, given the access that can be gained by getting unauthorised access to package repositories. As a side note it also shows that Jenkins tends to be a tempting target for attackers as it often has access to a wide range of systems to carry out it's functions.
- mikemcquaid 8y ago> As a side note it also shows that Jenkins tends to be a tempting target for attackers as it often has access to a wide range of systems to carry out it's functions. This. I'd really love to stop us using Jenkins but none of the hosted macOS CI services scale to meet our needs (i.e. having jobs that run for multiple hours on better hardware). The ideal solution for me would be for us to have some sort of modified Travis or Circle CI setup. We can even pay for it now we've got Patreon money coming in.
- clarkdave 8y agoHave you looked at Buildkite[0]? The management/pipeline part is hosted (like Travis), but you run the build agents on your own infrastructure (they run on pretty much everything, including macs). Not affiliated, just a happy user. [0] http://buildkite.com http://buildkite.com
- mikemcquaid 8y agoThis looks really interesting, will take a look, thanks.
- joncampbelldev 8y agoI can confirm that buildkite is a fantastic replacement for jenkins. - pipelines stored in source control - hosted on your own servers (with easy one click setup in AWS if you're too busy for all that) - excellent cloud hosted UI Also not affliated, just a happy user who replaced a large jenkins setup where I work with buildkite.
- alien_ 8y agoJenkins is not the problem, we have quite a few secured instances which wouldn't leak secrets like this, or at least not to non-admin or unauthenticated users. It's just often misconfigured because there are a plethora of plugins and ways to store and use secrets, and nobody audits it enough to look at the console output or build artifacts for leaked secrets. The project is simply missing someone familiar enough to configure Jenkins properly.
- mmt 8y ago> The project is simply missing someone familiar enough to configure Jenkins properly. I would say that this is a specific case of the far more general one. Substitute "any small organization" for "the project" and substitute any configuration familiarity (i.e. Ops skill) for Jenkins configuration familiarity. Even in "Devops" job postings for startups, when mentioning CI/CD tools like Jenkins, the main desire seems to be to hire someone who's more Dev than Ops, to create the code to run the CI/CD pipeline, with something like configuration or security a mere afterthough, if that.
- peterwwillis 8y agoWhy not use that Patreon money to pay someone to set up Jenkins for you the right way?
- snowwolf 8y agoYeah. Let’s just blow the entire months budget on a single day of consulting. And I’m sure one day is all they would need to understand the workflow requirements of the team, implement and configure it and test it. That seems reasonable.
- janwh 8y ago> (...) very hard for small projects, like the team running Homebrew, to fund their security, yet they are likely to be a target for quite high end attackers, (...) I disagree. Homebrew’s security considerations in this case have nothing to do with their funding. There are lot of terrific services available for next to nothing for open source projects, Jenkins is one of them. It must have been a conscious decision the way HB set up their CI, unaffected by funding. And such lapses are not an open door jyst to “high end attackers”. This was a single person with just internet access and a little knowledge about how modern OSS projects work.
- raesene9 8y agoIf they had better funding they could have afforded 3rd party security reviews, which (if done competently) would have flagged this issue. If they had better funding they could have hired dedicated security staff, who would likely be versed in the ways of securing DevOps pipelines. Time is money, more funding == more resources to look at security properly.
- mhneu 8y agoTime is money. Takes work to set up. Resources are being directed at development.
- falcolas 8y agoWhen time and money is limited, new features will always win both time and money, until something goes wrong. At that point where it's gone wrong, people will step in and lament "why didn't you just do X" for a few days, before they go back to wanting more features. The cost of good security is high - audits, slowed down development, limited data retention, higher compute costs... and the return on that investment is only ever going to be a reduction of liability. Big company with lots of resources, small company with no resources; it doesn't matter. Security is a cost center, and will only ever get a token amount of resources until the costs of doing nothing outweigh the costs of doing something.
- mhneu 8y agoAll three responses 'time is money'. Yes, security is a cost. There's a bit of tragedy of the commons effect here - many of the downsides are pushed onto others. I like Doctorow's general take on socializing costs of privacy and security breaches while privatizing profit: https://locusmag.com/2018/07/cory-doctorow-zucks-empire-of-oily-rags/ https://locusmag.com/2018/07/cory-doctorow-zucks-empire-of-o...
- sonofblah 8y agoHow many billions or trillions of market value are built upon foundational technology like this, while the maintainers have to go begging for funds not for themselves, but for the good of the project(s)?
- ejholmes 8y agoIt’s a pretty unfortunate state of affairs. Google/Microsoft/Amazon should be throwing money at these projects. It’s in their best interest if they use it.
- kakwa_ 8y agoI would love if the following existed: A policy in tech companies to grant a given amount (let say 50$) per month to each employee, the employee can attribute this amount to whichever OSS project(s) he sees fit. If the employee doesn't chose a project, this amount is given to OSS projects in need. To streamline this, the ideal would be to have a service where everything could be done: * OSS projects register to it * Tech Companies then give the money to the service, which then dispatch it to the OSS projects * Employees of tech companies logs into the service to select which projects they chose to give money to (with maybe some suggestion to avoid over/under funding projects).
- saagarjha 8y agoOr pay employees to work in open source?
- majewsky 8y ago> to grant a given amount (let say 50$) per month to each employee Let's make that amount dependent on the employee's salary. Not 50$ per employee per month, but e.g. salary for 1 hour per employee per month. Salaries vary a lot between countries (and even within countries).
- alien_ 8y agoThis is happening all over the industry, pretty much all companies just make use of open source software without giving a penny to the thousands of projects they leverage on a daily basis to make profit or even keep the lights on. As someone who runs a small open source project that clearly states that Patreon donations are accepted and even offers some convenience benefits to donors, I often see people jumping extra hoops to avoid it, including clearly profitable companies that saved many thousands by using my software, spending extra time that would amount to more than those donations. So far I get donations from less than one percent of my estimated users.
- deleted 8y ago[deleted]
- justicz 8y ago> The article makes a good point that it's very hard for small projects, like the team running Homebrew, to fund their security, yet they are likely to be a target for quite high end attackers, given the access that can be gained by getting unauthorised access to package repositories. I know this is kind of a shameless plug, but this is the exact reason I launched BountyGraph [0] last week. I think that we can crowdfund security budgets for these projects to help encourage the discovery of issues like this one. [0] https://bountygraph.com/ https://bountygraph.com/
- thecodemonkey 8y agoFrom the Homebrew post about the incident: “The security researcher also recommended we consider using GPG signing for Homebrew/homebrew-core. The Homebrew project leadership committee took a vote on this and it was rejected non-unanimously due to workflow concerns.” How is PGP signing not a no-brainer. What kind of workflow concerns would prevent them from signing commits?!
- ssutch3 8y agoEspecially when GPG signing is so easy: https://help.github.com/articles/signing-commits-using-gpg/ https://help.github.com/articles/signing-commits-using-gpg/
- mnarayan01 8y agoLinus (from 2009) on problems with signing every commit: http://git.661346.n2.nabble.com/GPG-signing-for-git-commit-tp2582986p2583316.html http://git.661346.n2.nabble.com/GPG-signing-for-git-commit-t...
- shittyadmin 8y agoHe seems to be discouraging signing every commit's individual data but encouraging signing the actual commit ID (SHA1) which should be perfectly feasible for something like homebrew.
- mnarayan01 8y ago> Signing each commit is totally stupid. It just means that you automate it, and you make the signature worth less.
- shittyadmin 8y agoYou're still getting a signature directly from the developer's machine, not from the repository server and as such you're still vastly shrinking the attack surface.
- 8y ago
- hartator 8y ago> I want to give special thanks to Mike McQuaid for his quick and professional handling of my report while on his paternity leave. Kudos on that. Specially when the all team make only around $700-800 per month. [1] [1] https://www.patreon.com/homebrew https://www.patreon.com/homebrew
- mikemcquaid 8y agoThanks! To be explicit though: that money all goes to the project and not to individuals. I've never made a cent from working on Homebrew.
- decasia 8y agoOne would think that Apple would be a prime candidate for contributing to Homebrew security funding, since in practice it is a project with security implications for so very many OSX developer workstations.
- sanderjd 8y agoUnfortunately, it has become increasingly clear that developer workstations are not on their radar as a primary target market.
- tambourine_man 8y agoI wouldn’t say that. Apple's hardware neglect is not a privilege only developers experience.
- dep_b 8y agoI don't think the 2018 MacBook Pro leaves much to be desired unless you want features that would turn it into a ThinkBook-like brick. For $50 you get a charging USB-C hub that allows you even to leave your power brick home. The iMac Pro is pretty great. Power Mac is a different story.....don't know how and when they're going to fix this. That's a real shame.
- kbd 8y ago> unless you want features that would turn it into a ThinkBook-like brick. Yeah literally all many of us want is a standard keyboard with function keys instead of the touchbar. A couple regular USB ports would be nice as well so we can plug in a mouse and keyboard without needing dongles. Neither of those things would turn their "pro" laptop into a brick.
- dep_b 8y agoMacBook 2015: power brick (with broken irreplaceable cord) + Ethernet dongle MacBook 2018: USB-C dock with replaceable cord So in my use case I actually got rid of one dongle at a price lower than a replacement MagSafe 2 adapter. I really liked MagSafe though. Never used the F-keys really. ESC is not that hard for me to live without but that's all personal. But the difference in performance is quite noticeable.
- binbag 8y agoI bet it's just what the guy wanted to find out he had to deal with while on paternity leave!
- mikemcquaid 8y agoThank god for naps.
- jlv2 8y ago> "On Jun 31st," As said in Raiders Of The Lost Ark, bad dates.
- closeparen 8y agoThis is a fundamental problem of doing CI for open source. Running tests for random people’s changes means giving them RCE in your test environment. Sandboxing rarely gets approached with the seriousness it requires.
- kovrik 8y agoI think that publicly exposing your build/deployment system is a very bad idea. It contains too much sensitive and valuable information (credentials, commits, commit author names, paths, hostnames, scripts etc.) and it is too hard to make it secure the right way. Unfortunately, many people do that. For example, here in New Zealand there is a new startup Onzo (bike sharing). When they launched, I tried to google about them (just out of curiosity). And in 1 minute I found their Jenkins server exposed to everyone as well. I could see how their build process works, who commits what etc. I decided to try to login using simple credentials (something like admin:password) and it didn't work. But there was a Register button. "Why not?" I thought and clicked it and created my own account. And voila, it gave admin permissions by default - I could delete their projects, change variables etc. Emailed them about that. Moral: never expose your build/deployment systems. If you really want to expose some parts (for whatever reason), then use/write client/UI that has no permissions. 'Build Status' badge is a good example -- it exposes build status info, but doesn't show too much and doesn't give any permissions whatsoever.
- falsedan 8y agoJenkins sucks for a lot of reasons but it does have a perfectly serviceable credentials store exactly for hiding these kinds of secrets from the parameters page and the build output. Any release engineer with the slightest inclination to avoid incidents would have set it up, this just looks like a lack of experience at breaking everything for everyone.
- sytelus 8y agoThis is an excellent piece. I often wonder about adversarial security issues in large scale OSS projects like Linux kernels. You don't even need to hack commit access to repo. One can intentionally wrap malicious code out in plain sight in an otherwise what would appear as benign change (thanks to undefined behaviour in C/C++). What if a black hat hacker climbs up in Linux contributors hierarchy? What if a person who is already higher up in OSS hierarchy decide to defect and plant a logic bomb? Given that Linux kernel now runs majority of our world from servers in data center to mobile phones in our pockets and hospitals to war machines, security issues like this is a huge deal.
- ejholmes 8y agoIt's a pretty scary prospect, to the point that I have to imagine it's already happening to some degree. If a nation state wants a backdoor, what better way than to bribe the cash-strapped OSS maintainer of that little project that every company depends on.
- ddalex 8y agoThe problem is that the type of engineers that work on OSS takes own integrity very seriously, and they build their network of trust on that integrity.
- sleepybrett 8y agoLol, always jenkins (CI).