6 ms·

This is an interesting example of a long-standing problem which is that in general we make use of huge amounts of software and trust our security with it, whils

This is an interesting example of a long-standing problem which is that in general we make use of huge amounts of software and trust our security with it, whilst knowing very little about the security practices of the people developing it. The article makes a good point that it's very hard for small projects, like the team running Homebrew, to fund their security, yet they are likely to be a target for quite high end attackers, given the access that can be gained by getting unauthorised access to package repositories. As a side note it also shows that Jenkins tends to be a tempting target for attackers as it often has access to a wide range of systems to carry out it's functions.

mikemcquaid 8y ago
> As a side note it also shows that Jenkins tends to be a tempting target for attackers as it often has access to a wide range of systems to carry out it's functions. This. I'd really love to stop us using Jenkins but none of the hosted macOS CI services scale to meet our needs (i.e. having jobs that run for multiple hours on better hardware). The ideal solution for me would be for us to have some sort of modified Travis or Circle CI setup. We can even pay for it now we've got Patreon money coming in.
- clarkdave 8y ago
  Have you looked at Buildkite[0]? The management/pipeline part is hosted (like Travis), but you run the build agents on your own infrastructure (they run on pretty much everything, including macs). Not affiliated, just a happy user. [0] http://buildkite.com http://buildkite.com
  - mikemcquaid 8y ago
    This looks really interesting, will take a look, thanks.
    - joncampbelldev 8y ago
      I can confirm that buildkite is a fantastic replacement for jenkins. - pipelines stored in source control - hosted on your own servers (with easy one click setup in AWS if you're too busy for all that) - excellent cloud hosted UI Also not affliated, just a happy user who replaced a large jenkins setup where I work with buildkite.
- alien_ 8y ago
  Jenkins is not the problem, we have quite a few secured instances which wouldn't leak secrets like this, or at least not to non-admin or unauthenticated users. It's just often misconfigured because there are a plethora of plugins and ways to store and use secrets, and nobody audits it enough to look at the console output or build artifacts for leaked secrets. The project is simply missing someone familiar enough to configure Jenkins properly.
  - mmt 8y ago
    > The project is simply missing someone familiar enough to configure Jenkins properly. I would say that this is a specific case of the far more general one. Substitute "any small organization" for "the project" and substitute any configuration familiarity (i.e. Ops skill) for Jenkins configuration familiarity. Even in "Devops" job postings for startups, when mentioning CI/CD tools like Jenkins, the main desire seems to be to hire someone who's more Dev than Ops, to create the code to run the CI/CD pipeline, with something like configuration or security a mere afterthough, if that.
- peterwwillis 8y ago
  Why not use that Patreon money to pay someone to set up Jenkins for you the right way?
  - snowwolf 8y ago
    Yeah. Let’s just blow the entire months budget on a single day of consulting. And I’m sure one day is all they would need to understand the workflow requirements of the team, implement and configure it and test it. That seems reasonable.
janwh 8y ago
> (...) very hard for small projects, like the team running Homebrew, to fund their security, yet they are likely to be a target for quite high end attackers, (...) I disagree. Homebrew’s security considerations in this case have nothing to do with their funding. There are lot of terrific services available for next to nothing for open source projects, Jenkins is one of them. It must have been a conscious decision the way HB set up their CI, unaffected by funding. And such lapses are not an open door jyst to “high end attackers”. This was a single person with just internet access and a little knowledge about how modern OSS projects work.
- raesene9 8y ago
  If they had better funding they could have afforded 3rd party security reviews, which (if done competently) would have flagged this issue. If they had better funding they could have hired dedicated security staff, who would likely be versed in the ways of securing DevOps pipelines. Time is money, more funding == more resources to look at security properly.
- mhneu 8y ago
  Time is money. Takes work to set up. Resources are being directed at development.
- falcolas 8y ago
  When time and money is limited, new features will always win both time and money, until something goes wrong. At that point where it's gone wrong, people will step in and lament "why didn't you just do X" for a few days, before they go back to wanting more features. The cost of good security is high - audits, slowed down development, limited data retention, higher compute costs... and the return on that investment is only ever going to be a reduction of liability. Big company with lots of resources, small company with no resources; it doesn't matter. Security is a cost center, and will only ever get a token amount of resources until the costs of doing nothing outweigh the costs of doing something.
  - mhneu 8y ago
    All three responses 'time is money'. Yes, security is a cost. There's a bit of tragedy of the commons effect here - many of the downsides are pushed onto others. I like Doctorow's general take on socializing costs of privacy and security breaches while privatizing profit: https://locusmag.com/2018/07/cory-doctorow-zucks-empire-of-oily-rags/ https://locusmag.com/2018/07/cory-doctorow-zucks-empire-of-o...
sonofblah 8y ago
How many billions or trillions of market value are built upon foundational technology like this, while the maintainers have to go begging for funds not for themselves, but for the good of the project(s)?
- ejholmes 8y ago
  It’s a pretty unfortunate state of affairs. Google/Microsoft/Amazon should be throwing money at these projects. It’s in their best interest if they use it.
  - kakwa_ 8y ago
    I would love if the following existed: A policy in tech companies to grant a given amount (let say 50$) per month to each employee, the employee can attribute this amount to whichever OSS project(s) he sees fit. If the employee doesn't chose a project, this amount is given to OSS projects in need. To streamline this, the ideal would be to have a service where everything could be done: * OSS projects register to it * Tech Companies then give the money to the service, which then dispatch it to the OSS projects * Employees of tech companies logs into the service to select which projects they chose to give money to (with maybe some suggestion to avoid over/under funding projects).
    - saagarjha 8y ago
      Or pay employees to work in open source?
    - majewsky 8y ago
      > to grant a given amount (let say 50$) per month to each employee Let's make that amount dependent on the employee's salary. Not 50$ per employee per month, but e.g. salary for 1 hour per employee per month. Salaries vary a lot between countries (and even within countries).
  - alien_ 8y ago
    This is happening all over the industry, pretty much all companies just make use of open source software without giving a penny to the thousands of projects they leverage on a daily basis to make profit or even keep the lights on. As someone who runs a small open source project that clearly states that Patreon donations are accepted and even offers some convenience benefits to donors, I often see people jumping extra hoops to avoid it, including clearly profitable companies that saved many thousands by using my software, spending extra time that would amount to more than those donations. So far I get donations from less than one percent of my estimated users.
deleted 8y ago
[deleted]
justicz 8y ago
> The article makes a good point that it's very hard for small projects, like the team running Homebrew, to fund their security, yet they are likely to be a target for quite high end attackers, given the access that can be gained by getting unauthorised access to package repositories. I know this is kind of a shameless plug, but this is the exact reason I launched BountyGraph [0] last week. I think that we can crowdfund security budgets for these projects to help encourage the discovery of issues like this one. [0] https://bountygraph.com/ https://bountygraph.com/

mikemcquaid 8y ago

> As a side note it also shows that Jenkins tends to be a tempting target for attackers as it often has access to a wide range of systems to carry out it's functions. This. I'd really love to stop us using Jenkins but none of the hosted macOS CI services scale to meet our needs (i.e. having jobs that run for multiple hours on better hardware). The ideal solution for me would be for us to have some sort of modified Travis or Circle CI setup. We can even pay for it now we've got Patreon money coming in.

clarkdave 8y ago
Have you looked at Buildkite[0]? The management/pipeline part is hosted (like Travis), but you run the build agents on your own infrastructure (they run on pretty much everything, including macs). Not affiliated, just a happy user. [0] http://buildkite.com http://buildkite.com
- mikemcquaid 8y ago
  This looks really interesting, will take a look, thanks.
  - joncampbelldev 8y ago
    I can confirm that buildkite is a fantastic replacement for jenkins. - pipelines stored in source control - hosted on your own servers (with easy one click setup in AWS if you're too busy for all that) - excellent cloud hosted UI Also not affliated, just a happy user who replaced a large jenkins setup where I work with buildkite.
alien_ 8y ago
Jenkins is not the problem, we have quite a few secured instances which wouldn't leak secrets like this, or at least not to non-admin or unauthenticated users. It's just often misconfigured because there are a plethora of plugins and ways to store and use secrets, and nobody audits it enough to look at the console output or build artifacts for leaked secrets. The project is simply missing someone familiar enough to configure Jenkins properly.
- mmt 8y ago
  > The project is simply missing someone familiar enough to configure Jenkins properly. I would say that this is a specific case of the far more general one. Substitute "any small organization" for "the project" and substitute any configuration familiarity (i.e. Ops skill) for Jenkins configuration familiarity. Even in "Devops" job postings for startups, when mentioning CI/CD tools like Jenkins, the main desire seems to be to hire someone who's more Dev than Ops, to create the code to run the CI/CD pipeline, with something like configuration or security a mere afterthough, if that.
peterwwillis 8y ago
Why not use that Patreon money to pay someone to set up Jenkins for you the right way?
- snowwolf 8y ago
  Yeah. Let’s just blow the entire months budget on a single day of consulting. And I’m sure one day is all they would need to understand the workflow requirements of the team, implement and configure it and test it. That seems reasonable.

janwh 8y ago

> (...) very hard for small projects, like the team running Homebrew, to fund their security, yet they are likely to be a target for quite high end attackers, (...) I disagree. Homebrew’s security considerations in this case have nothing to do with their funding. There are lot of terrific services available for next to nothing for open source projects, Jenkins is one of them. It must have been a conscious decision the way HB set up their CI, unaffected by funding. And such lapses are not an open door jyst to “high end attackers”. This was a single person with just internet access and a little knowledge about how modern OSS projects work.

raesene9 8y ago
If they had better funding they could have afforded 3rd party security reviews, which (if done competently) would have flagged this issue. If they had better funding they could have hired dedicated security staff, who would likely be versed in the ways of securing DevOps pipelines. Time is money, more funding == more resources to look at security properly.
mhneu 8y ago
Time is money. Takes work to set up. Resources are being directed at development.
falcolas 8y ago
When time and money is limited, new features will always win both time and money, until something goes wrong. At that point where it's gone wrong, people will step in and lament "why didn't you just do X" for a few days, before they go back to wanting more features. The cost of good security is high - audits, slowed down development, limited data retention, higher compute costs... and the return on that investment is only ever going to be a reduction of liability. Big company with lots of resources, small company with no resources; it doesn't matter. Security is a cost center, and will only ever get a token amount of resources until the costs of doing nothing outweigh the costs of doing something.
- mhneu 8y ago
  All three responses 'time is money'. Yes, security is a cost. There's a bit of tragedy of the commons effect here - many of the downsides are pushed onto others. I like Doctorow's general take on socializing costs of privacy and security breaches while privatizing profit: https://locusmag.com/2018/07/cory-doctorow-zucks-empire-of-oily-rags/ https://locusmag.com/2018/07/cory-doctorow-zucks-empire-of-o...

sonofblah 8y ago

How many billions or trillions of market value are built upon foundational technology like this, while the maintainers have to go begging for funds not for themselves, but for the good of the project(s)?

ejholmes 8y ago
It’s a pretty unfortunate state of affairs. Google/Microsoft/Amazon should be throwing money at these projects. It’s in their best interest if they use it.
- kakwa_ 8y ago
  I would love if the following existed: A policy in tech companies to grant a given amount (let say 50$) per month to each employee, the employee can attribute this amount to whichever OSS project(s) he sees fit. If the employee doesn't chose a project, this amount is given to OSS projects in need. To streamline this, the ideal would be to have a service where everything could be done: * OSS projects register to it * Tech Companies then give the money to the service, which then dispatch it to the OSS projects * Employees of tech companies logs into the service to select which projects they chose to give money to (with maybe some suggestion to avoid over/under funding projects).
  - saagarjha 8y ago
    Or pay employees to work in open source?
  - majewsky 8y ago
    > to grant a given amount (let say 50$) per month to each employee Let's make that amount dependent on the employee's salary. Not 50$ per employee per month, but e.g. salary for 1 hour per employee per month. Salaries vary a lot between countries (and even within countries).
- alien_ 8y ago
  This is happening all over the industry, pretty much all companies just make use of open source software without giving a penny to the thousands of projects they leverage on a daily basis to make profit or even keep the lights on. As someone who runs a small open source project that clearly states that Patreon donations are accepted and even offers some convenience benefits to donors, I often see people jumping extra hoops to avoid it, including clearly profitable companies that saved many thousands by using my software, spending extra time that would amount to more than those donations. So far I get donations from less than one percent of my estimated users.

deleted 8y ago

[deleted]

justicz 8y ago

> The article makes a good point that it's very hard for small projects, like the team running Homebrew, to fund their security, yet they are likely to be a target for quite high end attackers, given the access that can be gained by getting unauthorised access to package repositories. I know this is kind of a shameless plug, but this is the exact reason I launched BountyGraph [0] last week. I think that we can crowdfund security budgets for these projects to help encourage the discovery of issues like this one. [0] https://bountygraph.com/ https://bountygraph.com/