6 ms·

The vast majority of repos should be able to run CI on pull requests with no privileges at all. GitHub can manage any resource utilization issues on their end.

by trevyn 3y ago

The vast majority of repos should be able to run CI on pull requests with no privileges at all. GitHub can manage any resource utilization issues on their end. Is the issue here that a self-hosted runner was needed for some hardware tests?

withinboredom 3y ago
Self-hosted runners is the way to go, IMHO. Especially if you have bare metal resources. I love how fast my builds are with 16 cores, and gobs of ram.
- ethbr1 3y ago
  What's the GitHub Actions tooling like for emphemeral self-hosted runners? Afaict, a huge portion of this attack came from persistence on the self-hosted runner. Absent that, they would have needed a container jailbreak as well, which substantially ups the difficulty. And if a repo is running <100 builds a day, spin up + kill container seems a small per-build price to pay for the additional security isolation.
  - jackwilsdon 3y ago
    GitHub themselves don't seem to provide any mechanism to make runners ephemeral. It looks like all they allow you to do is flag a runner as ephemeral, meaning it will be de-registered once a job is completed - you need to write your own tooling to wipe it yourself (either via starting a whole new runner in a new environment and registering that or wiping the existing runner and re-registering it). https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners#using-ephemeral-runners-for-autoscaling https://docs.github.com/en/actions/hosting-your-own-runners/...
    - gz5 3y ago
      there are 3rd party foss options (1): 1. ephemeral + zero implicit trust (2) https://blog.openziti.io/my-intern-assignment-call-a-dark-webhook-from-aws-lambda https://blog.openziti.io/my-intern-assignment-call-a-dark-we... 2. zero implicit trust: https://github.com/openziti/ziti-webhook-action https://github.com/openziti/ziti-webhook-action (1) disclosure, maintainer (2) zero implicit trust in this case = no open inbound ports on underlay; need to access via app-specific overlay which requires strong identity, authN, authZ
    - crohr 3y ago
      I've just made runs-on [1] for that purpose: self-hosted, ephemeral runners for GitHub Action workflows. Long-running self-hosted runners are simply too risky if your project is public. [1]: https://runs-on.com https://runs-on.com
  - withinboredom 3y ago
    The default kubernetes implementation owned by github[1] assumes ephemeral runners by default. You can also specify what policies they should have using regular network policies provided by kubernetes. So, if you have a kubernetes cluster, that's the way to go. [1]: https://github.com/actions/actions-runner-controller https://github.com/actions/actions-runner-controller
o11c 3y ago
The problem is that there are fundamentally 2 different kinds of builds, but the current tooling is weak: * pre-merge builds on PRs. These should not have privileges, but making the distinction between the two cases requires a lot of care. * official builds of "master" or a feature branch from the main repo. These "need" privileges to upload the resulting artifacts somewhere. Of course, if all it did was wake up some daemon elsewhere, which could download straight from the CI in a verified way based on the CI's notion of the job name, it would be secure without privileges, but most CI systems don't want to preserve huge artifacts, and maintaining the separate daemon is also annoying.
- akx 3y ago
  For the second point, PyPI's trusted publisher implementation does this very well: https://docs.pypi.org/trusted-publishers/ https://docs.pypi.org/trusted-publishers/
  - LtWorf 3y ago
    Isn't that what's causing the problem? Without that there would be no need to have an action to do an upload. It could be comfortably and safely done offline.
- trevyn 3y ago
  It’s called GitHub secrets. Builds off of main get the secrets, pull requests from randos don’t. And public repos don’t pay for CI on GitHub. Not rocket science, people.
  - n2d4 3y ago
    Pytorch did use GH secrets for the valuables and you can see that this wasn't enough, right there in the OP, because the self-hosted runners are still shared
    - adnanthekhan 3y ago
      Yup! This is what makes this kind of attack scary and very unique to GitHub Actions. The baseline GITHUB_TOKEN just blows the door open on lateral movement via workflow_dispatch and and repository_dispatch events. In several of our other operations, not just PyTorch, we leveraged workflow_dispatch to steal a PAT from another workflows. Developers tend to over-provision PATs so often. More often than not we'd end up with a PAT that has all scopes checked and org admin permissions. With that one could clean out all of the secrets from an organization in minutes using automated tools such as https://github.com/praetorian-inc/gato https://github.com/praetorian-inc/gato.
Groxx 3y ago
>The vast majority of repos should be able to run CI on pull requests with no privileges at all When there are no side effects and no in-container secrets and the hosting is free or reasonably limited to prevent abusers, ideally yes. Outside that, heck no, that'd be crazy. You're allowing randos to run arbitrary code on your budget. Locking it down until it's reviewed is like step 1, they can validate locally until then.
- KptMarchewa 3y ago
  GH actions are free on public repos.
  - Groxx 3y ago
    Until someone tries to mine Bitcoin in yours, yeah
    - trevyn 3y ago
      Do you have a case study? My prior is that GitHub is pretty competent in understanding malicious PRs to open source repos, and wouldn’t penalize the repo owner without other evidence of wrongdoing.
- guappa 3y ago
  They are so difficult. I wanted to stop random people to run code on my repository… I don't have any secrets or write access or anything to exploit. Just to avoid burning quota. The issue is that now the pull requests don't get tested at all. I have to manually, locally, get all the commits, make a branch on the main repository with them, and then the actions run.
  - growse 3y ago
    One approach might be to review manually, then label the pr and trigger the pr based off the label being added. Challenge there is that if the pr changes, it's a bit clunky to retrigger the CI (you have to remove, then re-add). I guess you could also do this with comments - can you trigger a workflow based on a specific comment being added from a specific user?

by trevyn 3y ago

withinboredom 3y ago

Self-hosted runners is the way to go, IMHO. Especially if you have bare metal resources. I love how fast my builds are with 16 cores, and gobs of ram.

ethbr1 3y ago
What's the GitHub Actions tooling like for emphemeral self-hosted runners? Afaict, a huge portion of this attack came from persistence on the self-hosted runner. Absent that, they would have needed a container jailbreak as well, which substantially ups the difficulty. And if a repo is running <100 builds a day, spin up + kill container seems a small per-build price to pay for the additional security isolation.
- jackwilsdon 3y ago
  GitHub themselves don't seem to provide any mechanism to make runners ephemeral. It looks like all they allow you to do is flag a runner as ephemeral, meaning it will be de-registered once a job is completed - you need to write your own tooling to wipe it yourself (either via starting a whole new runner in a new environment and registering that or wiping the existing runner and re-registering it). https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners#using-ephemeral-runners-for-autoscaling https://docs.github.com/en/actions/hosting-your-own-runners/...
  - gz5 3y ago
    there are 3rd party foss options (1): 1. ephemeral + zero implicit trust (2) https://blog.openziti.io/my-intern-assignment-call-a-dark-webhook-from-aws-lambda https://blog.openziti.io/my-intern-assignment-call-a-dark-we... 2. zero implicit trust: https://github.com/openziti/ziti-webhook-action https://github.com/openziti/ziti-webhook-action (1) disclosure, maintainer (2) zero implicit trust in this case = no open inbound ports on underlay; need to access via app-specific overlay which requires strong identity, authN, authZ
  - crohr 3y ago
    I've just made runs-on [1] for that purpose: self-hosted, ephemeral runners for GitHub Action workflows. Long-running self-hosted runners are simply too risky if your project is public. [1]: https://runs-on.com https://runs-on.com
- withinboredom 3y ago
  The default kubernetes implementation owned by github[1] assumes ephemeral runners by default. You can also specify what policies they should have using regular network policies provided by kubernetes. So, if you have a kubernetes cluster, that's the way to go. [1]: https://github.com/actions/actions-runner-controller https://github.com/actions/actions-runner-controller

o11c 3y ago

The problem is that there are fundamentally 2 different kinds of builds, but the current tooling is weak: * pre-merge builds on PRs. These should not have privileges, but making the distinction between the two cases requires a lot of care. * official builds of "master" or a feature branch from the main repo. These "need" privileges to upload the resulting artifacts somewhere. Of course, if all it did was wake up some daemon elsewhere, which could download straight from the CI in a verified way based on the CI's notion of the job name, it would be secure without privileges, but most CI systems don't want to preserve huge artifacts, and maintaining the separate daemon is also annoying.

akx 3y ago
For the second point, PyPI's trusted publisher implementation does this very well: https://docs.pypi.org/trusted-publishers/ https://docs.pypi.org/trusted-publishers/
- LtWorf 3y ago
  Isn't that what's causing the problem? Without that there would be no need to have an action to do an upload. It could be comfortably and safely done offline.
trevyn 3y ago
It’s called GitHub secrets. Builds off of main get the secrets, pull requests from randos don’t. And public repos don’t pay for CI on GitHub. Not rocket science, people.
- n2d4 3y ago
  Pytorch did use GH secrets for the valuables and you can see that this wasn't enough, right there in the OP, because the self-hosted runners are still shared
  - adnanthekhan 3y ago
    Yup! This is what makes this kind of attack scary and very unique to GitHub Actions. The baseline GITHUB_TOKEN just blows the door open on lateral movement via workflow_dispatch and and repository_dispatch events. In several of our other operations, not just PyTorch, we leveraged workflow_dispatch to steal a PAT from another workflows. Developers tend to over-provision PATs so often. More often than not we'd end up with a PAT that has all scopes checked and org admin permissions. With that one could clean out all of the secrets from an organization in minutes using automated tools such as https://github.com/praetorian-inc/gato https://github.com/praetorian-inc/gato.

Groxx 3y ago

>The vast majority of repos should be able to run CI on pull requests with no privileges at all When there are no side effects and no in-container secrets and the hosting is free or reasonably limited to prevent abusers, ideally yes. Outside that, heck no, that'd be crazy. You're allowing randos to run arbitrary code on your budget. Locking it down until it's reviewed is like step 1, they can validate locally until then.

KptMarchewa 3y ago
GH actions are free on public repos.
- Groxx 3y ago
  Until someone tries to mine Bitcoin in yours, yeah
  - trevyn 3y ago
    Do you have a case study? My prior is that GitHub is pretty competent in understanding malicious PRs to open source repos, and wouldn’t penalize the repo owner without other evidence of wrongdoing.
guappa 3y ago
They are so difficult. I wanted to stop random people to run code on my repository… I don't have any secrets or write access or anything to exploit. Just to avoid burning quota. The issue is that now the pull requests don't get tested at all. I have to manually, locally, get all the commits, make a branch on the main repository with them, and then the actions run.
- growse 3y ago
  One approach might be to review manually, then label the pr and trigger the pr based off the label being added. Challenge there is that if the pr changes, it's a bit clunky to retrigger the CI (you have to remove, then re-add). I guess you could also do this with comments - can you trigger a workflow based on a specific comment being added from a specific user?