7 ms·

This is a cat and mouse game. We add code to detect and disable abuse – sometimes in very clever ways – and then the abusers come up with a new way of circumven

This is a cat and mouse game. We add code to detect and disable abuse – sometimes in very clever ways – and then the abusers come up with a new way of circumventing that detection. In order to prevent miners from creating long queues for legitimate free users of GitHub Actions, we have to stay on top of this all the time. So the miners are not just stealing CPU time, they are also stealing engineer time. Because without mitigations the miners will consume all available CPU, and because devising abuse countermeasures is, for whatever reason, a very powerful nerd snipe (including for me!). The sad thing is that it's displacing time that would be spent improving Actions in other ways.

anoojb 6y ago
Thanks Nat. What if we made new GitHub Actions temporarily only available to users with a verified second factor? Could temporarily reduce the population of abusers while we figure out a more sustainable strategy?
- chatmasta 6y ago
  A TOTP code response is trivial to implement on the client. So if you wanted this to be meaningful, you would need to force users to use SMS 2FA, which is widely considered insecure. Not a great solution IMO.
deleted 6y ago
[deleted]
sorbits 6y ago
Do I understand correctly that the attacker forks a repository with GitHub Actions enabled, modifies the action, submits a PR, which makes GitHub run the altered action? If so, I wonder if there is a legit need for running modified GitHub actions from non-collaborators? Could also subject modified actions coming in via pull requests (from non-collaborators) to heavy resources constraints and timeouts.
- natfriedman 6y ago
  The mitigations you suggest are all logical. However, there are legitimate reasons to run CI and tests for outside contributions without taxing maintainers with the cognitive load of having to evaluate whether each contribution is CI-worthy. The attack vector in the article is not the main way miners try to steal CPU from the GitHub community. It's just an interesting one that the journalist chose to write about.
  - teachtwolearn 6y ago
    But when a PR is submitted that modifies an Actions workflow, shouldn't GitHub run the old unmodified workflow until that PR is accepted? IIRC, they already treat the .github folder as a special case; you can't push modifications to workflow files with a personal access token. So why not ensure that an action or workflow will only run if it is checked into the base branch? That wouldn't stop PRs from modifying scripts that the action runs, but the current behavior seems a bit counter-intuitive.
    - lmeyerov 6y ago
      If that action is "./run_tests.sh", which is a top use case, the attacker just changes "./run_tests.sh", so while I agree that's useful, it doesn't secure the typical case, and makes for a hard cost/value stance. The threat models are probably more like 1. "make sure only the right people run actions" and separately, 2. "make sure authorized events/actions only use the expected capabilities." Both largely fail today.
    - cortesoft 6y ago
      Well the idea is that a person submits a PR, and the action runs to verify that the tests pass BEFORE the PR is accepted. You don’t want to wait until after the code is merged in order to see if tests still pass. The issue is that even if you don’t allow changes to the actual action workflow, running tests gives an attacker the ability to run arbitrary code. They just need to add the code they want to run to the tests (e.g. have the tests mine crypto)
    - 6y ago
bko 6y ago
Can't you say the same thing about any defensive measures? The need for security is displacing developer time to be building out cool new features.
- natfriedman 6y ago
  Yes. This is just a category of attack whose growth has been incentivized by rising crypto prices. All providers of free compute are experiencing some level of mining attack right now. Eventually a new equilibrium will emerge.
  - knorker 6y ago
    A new equilibrium that thanks to cryptocurrency speculators is a worse world. This speculation is making some people rich, yes. But the amount of externalities is staggering. Thanks to PoW nobody can provide a free compute anymore without getting owned, and of course the environmental impact of bitcoin alone is worse than when Saddam Hussein set oil wells on fire while retreating. Let the world burn, and products rot to shit, as long as my HODL portfolio goes up. Cryptocurrency supporters really are sociopaths, worse than any hedge fund manager.
    - teitoklien 6y ago
      So you’re blaming cryptocurrencies for .... Proof Of Work ?. A lot of em are already trying to shift to other viable alternative proofs. Your analogy of calling Cryptocurrency supporters as sociopaths. sounds similar to insulting Edison because he designed the inefficient incandescent bulbs , which consume waaay more energy compared to LEDs built these days. How would it sound , if someone insults artificial light , just because of that ? . Cryptocurrencies are perfectly good ideas/products. Proof of Work’s viability isn’t. The hate is aimed in the wrong direction.
      - matkoniecz 6y ago
        > Cryptocurrencies are perfectly good ideas/products. for scammers and speculators
lmeyerov 6y ago
Actions are awesome... but scary as soon as you have a public repos, contractor, rogue employee, etc. They seem to go against security fundamentals. Ex: Actions should allow going into default-deny mode for all basic runtime capabilities and resource use, and only brought back on via RBAC. Today, it's not hard to steal npm/pip/etc creds or get into people's corp runners. Having gone through the browser security policy heyday, this is deja vu, except now for exposing the server side and supply chain. Ex: - do not run on any event.. unless user authorized for that event. Same for actions. - separate out policies and users cannot edit policies unless authorized to do that - do not get physical/logical resources (runners, disk quota, long runs, ...) unless given - default-deny network outbound with url safe-listing That way only trusted users can run them, and a bit harder for them to get hurt when there is a surprising action that they run The next level would probably be something like sandboxing : allow anyone to run an action , but a sandbox mode can autofail if violated, and have explicit imports/exports to lock down for how it gets used. A lot possible.. but need to invest in the basics first..
- slrz 6y ago
  Getting a hold of someone's secrets is not possible just by doing a pull request. It's really only about resource usage, at least when the runners in question provide sufficient isolation (true at least for the Github-hosted ones, or we're all in big trouble). Unfortunately, using self-hosted runners to provide additional capabilities not supported by Github-hosted ones is basically impossible (for public repos at least) as you can't restrict a runner to an organization or project. Set up a bare-metal runner and it will receive jobs from random forks.
  - chatmasta 6y ago
    > Getting a hold of someone's secrets is not possible just by doing a pull request Only if you've configured the actions correctly. I would bet that there is a high number of repositories on both GitLab and GitHub with misconfigured CI pipelines where someone can submit a PR with `env | curl` to grab any secrets defined as environment variables.
kerng 6y ago
I wonder if it would help forming an internal red team that could help stay a step ahead with such and related attacks and abuse scenarios by running such attacks against yourself?
merb 6y ago
limit the action configurations file to be only editable by a configured set for a specific repo and give us a special folder like .github-action-commands or so that is scoped like that aswell..

anoojb 6y ago

Thanks Nat. What if we made new GitHub Actions temporarily only available to users with a verified second factor? Could temporarily reduce the population of abusers while we figure out a more sustainable strategy?

chatmasta 6y ago
A TOTP code response is trivial to implement on the client. So if you wanted this to be meaningful, you would need to force users to use SMS 2FA, which is widely considered insecure. Not a great solution IMO.

deleted 6y ago

[deleted]

sorbits 6y ago

Do I understand correctly that the attacker forks a repository with GitHub Actions enabled, modifies the action, submits a PR, which makes GitHub run the altered action? If so, I wonder if there is a legit need for running modified GitHub actions from non-collaborators? Could also subject modified actions coming in via pull requests (from non-collaborators) to heavy resources constraints and timeouts.

natfriedman 6y ago
The mitigations you suggest are all logical. However, there are legitimate reasons to run CI and tests for outside contributions without taxing maintainers with the cognitive load of having to evaluate whether each contribution is CI-worthy. The attack vector in the article is not the main way miners try to steal CPU from the GitHub community. It's just an interesting one that the journalist chose to write about.
- teachtwolearn 6y ago
  But when a PR is submitted that modifies an Actions workflow, shouldn't GitHub run the old unmodified workflow until that PR is accepted? IIRC, they already treat the .github folder as a special case; you can't push modifications to workflow files with a personal access token. So why not ensure that an action or workflow will only run if it is checked into the base branch? That wouldn't stop PRs from modifying scripts that the action runs, but the current behavior seems a bit counter-intuitive.
  - lmeyerov 6y ago
    If that action is "./run_tests.sh", which is a top use case, the attacker just changes "./run_tests.sh", so while I agree that's useful, it doesn't secure the typical case, and makes for a hard cost/value stance. The threat models are probably more like 1. "make sure only the right people run actions" and separately, 2. "make sure authorized events/actions only use the expected capabilities." Both largely fail today.
  - cortesoft 6y ago
    Well the idea is that a person submits a PR, and the action runs to verify that the tests pass BEFORE the PR is accepted. You don’t want to wait until after the code is merged in order to see if tests still pass. The issue is that even if you don’t allow changes to the actual action workflow, running tests gives an attacker the ability to run arbitrary code. They just need to add the code they want to run to the tests (e.g. have the tests mine crypto)
  - 6y ago

bko 6y ago

Can't you say the same thing about any defensive measures? The need for security is displacing developer time to be building out cool new features.

natfriedman 6y ago
Yes. This is just a category of attack whose growth has been incentivized by rising crypto prices. All providers of free compute are experiencing some level of mining attack right now. Eventually a new equilibrium will emerge.
- knorker 6y ago
  A new equilibrium that thanks to cryptocurrency speculators is a worse world. This speculation is making some people rich, yes. But the amount of externalities is staggering. Thanks to PoW nobody can provide a free compute anymore without getting owned, and of course the environmental impact of bitcoin alone is worse than when Saddam Hussein set oil wells on fire while retreating. Let the world burn, and products rot to shit, as long as my HODL portfolio goes up. Cryptocurrency supporters really are sociopaths, worse than any hedge fund manager.
  - teitoklien 6y ago
    So you’re blaming cryptocurrencies for .... Proof Of Work ?. A lot of em are already trying to shift to other viable alternative proofs. Your analogy of calling Cryptocurrency supporters as sociopaths. sounds similar to insulting Edison because he designed the inefficient incandescent bulbs , which consume waaay more energy compared to LEDs built these days. How would it sound , if someone insults artificial light , just because of that ? . Cryptocurrencies are perfectly good ideas/products. Proof of Work’s viability isn’t. The hate is aimed in the wrong direction.
    - matkoniecz 6y ago
      > Cryptocurrencies are perfectly good ideas/products. for scammers and speculators

lmeyerov 6y ago

Actions are awesome... but scary as soon as you have a public repos, contractor, rogue employee, etc. They seem to go against security fundamentals. Ex: Actions should allow going into default-deny mode for all basic runtime capabilities and resource use, and only brought back on via RBAC. Today, it's not hard to steal npm/pip/etc creds or get into people's corp runners. Having gone through the browser security policy heyday, this is deja vu, except now for exposing the server side and supply chain. Ex: - do not run on any event.. unless user authorized for that event. Same for actions. - separate out policies and users cannot edit policies unless authorized to do that - do not get physical/logical resources (runners, disk quota, long runs, ...) unless given - default-deny network outbound with url safe-listing That way only trusted users can run them, and a bit harder for them to get hurt when there is a surprising action that they run The next level would probably be something like sandboxing : allow anyone to run an action , but a sandbox mode can autofail if violated, and have explicit imports/exports to lock down for how it gets used. A lot possible.. but need to invest in the basics first..

slrz 6y ago
Getting a hold of someone's secrets is not possible just by doing a pull request. It's really only about resource usage, at least when the runners in question provide sufficient isolation (true at least for the Github-hosted ones, or we're all in big trouble). Unfortunately, using self-hosted runners to provide additional capabilities not supported by Github-hosted ones is basically impossible (for public repos at least) as you can't restrict a runner to an organization or project. Set up a bare-metal runner and it will receive jobs from random forks.
- chatmasta 6y ago
  > Getting a hold of someone's secrets is not possible just by doing a pull request Only if you've configured the actions correctly. I would bet that there is a high number of repositories on both GitLab and GitHub with misconfigured CI pipelines where someone can submit a PR with `env | curl` to grab any secrets defined as environment variables.

kerng 6y ago

I wonder if it would help forming an internal red team that could help stay a step ahead with such and related attacks and abuse scenarios by running such attacks against yourself?

merb 6y ago

limit the action configurations file to be only editable by a configured set for a specific repo and give us a special folder like .github-action-commands or so that is scoped like that aswell..