25 ms·

GitHub: Widespread Injection Vulnerabilities in Actions

624 points by looperhacks 6y ago

kerng 6y ago
Wonder when the security industry comes up with better rating system for vulnerabilities. The rating for this one is all over the place...
est31 6y ago
Wow this looks quite serious. I hope that github will at least provide a way to turn these commands off entirely. I hope they also fix the stupid authwalling of the execution logs that happens for public repos. The code is public, but the logs need an account? There is barely any security benefit in doing this for public repos, but it does make it harder for me to check PRs that I make to open source projects while not being logged in (e.g. from my phone).
- whymarrh 6y ago
  At their scale I do believe there’s a benefit to authenticating to see logs: a lot of people scrape GitHub for secrets. CI logs are at high risk for user error, errors where a user unintentionally marks something as non-secret when it should’ve been secret. Putting these logs behind auth feels like an easy filter for some scraping.
  - est31 6y ago
    That's a good point I guess. Per default, secrets are redacted by github in the logs, but some might slip through, e.g. if only a part of them is printed. Doesn't make me really happy though, I don't want to have to use an account :).
  - holman 6y ago
    Yeah, definitely. When I was at GitHub we were seeing secrets getting lifted from public pushes and tried within 7 seconds or so, if I recall correctly, and this was five years ago. This was a big reason why there’s a real scanning API now for service providers to be informed if a secret leaks. By the time a human discovers their mistake it’s usually far too late.
- nexuist 6y ago
  The code is public but the secrets are obviously not, and if they are ever put in the logs then that's a security issue. Even if the action isn't logging credentials directly, it may be logging privileged information (e.g. S3 bucket names or contents) that can only be accessed with privileged credentials. Therefore it makes sense to keep the logs private. Maybe there should be a setting for Action owners where they can set logs public if they think it would help contributors.
chrisrpatterson 6y ago
https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/ https://github.blog/changelog/2020-10-01-github-actions-depr...
deleted 6y ago
[deleted]
jasonmp85 6y ago
Is anyone who’s used Actions surprised? They launched VERY unpolished and this sort of lack of care seems par for the course, tbh.
- bredren 6y ago
  I am in the process of migrating a client’s CI to GA and have seen rough edges. One example is there is no native way to serialize workflow execution. This means if you run two commit-based triggered actions in quick succession, you can have tests pass that should not. (They run against the wrong checked out code!) I had this happen in front of the client, where it appeared a unit test of assertTrue(False) passed! It was so undermining of GA I had to chase down the exact status and prove ability to avoid (using Turnstyle) to the client. I wrote an email to the GA team specifically telling this story and expressing my concern. Another is scheduled actions, where even if you specify a cron, the action has a window of several minutes when a GitHub runner will execute it, and there are cases where the window is missed and the action is not run! This isn’t documented, though GitHub recently made the docs community-editable, possibly realizing there is too much to do. All that said, the product is still relatively new, and even with these rough edges it is __really__ good. Having CI definition alongside code, with free-ish VMs available at all times, (except when it goes down,) and results all inside GitHub is amazing. Also, the self-hosted runners are amazing, as they allow some really simple matrix definition to run CI on custom HW and network environments. If it weren’t for all the recent negative attention recently, I would have said GH was doing really well, inclusive of GitHub Actions.
  - amscanne 6y ago
    Are you joking about the self hosted runners? The fact that you can’t: 1) Apply any security policy for runners (e.g. require a label before running a PR) 2) Have runners quit after a single job so you can build ephemeral runners 3) Build your own runner against an open API ... means that self hosted runners are non starter for anything open source. It’s like they had to try hard to make the architecture that obtuse and closed. It’s unclear if it’s really poor/design or an active attempt to somehow drive the business.
nixgeek 6y ago
Interesting that Project Zero considers this “High” severity, and GitHub says “Moderate”. Legitimate disagreement between security teams, or is GitHub applying spin and trying to downplay the severity, in light of there being no easy fix?
- dathinab 6y ago
  If you look at how GitHub operates wrt. GitHub releases or GitHub actions it becomes clear (to me) that they have very different standards to certain security aspects then me or e.g. project zero. So I would argue it's a case about having less strict standards wrt. Security vulnerabilities which requir your to pull in corrupted (or very careless) 3rd party code.
- arianvanp 6y ago
  and the CVE got registered as "Low" https://nvd.nist.gov/vuln/detail/CVE-2020-15228 https://nvd.nist.gov/vuln/detail/CVE-2020-15228 So what is it? High, Moderate, or Low? Super confusing
  - csnover 6y ago
    I’ve found that consistent CVSS scoring is practically impossible, both because many criteria are defined in ambiguous or confusing ways, and because rating any obvious knock-on effects is proscribed by the official scoring rules[0][1]. For example, if a software component has a vulnerability which allows an attacker to steal admin credentials by exfiltrating a password file, the fact that this would allow the attacker to then have admin access to do whatever they wanted in other software that shares credentials doesn’t matter when calculating the core score of the vulnerability. (As a user, you would need to calculate the “environmental score” in the calculator[2] to decide what your score is given whatever configuration you/your company uses.) Additionally, if some software enables privilege escalation at the OS level by misusing APIs, this escalation is not a “scope change” unless the software tried to create a separate security authority for itself by having some form of sandbox or authentication, so a vulnerability of this kind in software that has no added security mechanisms at all will be scored lower than a vulnerability in software which has implemented some additional access control which was bypassed. I’m not saying that the limits are nonsensical since without them you’d end up with a whole lot of Critical base scores, but in my limited experience the CVSS core score is pretty worthless when it comes to getting a sense of how bad a vulnerability actually is and I don’t know how security analysts cope with this system. [0] https://www.first.org/cvss/v3.0/user-guide https://www.first.org/cvss/v3.0/user-guide [1] https://www.first.org/cvss/v3.0/examples https://www.first.org/cvss/v3.0/examples [2] https://www.first.org/cvss/calculator/3.0 https://www.first.org/cvss/calculator/3.0
dathinab 6y ago
Not really surprised tbh. GitHub actions while very flexible are also prone to ask kinds of problems. E.g. versions of actions on the market place are not pinned so people can republish different code under the same version. So you need to pin to a git commit hash which is much easier to get wrong and also easier to abuse for social engineering. And that's assuming you trust git commit hashes, if not you need to fork actions just to make sure the public code you reviewed doesn't get changed without new reviewes .
- zomgwat 6y ago
  I've recently started using GitHub Actions and I was very disappointed to learn how few "native" actions there are. I don't want to introduce yet another avenue of third-party code via the GitHub Actions marketplace.
  - Arnavion 6y ago
    My GH workflows are a thin layer over the `run` action that runs shell scripts. Not only does my CI not have to rely on third-party actions, it's also extremely portable between CI providers, can be tested locally, and is infinitely more flexible. (Apart from having run actions, the only other actions I use are the upload/download-artifact ones to share binaries between jobs, and the matrix feature for parallelizing runs.) But to be clear, the kind of vulnerability that the P0 issue is talking about is not necessarily from malicious actions. As the issue itself says, the vuln also happens from untrusted input being processed by a trusted action, like issues and PRs.
    - bastardoperator 6y ago
      This is what every build should be. If I can't build it locally and in CI using the same process, I consider it broken. I tend to use Makefile, but what your describing is what I consider best practice.
    - dathinab 6y ago
      I plan to do something similar soon ;=) I played around with some actions wrt. commit signing and originally wrote a action a verify commit and tag signatures (in the way I need ;=) ). But now I plan to port it to a more general interface I can call from a GH action.
Pfhreak 6y ago
Interpreting text for commands has an interest parallel to a major World of Warcraft bug. It turns out that you could run arbitrary lua code in your chat terminal by using the "/run ..." command. Enterprising thieves figured out that one of the things you could do was redefine functions, in particular the 'RemoveExtraSpaces' function that gets run against every chat line. If you set RemoveExtraSpaces=RunScript you've told WoW to execute every chat message displayed in your terminal. Attackers just have to trick you into writing "/run RemoveExtraSpaces=RunScript" into your terminal, and after that all they had to do was chat code to you and it would execute. They basically get full control of anything lua can do in your game -- including sending messages to your friends asking them to also run the attack command.
- est31 6y ago
  Kinda reminds me of the "don't type anything into the browser console" warnings that sites like facebook etc. emit so that people don't type in commands strangers gave them, believing it allows them "hack" others while in reality it only hacks themselves.
- diarrhea 6y ago
  That is super interesting. When did this happen? What is the worst that did happen through this?
  - Pfhreak 6y ago
    I'm not sure. I do know that you could, for instance, have your gold sent to a scammer with little to no recourse. It may be possible that this bug still exists because it's one of those "bugs that is also a feature".
    - lilyball 6y ago
      Blizzard's response was to modify /run to pop up a dialog first warning you that /run is used by scammers and requires you to confirm that you know what you're doing.
  - 6y ago
abcxjeb284 6y ago
It’s pretty absurd for github to suggest that you should go through multiple steps to disable commands to log untrusted output. [1] Poor form to expect developers to understand and check for a new way that they need to sanitize their input rather than Github fixing it (possibly in a backwards incompatible way) At a minimum they should provide a shell script (`show $XYZ`) and a js function that handles generating those tokens and enabling/disabling workflow commands for you. [1] https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/ https://github.blog/changelog/2020-10-01-github-actions-depr...
- hn_throwaway_99 6y ago
  IMO at the very least they should have an org-wide option in settings to disable command interpretation.
  - mjgs 6y ago
    Yeah the ability to disable the insecure commands is crucial. That should be the first thing they should do. Probably a lot of people aren’t even using this functionality in their workflows anyway. I think the only one of these commands that I actually use is setting outputs, and I don’t even use that very often. There should also be a way to see which commands have been triggered via stdout, so you can at least see what happened if something malicious happens.
dhagz 6y ago
My employer recently decided to move from using self-hosted Jenkins servers living behind a VPN to GitHub actions. I wonder how much vulnerabilities like this were weighed against the reduced maintenance burden.
Karliss 6y ago
I surprised that Project Zero waited the last 30 days for disclosure since Github themselves publicly announced at 1. October that the whole searching for magic strings in stdout as RPC channel is broken by design. When I saw the announcement month ago I assumed that Github fully disabled the reading commands from stdout mechanism but looking at current docs it seems that they disabled only 5 most easily exploitable out of 14 commands which can be called using stdout. This seems like typical "we fixed your POC doesn't work anymore" but the main cause remains. As can be sometimes seen with hardware vendors "we fixed hard-coded backdoor password, now there is new hard-coded backdoor password".
- guitarbill 6y ago
  I'm not sure. When using `::set-output name=spam::eggs`, you seem to have to refer to it via `steps.foo.outputs.spam`. So if you keep steps that set outputs very simple, and/or the set-output echo is the last command of the step, then it should mitigate that? Is the general consensus that `$GITHUB_ENV` offers less attack surface, despite not being scoped to a step? Some guidance on this would be great from Github's side. Also appreciated: a way of turning off magic stdout strings in the YAML, and a way to set outputs via a more robust mechanism, and/or to scope which steps can modify which environmental variables. I get not wanting to break existing CI runs, but giving the people who want it an option to harden the CI proactively would be great. I don't really care if this would break some existing actions, either - seems fine to do for opt-in?
  - mjgs 6y ago
    Most useful comment so far in this thread. I’ve been using $GIT_ENV for setting environment variables, and though I read about the stdout commands I never got to using them. It’s one of those “but of course” moments realising that any cli tool can trigger commands with it’s stdout. As you mentioned setting outputs is rather crucial, I guess using stdout is the only way todo that. There are definitely a lot of security things to consider when using actions. I read the code of any community actions I use but, especially after reading this alert, I feel like there are a lot of places where malicious code could cause damage.
deleted 6y ago
[deleted]
kanox 6y ago
Running CI against arbitrary code is extremely difficult to do securely and I'm not sure anyone does more than a token effort. Test infrastructure is not designed to run against malicious code. One of the examples dumps environment variables which may contain secrets by injecting code through a github action. What's stopping code inside a PR from sending the same info over email?
- takluyver 6y ago
  I believe CI on pull requests runs without the secrets, to avoid precisely that issue.
  - amscanne 6y ago
    Yes, the problem is that GitHub did not seem to consider that “malicious input” can include any content that is provided and parsed in some way. Unfortunately, all of stdout is parsed, and often includes things like issue titles, descriptions, commit messages, etc.
sounds 6y ago
I've often heard HNers state that Project Zero is unwilling/uninterested in protecting the interests of the vulnerable host, but this seems like an excellent case study where Github, the host, just doesn't think this is a big deal. I think that applies to any host who doesn't patch their vulnerabilities - if they thought it was a big deal, they'd do something about it. Actions speak. Quoting from the post: 2020-07-21 Report sent to Github and triaged by Github security team. Disclosure date is set for 2020-10-18. 2020-08-13 Project Zero requests a status update. 2020-08-21 Github say that they are are still working on a fix and a deprecation plan 2020-10-01 Github issued an advisory[1] deprecating the vulnerable commands, assigned CVE-2020-15228, and asked users to patch and update their workflows. 2020-10-12 Project Zero reach out and proactively mentions that a grace period is available if Github wants more time to disable the vulnerable commands. 2020-10-16 Github requested the additional 14 day grace period, with the hope of disabling the vulnerable commands after 2020-10-19. 2020-10-16 Project Zero grants grace period, new disclosure date is 2020-11-02. 2020-10-28 Project Zero reaches out, noting the deadline expires next week. No response is received. 2020-10-30 Due to no response and the deadline closing in, Project Zero reaches out to other informal Github contacts. The response is that the issue is considered fixed and that we are clear to go public on 2020-11-02 as planned. 2020-11-01 Github responds and mentions that they won't be disabling the vulnerable commands by 2020-11-02. They request an additional 48 hours, not to fix the issue, but to notify customers and determine a "hard date" at some point in the future. 2020-11-02 Project Zero responds that there is no option to further extend the deadline as this is day 104 (90 days + 14 day grace extension) and that the disclosure will be today.
- zenexer 6y ago
  Indeed, if that timeline is to be believed—and I have no reason to doubt it—it’s pretty damning. GitHub just kept putting it off. When they asked to delay the disclosure indefinitely at the very last second, of course they were told that would be unacceptable.
  - naikrovek 6y ago
    I dunno. I've worked in many places that ABSOLUTELY had the desire, willingness, money, time, and ability to do something, and were still unable to make that thing happen, for reasons that can't be spoken about. While it is true that if an entire organization wants something done that it will happen, it is not true that inaction by an organization always signifies unwillingness or inability to act. It's easy to sit on the sidelines and throw tomatoes at the parties involved. It is another matter entirely when you are hamstrung by internal process or a specific power-tripping and ignorant person in management.
nickcw 6y ago
GitHub has been nagging about needing to fix this for a week or two. Just in case anyone is interested here is what the fix to rclone's build workflow looked like: https://github.com/rclone/rclone/commit/a152351a71ce21090a3be1944e2c016682d6e525 https://github.com/rclone/rclone/commit/a152351a71ce21090a3b...
- strbean 6y ago
  Love rclone, thanks for all your hard work!
withinboredom 6y ago
Does anyone actually trust their CI for running PRs on a public repo (in repo or forks)? I was taught from my first forays into commercial software never to trust CI's until after a review (even if it's another coworker).
ffpip 6y ago
Every time I visit bugs.chromium.org in Firefox, something weird happens with the fonts and text selection. This doesn't happen in Chromium Are they publicly disclosing/warning about a bug in FF, or is it just some other problem?
- yellowfool 6y ago
  How do your font issue relate to a github security issue ?
  - ffpip 6y ago
    No this is not related to that. I just posted a comment because many people will visit the site I mentioned, probably using Firefox. I am able to reproduce this 'bug'. It's like mouse hijacking. Not able to select text on bugs.chromium.org when using Firefox.
junon 6y ago
I remember thinking "yeah this isn't going to be safe..." when I saw how Github actions parsed those commands. It just seemed like a huge security vulnerability waiting to happen. Glad to know my gut still sort-of works. Not sure who thought this was a good idea but I can think of a handful of other ways off the top of my head that this could have been implemented in a more secure way.
prepend 6y ago
I’m not happy with the security on GitHub actions. The current options I have for my orgs is to either allow all actions, allow actions hosted in the org, or whitelist specific actions by name. It’s not clear to see actions and any testing or certification they may have. Even github built actions might be under an individual’s account. So it takes quite a bit of effort to check out an action to see if it’s safe or dangerous. And I worry about actions a little more because they do things like copy files, deploy systems, and other things where I wouldn’t want rogue code interjecting malware. Currently I require all actions to be hosted in the org so devs have forked actions into our org. That at least makes a developer vouch for a particular version and makes it harder for an update to that action to break something. I think they need an equivalent of “gold actions” endorsed by GitHub. Or a vetted repo like pypi or cran or something else that actually has some controls around versions and changes.
- Kliment 6y ago
  I suggest you fork any actions you depend on.
  - a_t48 6y ago
    Given that actions can reference other actions, is this even feasible? You'd have to go and fork all deps, and then if you ever want to pull upstream, repatch.
    - mjgs 6y ago
      I don’t think that’s the case. As far as I have read, a multi-step action that you create cannot use a community action. I was wanting to do this so as to be able to break up some lengthy workflows into reusable pieces, so I figured I’d create some actions, but when I tried to use for example the standard checkout action, inside my action, it failed. I emailed support and they said that actions inside actions was not supported. Perhaps you are referring to something else? A different type of action?
      - a_t48 6y ago
Animats 6y ago
As the runner process parses every line printed to STDOUT looking for workflow commands, every Github action that prints untrusted content as part of its execution is vulnerable. What could possibly go wrong?
deleted 6y ago
[deleted]
curryhoward 6y ago
A small plug: Toast (https://github.com/stepchowfun/toast https://github.com/stepchowfun/toast) has already introduced a patch for this vulnerability (CVE-2020-15228), so arbitrary user code running inside Toast will not be able to trigger this vulnerability when run via a GitHub Action. I highly recommend using Toast for your CI jobs (with or without GitHub Actions), not just for this security fix but also because it also allows you to run your CI jobs locally. It just runs everything inside Docker containers, so it works the same in CI, on your laptop, on your coworker's machine, etc.
paulstovell 6y ago
Octopus Deploy and TeamCity both parse output looking for commands in the same way (called Service Messages). I believe it is a common pattern in all CI servers. The issue here doesn’t seem to be parsing the output, but what the parsed commands can do. In Octopus there is no way to use them to run commands or set environment variables though. I can’t speak for TeamCity. Update: TeamCity's documentation page about service messages: https://www.jetbrains.com/help/teamcity/service-messages.html https://www.jetbrains.com/help/teamcity/service-messages.htm... The commands are quite specific though: test results, build numbers, etc. no generic commands like running commands or setting environment variables.
Serow225 6y ago
I created this ticket six months ago about the the inability to mask secrets that are step outputs (like from Terraform) and them subsequently leaking to logs, no action taken yet... If anyone has influence to raise visibility of this (Twitter, GitHub, whatever) that would be appreciated :) https://github.com/actions/runner/issues/475 https://github.com/actions/runner/issues/475
based2 6y ago
https://about.gitlab.com/releases/2020/11/02/security-release-gitlab-13-5-2-released/ https://about.gitlab.com/releases/2020/11/02/security-releas... GitLab Security Release: 13.5.2, 13.4.5, and 13.3.9
- luhn 6y ago
  I don't see anything to indicate this is related?
reilly3000 6y ago
I’ve always worried about env leakage, especially in npm packages. GHA does a decent job of masking but there is no stopping arbitrary code. One thing they could do is pass log output against known GHA Secrets and run an additional masking on the output string. To put this into context everything about this attack is logged in the open I believe. If that shows up in log output, you’ll know who did it and that everyone can see it - and proceed with rotating secrets promptly and report the hack to GitHub.
fsdgd 6y ago
https://b.hatena.ne.jp/entry/s/www.wigmoremedical.com/midia/v-vor-vrg-kf.html https://b.hatena.ne.jp/entry/s/www.wigmoremedical.com/midia/...
fsdgd 6y ago
https://www.wigmoremedical.com/midia/v-vor-vrg-kf.html https://www.wigmoremedical.com/midia/v-vor-vrg-kf.html
sadfghjfghjkl 6y ago
"https://b.hatena.ne.jp/entry/s/www.wigmoremedical.com/midia/v-vor-vrg-kf.html https://b.hatena.ne.jp/entry/s/www.wigmoremedical.com/midia/...
deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
alihamuh 6y ago
Great post. Thanks for the info.
mjgs 6y ago
Hasn’t this already been addressed in this post (1st Oct)? GitHub Actions: Deprecating set-env and add-path commands https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands https://github.blog/changelog/2020-10-01-github-actions-depr...

624 points by looperhacks 6y ago

Wonder when the security industry comes up with better rating system for vulnerabilities. The rating for this one is all over the place...

est31 6y ago

Wow this looks quite serious. I hope that github will at least provide a way to turn these commands off entirely. I hope they also fix the stupid authwalling of the execution logs that happens for public repos. The code is public, but the logs need an account? There is barely any security benefit in doing this for public repos, but it does make it harder for me to check PRs that I make to open source projects while not being logged in (e.g. from my phone).

whymarrh 6y ago
At their scale I do believe there’s a benefit to authenticating to see logs: a lot of people scrape GitHub for secrets. CI logs are at high risk for user error, errors where a user unintentionally marks something as non-secret when it should’ve been secret. Putting these logs behind auth feels like an easy filter for some scraping.
- est31 6y ago
  That's a good point I guess. Per default, secrets are redacted by github in the logs, but some might slip through, e.g. if only a part of them is printed. Doesn't make me really happy though, I don't want to have to use an account :).
- holman 6y ago
  Yeah, definitely. When I was at GitHub we were seeing secrets getting lifted from public pushes and tried within 7 seconds or so, if I recall correctly, and this was five years ago. This was a big reason why there’s a real scanning API now for service providers to be informed if a secret leaks. By the time a human discovers their mistake it’s usually far too late.
nexuist 6y ago
The code is public but the secrets are obviously not, and if they are ever put in the logs then that's a security issue. Even if the action isn't logging credentials directly, it may be logging privileged information (e.g. S3 bucket names or contents) that can only be accessed with privileged credentials. Therefore it makes sense to keep the logs private. Maybe there should be a setting for Action owners where they can set logs public if they think it would help contributors.

chrisrpatterson 6y ago

https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/ https://github.blog/changelog/2020-10-01-github-actions-depr...

deleted 6y ago

[deleted]

jasonmp85 6y ago

Is anyone who’s used Actions surprised? They launched VERY unpolished and this sort of lack of care seems par for the course, tbh.

bredren 6y ago
I am in the process of migrating a client’s CI to GA and have seen rough edges. One example is there is no native way to serialize workflow execution. This means if you run two commit-based triggered actions in quick succession, you can have tests pass that should not. (They run against the wrong checked out code!) I had this happen in front of the client, where it appeared a unit test of assertTrue(False) passed! It was so undermining of GA I had to chase down the exact status and prove ability to avoid (using Turnstyle) to the client. I wrote an email to the GA team specifically telling this story and expressing my concern. Another is scheduled actions, where even if you specify a cron, the action has a window of several minutes when a GitHub runner will execute it, and there are cases where the window is missed and the action is not run! This isn’t documented, though GitHub recently made the docs community-editable, possibly realizing there is too much to do. All that said, the product is still relatively new, and even with these rough edges it is __really__ good. Having CI definition alongside code, with free-ish VMs available at all times, (except when it goes down,) and results all inside GitHub is amazing. Also, the self-hosted runners are amazing, as they allow some really simple matrix definition to run CI on custom HW and network environments. If it weren’t for all the recent negative attention recently, I would have said GH was doing really well, inclusive of GitHub Actions.
- amscanne 6y ago
  Are you joking about the self hosted runners? The fact that you can’t: 1) Apply any security policy for runners (e.g. require a label before running a PR) 2) Have runners quit after a single job so you can build ephemeral runners 3) Build your own runner against an open API ... means that self hosted runners are non starter for anything open source. It’s like they had to try hard to make the architecture that obtuse and closed. It’s unclear if it’s really poor/design or an active attempt to somehow drive the business.

nixgeek 6y ago

Interesting that Project Zero considers this “High” severity, and GitHub says “Moderate”. Legitimate disagreement between security teams, or is GitHub applying spin and trying to downplay the severity, in light of there being no easy fix?

dathinab 6y ago
If you look at how GitHub operates wrt. GitHub releases or GitHub actions it becomes clear (to me) that they have very different standards to certain security aspects then me or e.g. project zero. So I would argue it's a case about having less strict standards wrt. Security vulnerabilities which requir your to pull in corrupted (or very careless) 3rd party code.
arianvanp 6y ago
and the CVE got registered as "Low" https://nvd.nist.gov/vuln/detail/CVE-2020-15228 https://nvd.nist.gov/vuln/detail/CVE-2020-15228 So what is it? High, Moderate, or Low? Super confusing
- csnover 6y ago
  I’ve found that consistent CVSS scoring is practically impossible, both because many criteria are defined in ambiguous or confusing ways, and because rating any obvious knock-on effects is proscribed by the official scoring rules[0][1]. For example, if a software component has a vulnerability which allows an attacker to steal admin credentials by exfiltrating a password file, the fact that this would allow the attacker to then have admin access to do whatever they wanted in other software that shares credentials doesn’t matter when calculating the core score of the vulnerability. (As a user, you would need to calculate the “environmental score” in the calculator[2] to decide what your score is given whatever configuration you/your company uses.) Additionally, if some software enables privilege escalation at the OS level by misusing APIs, this escalation is not a “scope change” unless the software tried to create a separate security authority for itself by having some form of sandbox or authentication, so a vulnerability of this kind in software that has no added security mechanisms at all will be scored lower than a vulnerability in software which has implemented some additional access control which was bypassed. I’m not saying that the limits are nonsensical since without them you’d end up with a whole lot of Critical base scores, but in my limited experience the CVSS core score is pretty worthless when it comes to getting a sense of how bad a vulnerability actually is and I don’t know how security analysts cope with this system. [0] https://www.first.org/cvss/v3.0/user-guide https://www.first.org/cvss/v3.0/user-guide [1] https://www.first.org/cvss/v3.0/examples https://www.first.org/cvss/v3.0/examples [2] https://www.first.org/cvss/calculator/3.0 https://www.first.org/cvss/calculator/3.0

dathinab 6y ago

Not really surprised tbh. GitHub actions while very flexible are also prone to ask kinds of problems. E.g. versions of actions on the market place are not pinned so people can republish different code under the same version. So you need to pin to a git commit hash which is much easier to get wrong and also easier to abuse for social engineering. And that's assuming you trust git commit hashes, if not you need to fork actions just to make sure the public code you reviewed doesn't get changed without new reviewes .

zomgwat 6y ago
I've recently started using GitHub Actions and I was very disappointed to learn how few "native" actions there are. I don't want to introduce yet another avenue of third-party code via the GitHub Actions marketplace.
- Arnavion 6y ago
  My GH workflows are a thin layer over the `run` action that runs shell scripts. Not only does my CI not have to rely on third-party actions, it's also extremely portable between CI providers, can be tested locally, and is infinitely more flexible. (Apart from having run actions, the only other actions I use are the upload/download-artifact ones to share binaries between jobs, and the matrix feature for parallelizing runs.) But to be clear, the kind of vulnerability that the P0 issue is talking about is not necessarily from malicious actions. As the issue itself says, the vuln also happens from untrusted input being processed by a trusted action, like issues and PRs.
  - bastardoperator 6y ago
    This is what every build should be. If I can't build it locally and in CI using the same process, I consider it broken. I tend to use Makefile, but what your describing is what I consider best practice.
  - dathinab 6y ago
    I plan to do something similar soon ;=) I played around with some actions wrt. commit signing and originally wrote a action a verify commit and tag signatures (in the way I need ;=) ). But now I plan to port it to a more general interface I can call from a GH action.

Pfhreak 6y ago

Interpreting text for commands has an interest parallel to a major World of Warcraft bug. It turns out that you could run arbitrary lua code in your chat terminal by using the "/run ..." command. Enterprising thieves figured out that one of the things you could do was redefine functions, in particular the 'RemoveExtraSpaces' function that gets run against every chat line. If you set RemoveExtraSpaces=RunScript you've told WoW to execute every chat message displayed in your terminal. Attackers just have to trick you into writing "/run RemoveExtraSpaces=RunScript" into your terminal, and after that all they had to do was chat code to you and it would execute. They basically get full control of anything lua can do in your game -- including sending messages to your friends asking them to also run the attack command.

est31 6y ago
Kinda reminds me of the "don't type anything into the browser console" warnings that sites like facebook etc. emit so that people don't type in commands strangers gave them, believing it allows them "hack" others while in reality it only hacks themselves.
diarrhea 6y ago
That is super interesting. When did this happen? What is the worst that did happen through this?
- Pfhreak 6y ago
  I'm not sure. I do know that you could, for instance, have your gold sent to a scammer with little to no recourse. It may be possible that this bug still exists because it's one of those "bugs that is also a feature".
  - lilyball 6y ago
    Blizzard's response was to modify /run to pop up a dialog first warning you that /run is used by scammers and requires you to confirm that you know what you're doing.
- 6y ago

abcxjeb284 6y ago

It’s pretty absurd for github to suggest that you should go through multiple steps to disable commands to log untrusted output. [1] Poor form to expect developers to understand and check for a new way that they need to sanitize their input rather than Github fixing it (possibly in a backwards incompatible way) At a minimum they should provide a shell script (`show $XYZ`) and a js function that handles generating those tokens and enabling/disabling workflow commands for you. [1] https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/ https://github.blog/changelog/2020-10-01-github-actions-depr...

hn_throwaway_99 6y ago
IMO at the very least they should have an org-wide option in settings to disable command interpretation.
- mjgs 6y ago
  Yeah the ability to disable the insecure commands is crucial. That should be the first thing they should do. Probably a lot of people aren’t even using this functionality in their workflows anyway. I think the only one of these commands that I actually use is setting outputs, and I don’t even use that very often. There should also be a way to see which commands have been triggered via stdout, so you can at least see what happened if something malicious happens.

dhagz 6y ago

My employer recently decided to move from using self-hosted Jenkins servers living behind a VPN to GitHub actions. I wonder how much vulnerabilities like this were weighed against the reduced maintenance burden.

Karliss 6y ago

I surprised that Project Zero waited the last 30 days for disclosure since Github themselves publicly announced at 1. October that the whole searching for magic strings in stdout as RPC channel is broken by design. When I saw the announcement month ago I assumed that Github fully disabled the reading commands from stdout mechanism but looking at current docs it seems that they disabled only 5 most easily exploitable out of 14 commands which can be called using stdout. This seems like typical "we fixed your POC doesn't work anymore" but the main cause remains. As can be sometimes seen with hardware vendors "we fixed hard-coded backdoor password, now there is new hard-coded backdoor password".

guitarbill 6y ago
I'm not sure. When using `::set-output name=spam::eggs`, you seem to have to refer to it via `steps.foo.outputs.spam`. So if you keep steps that set outputs very simple, and/or the set-output echo is the last command of the step, then it should mitigate that? Is the general consensus that `$GITHUB_ENV` offers less attack surface, despite not being scoped to a step? Some guidance on this would be great from Github's side. Also appreciated: a way of turning off magic stdout strings in the YAML, and a way to set outputs via a more robust mechanism, and/or to scope which steps can modify which environmental variables. I get not wanting to break existing CI runs, but giving the people who want it an option to harden the CI proactively would be great. I don't really care if this would break some existing actions, either - seems fine to do for opt-in?
- mjgs 6y ago
  Most useful comment so far in this thread. I’ve been using $GIT_ENV for setting environment variables, and though I read about the stdout commands I never got to using them. It’s one of those “but of course” moments realising that any cli tool can trigger commands with it’s stdout. As you mentioned setting outputs is rather crucial, I guess using stdout is the only way todo that. There are definitely a lot of security things to consider when using actions. I read the code of any community actions I use but, especially after reading this alert, I feel like there are a lot of places where malicious code could cause damage.

deleted 6y ago

[deleted]

kanox 6y ago

Running CI against arbitrary code is extremely difficult to do securely and I'm not sure anyone does more than a token effort. Test infrastructure is not designed to run against malicious code. One of the examples dumps environment variables which may contain secrets by injecting code through a github action. What's stopping code inside a PR from sending the same info over email?

takluyver 6y ago
I believe CI on pull requests runs without the secrets, to avoid precisely that issue.
- amscanne 6y ago
  Yes, the problem is that GitHub did not seem to consider that “malicious input” can include any content that is provided and parsed in some way. Unfortunately, all of stdout is parsed, and often includes things like issue titles, descriptions, commit messages, etc.

sounds 6y ago

I've often heard HNers state that Project Zero is unwilling/uninterested in protecting the interests of the vulnerable host, but this seems like an excellent case study where Github, the host, just doesn't think this is a big deal. I think that applies to any host who doesn't patch their vulnerabilities - if they thought it was a big deal, they'd do something about it. Actions speak. Quoting from the post: 2020-07-21 Report sent to Github and triaged by Github security team. Disclosure date is set for 2020-10-18. 2020-08-13 Project Zero requests a status update. 2020-08-21 Github say that they are are still working on a fix and a deprecation plan 2020-10-01 Github issued an advisory[1] deprecating the vulnerable commands, assigned CVE-2020-15228, and asked users to patch and update their workflows. 2020-10-12 Project Zero reach out and proactively mentions that a grace period is available if Github wants more time to disable the vulnerable commands. 2020-10-16 Github requested the additional 14 day grace period, with the hope of disabling the vulnerable commands after 2020-10-19. 2020-10-16 Project Zero grants grace period, new disclosure date is 2020-11-02. 2020-10-28 Project Zero reaches out, noting the deadline expires next week. No response is received. 2020-10-30 Due to no response and the deadline closing in, Project Zero reaches out to other informal Github contacts. The response is that the issue is considered fixed and that we are clear to go public on 2020-11-02 as planned. 2020-11-01 Github responds and mentions that they won't be disabling the vulnerable commands by 2020-11-02. They request an additional 48 hours, not to fix the issue, but to notify customers and determine a "hard date" at some point in the future. 2020-11-02 Project Zero responds that there is no option to further extend the deadline as this is day 104 (90 days + 14 day grace extension) and that the disclosure will be today.

zenexer 6y ago
Indeed, if that timeline is to be believed—and I have no reason to doubt it—it’s pretty damning. GitHub just kept putting it off. When they asked to delay the disclosure indefinitely at the very last second, of course they were told that would be unacceptable.
- naikrovek 6y ago
  I dunno. I've worked in many places that ABSOLUTELY had the desire, willingness, money, time, and ability to do something, and were still unable to make that thing happen, for reasons that can't be spoken about. While it is true that if an entire organization wants something done that it will happen, it is not true that inaction by an organization always signifies unwillingness or inability to act. It's easy to sit on the sidelines and throw tomatoes at the parties involved. It is another matter entirely when you are hamstrung by internal process or a specific power-tripping and ignorant person in management.

nickcw 6y ago

GitHub has been nagging about needing to fix this for a week or two. Just in case anyone is interested here is what the fix to rclone's build workflow looked like: https://github.com/rclone/rclone/commit/a152351a71ce21090a3be1944e2c016682d6e525 https://github.com/rclone/rclone/commit/a152351a71ce21090a3b...

strbean 6y ago
Love rclone, thanks for all your hard work!

withinboredom 6y ago

Does anyone actually trust their CI for running PRs on a public repo (in repo or forks)? I was taught from my first forays into commercial software never to trust CI's until after a review (even if it's another coworker).

ffpip 6y ago

Every time I visit bugs.chromium.org in Firefox, something weird happens with the fonts and text selection. This doesn't happen in Chromium Are they publicly disclosing/warning about a bug in FF, or is it just some other problem?

yellowfool 6y ago
How do your font issue relate to a github security issue ?
- ffpip 6y ago
  No this is not related to that. I just posted a comment because many people will visit the site I mentioned, probably using Firefox. I am able to reproduce this 'bug'. It's like mouse hijacking. Not able to select text on bugs.chromium.org when using Firefox.

junon 6y ago

I remember thinking "yeah this isn't going to be safe..." when I saw how Github actions parsed those commands. It just seemed like a huge security vulnerability waiting to happen. Glad to know my gut still sort-of works. Not sure who thought this was a good idea but I can think of a handful of other ways off the top of my head that this could have been implemented in a more secure way.

prepend 6y ago

I’m not happy with the security on GitHub actions. The current options I have for my orgs is to either allow all actions, allow actions hosted in the org, or whitelist specific actions by name. It’s not clear to see actions and any testing or certification they may have. Even github built actions might be under an individual’s account. So it takes quite a bit of effort to check out an action to see if it’s safe or dangerous. And I worry about actions a little more because they do things like copy files, deploy systems, and other things where I wouldn’t want rogue code interjecting malware. Currently I require all actions to be hosted in the org so devs have forked actions into our org. That at least makes a developer vouch for a particular version and makes it harder for an update to that action to break something. I think they need an equivalent of “gold actions” endorsed by GitHub. Or a vetted repo like pypi or cran or something else that actually has some controls around versions and changes.

Kliment 6y ago
I suggest you fork any actions you depend on.
- a_t48 6y ago
  Given that actions can reference other actions, is this even feasible? You'd have to go and fork all deps, and then if you ever want to pull upstream, repatch.
  - mjgs 6y ago
    I don’t think that’s the case. As far as I have read, a multi-step action that you create cannot use a community action. I was wanting to do this so as to be able to break up some lengthy workflows into reusable pieces, so I figured I’d create some actions, but when I tried to use for example the standard checkout action, inside my action, it failed. I emailed support and they said that actions inside actions was not supported. Perhaps you are referring to something else? A different type of action?
    - a_t48 6y ago

Animats 6y ago

As the runner process parses every line printed to STDOUT looking for workflow commands, every Github action that prints untrusted content as part of its execution is vulnerable. What could possibly go wrong?

deleted 6y ago

[deleted]

curryhoward 6y ago

A small plug: Toast (https://github.com/stepchowfun/toast https://github.com/stepchowfun/toast) has already introduced a patch for this vulnerability (CVE-2020-15228), so arbitrary user code running inside Toast will not be able to trigger this vulnerability when run via a GitHub Action. I highly recommend using Toast for your CI jobs (with or without GitHub Actions), not just for this security fix but also because it also allows you to run your CI jobs locally. It just runs everything inside Docker containers, so it works the same in CI, on your laptop, on your coworker's machine, etc.

paulstovell 6y ago

Octopus Deploy and TeamCity both parse output looking for commands in the same way (called Service Messages). I believe it is a common pattern in all CI servers. The issue here doesn’t seem to be parsing the output, but what the parsed commands can do. In Octopus there is no way to use them to run commands or set environment variables though. I can’t speak for TeamCity. Update: TeamCity's documentation page about service messages: https://www.jetbrains.com/help/teamcity/service-messages.html https://www.jetbrains.com/help/teamcity/service-messages.htm... The commands are quite specific though: test results, build numbers, etc. no generic commands like running commands or setting environment variables.

Serow225 6y ago

I created this ticket six months ago about the the inability to mask secrets that are step outputs (like from Terraform) and them subsequently leaking to logs, no action taken yet... If anyone has influence to raise visibility of this (Twitter, GitHub, whatever) that would be appreciated :) https://github.com/actions/runner/issues/475 https://github.com/actions/runner/issues/475

based2 6y ago

https://about.gitlab.com/releases/2020/11/02/security-release-gitlab-13-5-2-released/ https://about.gitlab.com/releases/2020/11/02/security-releas... GitLab Security Release: 13.5.2, 13.4.5, and 13.3.9

luhn 6y ago
I don't see anything to indicate this is related?

reilly3000 6y ago

I’ve always worried about env leakage, especially in npm packages. GHA does a decent job of masking but there is no stopping arbitrary code. One thing they could do is pass log output against known GHA Secrets and run an additional masking on the output string. To put this into context everything about this attack is logged in the open I believe. If that shows up in log output, you’ll know who did it and that everyone can see it - and proceed with rotating secrets promptly and report the hack to GitHub.

fsdgd 6y ago

https://b.hatena.ne.jp/entry/s/www.wigmoremedical.com/midia/v-vor-vrg-kf.html https://b.hatena.ne.jp/entry/s/www.wigmoremedical.com/midia/...

fsdgd 6y ago

https://www.wigmoremedical.com/midia/v-vor-vrg-kf.html https://www.wigmoremedical.com/midia/v-vor-vrg-kf.html

sadfghjfghjkl 6y ago

"https://b.hatena.ne.jp/entry/s/www.wigmoremedical.com/midia/v-vor-vrg-kf.html https://b.hatena.ne.jp/entry/s/www.wigmoremedical.com/midia/...

deleted 6y ago

[deleted]

deleted 6y ago

[deleted]

alihamuh 6y ago

Great post. Thanks for the info.

mjgs 6y ago

Hasn’t this already been addressed in this post (1st Oct)? GitHub Actions: Deprecating set-env and add-path commands https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands https://github.blog/changelog/2020-10-01-github-actions-depr...