20 ms·

Tj-actions/changed-files GitHub Action Compromised – used by over 23K repos

• varunsharma07 2y ago
  What Happened? • The compromised Action executes a Python script that dumps CI/CD secrets from the Runner Worker process. • Multiple v35 tags were modified four hours ago, indicating a recent supply chain attack. • The malicious behavior can be observed in StepSecurity Harden-Runner insights, showing the Action downloading and executing an unauthorized script.
• varunsharma07 2y ago
  https://github.com/tj-actions/changed-files/issues/2463 https://github.com/tj-actions/changed-files/issues/2463
• jasonthorsness 2y ago
  Wow that's scary, they updated tons of tags to an offending random commit. With the way repositories are included in automation and the fact that this adjusted the tags of older versions (so not requiring an upgrade) this sort of attack can have a huge impact very quickly :(. Maybe GitHub should have some kind of security setting a repo owner can make that locks-down things like old tags so after a certain time they can't be changed.
  • SAI_Peregrinus 2y ago
    Tags are a git concept, not a GitHub concept. Tags provide human-readable names to commits. They're intended to be changeable, to allow things like a "latest" tag pointing to the latest release. Tags aren't versions, commit hashes are.
• kurmiashish 2y ago
  Disclaimer: I am a co-founder of StepSecurity. StepSecurity Harden-Runner detected this security incident by continuously monitoring outbound network calls from GitHub Actions workflows and generating a baseline of expected behaviors. When the compromised tj-actions/changed-files Action was executed, Harden-Runner flagged it due to an unexpected endpoint appearing in the network traffic—an anomaly that deviated from the established baseline. You can checkout the project here: https://github.com/step-security/harden-runner https://github.com/step-security/harden-runner
  • shawabawa3 2y ago
    A simpler method to detect this would be to store GitHub action tag hashes and freeze an action if any tag is changed
  • cyrnel 2y ago
    The advertising in this article is making it actively difficult to figure out how to remediate this issue. The "recovery steps" section just says "start our 14 day free trial". The security industry tolerates self-promotion only to the extent that the threat research benefits everyone.
    • kurmiashish 2y ago
      Thank you, cyrnel, for the feedback! We are trying our best to help serve the community. Now, we have separate recovery steps for general users and our enterprise customers.
      • cyrnel 2y ago
        Thanks for the edit! In "incident response mode" every moment counts!
• mubou 2y ago
  In recent years, it's started to feel like you can't trust third-party dependencies and extensions at all anymore. I no longer install npm packages that have more than a few transitive dependencies, and I've started to refrain from installing vscode or chrome extensions altogether. Time and time again, they either get hijacked and malicious code added, or the dev themselves suddenly decides to betray everyone's trust and inject malicious code (see: Moq), or they sell out to some company that changes the license to one where you have to pay hundreds of dollars to keep using it (e.g. the recent FluentAssertions debacle), or one of those happens to any of the packages' hundreds of dependencies. Just take a look at eslint's dependency tree: https://npmgraph.js.org/?q=eslint https://npmgraph.js.org/?q=eslint Can you really say you trust all of these?
  • scrapcode 2y ago
    Are there examples of these types of actions in other circles outside of the .NET ecosystem? I knew about the FluentAssertions ordeal, but the Moq thing was news to me. I guess I've just missed it all.
    • do_not_redeem 2y ago
      node-ipc is a recent example from the Node ecosystem. The author released an update with some code that made a request to a geolocation webservice to decide whether to wipe the local filesystem.
      • mubou 2y ago
        Yeesh. Found an article for anyone interested: https://snyk.io/blog/peacenotwar-malicious-npm-node-ipc-package-vulnerability/ https://snyk.io/blog/peacenotwar-malicious-npm-node-ipc-pack... I like this comment from u/mailto_devnull (https://www.reddit.com/r/node/comments/tg451e/do_not_use_nodeipc_lib/ https://www.reddit.com/r/node/comments/tg451e/do_not_use_nod...): Where do I stand on the war? I stand with Ukraine. Where do I stand on software supply chain issues? I stand with not fucking around with the software supply chain.
    • sanex 2y ago
• dan_manges 2y ago
  GitHub Actions should use a lockfile for dependencies. Without it, compromised Actions propagate instantly. While it'd still be an issue even with locking, it would slow down the rollout and reduce the impact. Semver notation rather than branches or tags is a great solution to this problem. Specify the version that want, let the package manager resolve it, and then periodically update all of your packages. It would also improve build stability.
  • jasonthorsness 2y ago
    Since they edited old tags here … maybe GitHub should have some kind of security setting a repo owner can make that locks-down things like old tags so after a certain time they can't be changed.
    • CaliforniaKarl 2y ago
      In your GitHub Actions YAML, instead of referencing a specific tag, you can reference a specific commit. So, instead of … uses: actions/checkout@v4 … you can use … uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
      • OptionOfT 2y ago
        That still doesn't help when the action is a docker action only marked with a tag. So you need to check the action.yml itself to see if it has a sha256 pinned (in the case it uses Docker).
  • mceachen 2y ago
    GitHub actions supports version numbers, version ranges, and even commit hashes.
    • frenchtoast8 2y ago
      The version numbers aren't immutable, so an attacker can just update the versions to point to the compromised code, which is what happened here. Commit hashes are a great idea, but you still need to be careful: lots of people use bots like Renovate to update your pinned hashes whenever a new version is published, which runs into the same problem.
• jasonthorsness 2y ago
  I think this article from earlier today was the discoverer who opened the issue https://news.ycombinator.com/item?id=43367987 https://news.ycombinator.com/item?id=43367987
• simonw 2y ago
  I've always felt uncomfortable adding other people's actions to my GitHub workflows, and this is exactly the kind of thing I was worried about. I tend to stick to the official GitHub ones (actions/setup-python etc) plus the https://github.com/pypa/gh-action-pypi-publish https://github.com/pypa/gh-action-pypi-publish one because I trust the maintainers to have good security habits.
  • 3eb7988a1663 2y ago
    That's exactly where I stand, and I feel partially vindicated by this outcome. There are so many useful Github actions made by randos, but I am not adding more unvetted dependencies to my project. I will unhappily copy and paste some useful code into my project rather than relying upon yet another mutable dependency.
• mirashii 2y ago
  Not the first time this particular action has had a vulnerability, either. https://nvd.nist.gov/vuln/detail/CVE-2023-51664 https://nvd.nist.gov/vuln/detail/CVE-2023-51664
  • zahlman 2y ago
    I could have sworn that I've seen other GitHub Actions vulnerabilities that worked the same way, too. And/or HN submissions talking about this specific kind of vulnerability, the standard mitigation strategies, etc. Feels like the same kind of problem as SQL injection, where everybody kinda knows about it and some people are actively aware and there are standard ways to avoid it but it still happens all the time anyway. Might also be a good time to mention I'm really not a fan of YAML.
• nextts 2y ago
  Well timed for the weekend. Some teams may notice Monday morning.
• tsujamin 2y ago
  How does SBOM and such account for this? If you’re a package maintainer, do you need to include CI pipeline plugins, their dependencies, going down as far as the pipeline host, in your security-relevant dependencies? Hard problems :/
  • captn3m0 2y ago
    Most recommendations treat SBOM as the “ingredients” and are he build dependencies such as GitHub Actions as the recipe. However, I think the GitHub SBOM features include GitHub Actions as dependencies, but that is merely a side-effect of their Dependabot heritage.
• neuroelectron 2y ago
  CI was sold as a solution for engineers having too much sovereignty. So that's what they got.
• xyst 2y ago
  A company I worked at went all in on GH too. The internal gh team probably going to do a fire drill this whole weekend and app teams forced to rotate all secrets and credentials. Fortunately don’t have to deal with that shit anymore
  • frenchtoast8 2y ago
    My day job is also in the middle of moving everything to Github Actions, so this is fun. But in my case, we aren't affected by this vulnerability because it could only be exploited by workflows with public logs, and currently my company only uses Github Actions for private repositories.
    • werrett 2y ago
      I mean maybe! But only if you've removed all of the usage of this compromised `tj-actions/changedfiles` action, across all your repos and their branches. Otherwise, if you continue to use it and it will run anytime there has been a push. Potentially on any branch, not just `main`! Depending on your GH config. Unless you've blocked `tj-actions/changed-files` you're banking on the bad actor not coming back tonight and making malicious commit that exfils those secrets to pastebin.com.
  • arionmiles 2y ago
    It's possible to whitelist actions on an org level. You can whitelist - all actions from a specific org (e.g. actions/*) - a specific action (e.g. actions/setup-go) - a specific version of a specific action (e.g. actions/setup-go@commit-sha) Any workflow attempting to use actions outside of the whitelist will simply fail to start up.
• random17 2y ago
  I wish Github required some sort of immutability for actions by default as most package managers do, either by requiring reusable actions to be specified via commit hash or by preventing the code for a published tag to be changed. At the moment the convention is to only specify the tag, which is not only a security issue as we see here, but may also cause workflows to break if an action author updates the action.
  • wutwutwat 2y ago
    You can target `some/action@commithash` already, that's up to you. You're also free to fork or clone each action you use, vet the code, and consume your fork in your workflows. You can also disable the use of third party actions at an org level, or approve them on a case-by-case basis. This all depends on your threat model and risk tolerance, it's not so much a GitHub problem. There will always be bad code that exists, especially on the largest open source code hosting platform. You defend against it because that's more realistic than trying to eradicate it.
    • deleted 2y ago
      [deleted]
    • arianvanp 2y ago
      One problem with this is that actions can be Composite and call arbitrary other actions. So only if you use actions that themselves lock everything by commit for the actions they depend on you're safe.
      • wutwutwat 2y ago
        You just described a supply chain, and the risks that come with them, which is something every dep management system is dealing with, rubygems, npm, etc Again, it all comes down to your risk tolerance. There's a certain level of trust built into these systems.
    • chuckadams 2y ago
      Someone elsewhere suggested a lockfile, which seems a pretty obvious solution in hindsight. I'm fine with commit hashes, but the UX is terrible and consists of pasting the action into into StepSecurity's thingie, when this is something that GH should have built in.
• dboreham 2y ago
  Good policy to fork all actions not owned by a reputable organization (e.g. github themselves).
  • frenchtoast8 2y ago
    You can combine this with a policy that blocks third party actions from being used across your organization. https://docs.github.com/en/organizations/managing-organization-settings/disabling-or-limiting-github-actions-for-your-organization#allowing-select-actions-and-reusable-workflows-to-run https://docs.github.com/en/organizations/managing-organizati...
• xyst 2y ago
  Pretty good timing on the attacker. https://github.com/tj-actions/changed-files/tags?after=v35.9.3 https://github.com/tj-actions/changed-files/tags?after=v35.9... Most folks around the world signed off. B-squad probably left cleaning up remaining tasks or just fucking around with co-workers and pondering the weekend. Most GH actions run on a schedule (ie, backups of db, connecting to blob storage services). Attacker(s) likely to extract plenty of secrets and exfil data before the alarms get triggered (if any) at companies. The next data dumps are going to be wild.
• harrisi 2y ago
  It's always been shocking to me that the way people run CI/CD is just listing a random repository on GitHub. I know they're auditable and you pin versions, but it's crazy to me that the recommended way to ssh to a server is to just give a random package from a random GitHub user your ssh keys, for example. This is especially problematic with the rise of LLMs, I think. It's the kind of common task which is annoying enough, unique enough, and important enough that I'm sure there are a ton of GitHub actions that are generated from "I need to build and deploy this project from GitHub actions to production". I know, and do, know to manually run important things in actions related to ssh, keys, etc., but not everyone does.
  • sestep 2y ago
    The crazier part is, people typically don't even pin versions! It's possible to list a commit hash, but usually people just use a tag or branch name, and those can easily be changed (and often are, e.g. `v3` being updated from `v3.5.1` to `v3.5.2`).
    • harrisi 2y ago
      You and someone else pointed this out. I only use GitHub-org actions, and I just thought that surely there would be a "one version to rule them all" type rule.. how else can you audit things? I've never seen anything recommending specifying a specific commit hash or anything for GitHub actions. It's always just v1, v2, etc.
      • mcpherrinm 2y ago
        OpenSSF scorecard flags dependencies (including GitHub actions) which aren’t pinned by hash https://scorecard.dev/ https://scorecard.dev/ https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies https://github.com/ossf/scorecard/blob/main/docs/checks.md#p...
      • evntdrvn 2y ago
        it is documented as recommended here fwiw: https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions#using-third-party-actions https://docs.github.com/en/actions/security-for-github-actio...
• naikrovek 2y ago
  I’ve been saying for a while that there aren’t supply chain problems when the supply chain is the problem. I’m getting to the point where I feel that library use at all should be frowned upon, unless it is your own library, with obvious exceptions for the most widely used things like encryption and authentication. None of these things are particularly difficult, people just don’t want to do them “oh noes my velocity”
  • eviks 2y ago
    And writing your library should be frowned upon because you can easily add a major security vulnerability with your own two hands
    • naikrovek 2y ago
      If my library is left-padding a string with spaces, I don’t know how that could possibly introduce a major security vulnerability at all. People write the trusted, secure code, and people retire from that work, and new people need to come in and do the work. Inexperienced people are going to be writing code no matter what. So, you are either saying that no one should write new libraries because a major security vulnerability could be introduced by their hands, or you are saying that all libraries should be written by hands which will not introduce a major security vulnerability, and neither of those is at all feasible.
      • eviks 2y ago
        Well, broaden your horizon a bit from an irrelevant example of a "library" of a few lines to a real one a of few thousand lines, then you might know. Also those same people also write insecure code before they retire. Experienced people write security bugs all the time. What I'm saying is if you imagine a world where there is so much time to be wasted rewriting the same library a thousand times, you could try to imagine spending a small share of that time hardening the supply chain
        • naikrovek 2y ago
• londons_explore 2y ago
  > await exec.getExecOutput('bash', ['-c', `echo "aWYgW1sgIiRPU1RZUEUiID09ICJsaW51eC1nbnUiIF1dOyB0aGVuCiAgQjY0X0JMT This malicious code isn't hard to recognise... Surely someone can run an LLM over all code in GitHub and just ask it 'does this code looks like it's blatantly trying to hide some malicious functionality'? Then review the output and you'll probably discover far more cases of this sort of thing.
  • gkbrk 2y ago
    What if before the command, there is also a code comment that says "this is not malicious, it has been manually verified by the engineers" and the LLM just believes it?
• edoceo 2y ago
  Noob question: is it possible to version lock these things? Could one "vendor" these tools into a fork and use that in the pipeline? Maybe it's one of those possible but crazy endeavour?
  • werrett 2y ago
    You can pin GitHub Actions to specific versions or specific commits. But note you can change version tags arbitrarily. In this specific case, the bad actor changes all of the version tags to point to their malicious commit: https://github.com/tj-actions/changed-files/tags https://github.com/tj-actions/changed-files/tags So to avoid that you'd have to pin your GitHub Action to specific commits as outlined in this SO post: https://stackoverflow.com/a/78905195 https://stackoverflow.com/a/78905195
    • zahlman 2y ago
      > In this specific case, the bad actor changes all of the version tags to point to their malicious commit: https://github.com/tj-actions/changed-files/tags https://github.com/tj-actions/changed-files/tags This required compromising the entire repository, yes? It can't be explained as the maintainer being tricked into merging something malicious?
      • ImPostingOnHN 2y ago
        The repo looks like it uses itself in its workflows, so it's possible that the commit being merged resulted in the necessary credentials being leaked to the attacker.
        • rognjen 2y ago
          There doesn't seem to be a PR for the commit though.
      • werrett 2y ago
        Yes. It was probably a maintainer's creds being compromised. The [malicious commit is masquerading as a commit from Renovate](https://github.com/tj-actions/changed-files/commit/0e58ed8671d6b60d0890c21b07f8835ace038e67#diff-3d2b59189eeedc2d428ddd632e97658fe310f587f7cb63b01f9b98ffc11c0197R2999 https://github.com/tj-actions/changed-files/commit/0e58ed867...)((https://github.com/apps/renovate https://github.com/apps/renovate) but it's not a `verified` commit (and so it's trivial for a bad actor to masquerade as them). https://stackoverflow.com/questions/67609381/why-do-all-my-github-commits-show-as-unverified https://stackoverflow.com/questions/67609381/why-do-all-my-g...
• aptsolution 2y ago
  [dead]
• nomilk 2y ago
  I'd used GitHub Actions for at least 6-12 months before even realising <thing>/<thing> was not something that got parsed by the action (like a namespace and method/function), but was simply a reference to a github user name and repo. That whole object should really have been called a 'repo', because that's what it is, and that would alert users to use extreme caution whenever using one that wasn't created by themselves.
  • Griffinsauce 2y ago
    Another fun fact related to this is that when someone changes their github username, their actions break. Just a hilariously bad design IMO. There should be a repository in between with immutable versions.
• londons_explore 2y ago
  So this dumps env to stdout using some obfustucated code? And then relies on the fact logs are viewable publicly so the attacker can go scrape your secrets. If so, why did they use obfustucated code? Seems innocuous enough to load env into environment vars, and then later to dump all env vars as part of some debug routine. Eg. 'MYSQL env var not set, mysql integration will be unavailable. Current environment vars: ${dumpenv}'
  • werrett 2y ago
    No idea. But they didn't do a great job -- they broke the action, which caused build failures that people were going to notice. The malicious commit only landed at 09:57 PDT today (March 14) in one specific action (out of a number that is quite popular). Maybe they were planning on coming back and doing proper exfil?
  • mceachen 2y ago
    Presumably the cracker: 1. spoofed an account whose PRs were auto-merged (renovate[bot]) 2. found that `index.js` was marked as binary, and knew that GitHub is "helpful" (for the exploit), and hides diffs in the PR for that file by default 3. shoved the chunk of base64 wayyyy down the commit, so the maintainer had review fatigue by the time they scrolled. Having "memdump.py" in the commit in plaintext would certainly highlight the exploit more than the b64 string.
    • captn3m0 2y ago
      Sounds about right to me. We can use a few knowns about GitHub IAM to deduce a few things: 1. There are no deleted PRs or Issues on the repo (2461..2463 are all valid refs) 2. A legitimate `Renovate[Bot]` dep bump would have filed a PR. Last such PR was 5 days ago, and is presumably not the source for this. (I haven't gone through every dep change, but doesn't look like it). 3. That leaves us with the 0e58ed867 commit, which has to be a spoofed commit, since it doesn't belong to a branch and we don't have a corresponding PR(1). A complete takeover of the repo can result in a hanging commit (by deleting the renovate bump branch), but there must be a hanging PR-ref around. Since there isn't one: 4. All of the above points to a compromised account that has write access to the repo. There is also the https://github.com/tj-actions-bot https://github.com/tj-actions-bot account, but unclear if it has write access. Edit: gurchik's guess at https://github.com/tj-actions/changed-files/issues/2463#issuecomment-2726186994 https://github.com/tj-actions/changed-files/issues/2463#issu... seems more likely: > 1. Fork the repository > > 2. Push compromised code to the fork > > 3. Update the tags in the parent repository to point to the SHA of the fork
• cedws 2y ago
  I called it: https://cedwards.xyz/github-actions-are-an-impending-security-disaster/ https://cedwards.xyz/github-actions-are-an-impending-securit...
• btown 2y ago
  Does anyone know if https://github.com/tj-actions/verify-changed-files/ https://github.com/tj-actions/verify-changed-files/ was compromised as well?
• themgt 2y ago
  This is hilarious, the maven-lockfile project "Lockfiles for Maven. Pin your dependencies. Build with integrity" appears to have auto-merged a PR for the compromised action commit. So the real renovate bot immediately took the exfiltration commit from the fake renovate bot and started auto-merging it into other projects: https://github.com/chains-project/maven-lockfile/pull/1111 https://github.com/chains-project/maven-lockfile/pull/1111
  • sureIy 2y ago
    The fun part is that they used commits specifically for security, but then add an auto-updater. Might as well use tags.
  • mdaniel 2y ago
    heh, timing is everything https://github.com/chains-project/maven-lockfile/issues/1085#issuecomment-2721863241 https://github.com/chains-project/maven-lockfile/issues/1085... > After some cleanup the changed-files (https://github.com/tj-actions/changed-files https://github.com/tj-actions/changed-files) action seems to be more work to remove. It would be awesome if it could be added to the allowlist > Done. Allowed all versions of this action. Should I pin it to one version in the allowlist (won't be convenient if renovate updates this dependency)?
• joshka 2y ago
  Just leaving this here... https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions https://docs.github.com/en/actions/security-for-github-actio...
• sangeeth96 2y ago
  Don’t want to be alarmist but even if not using this action directly, I wonder what implications might be if this has leaked tokens from prominent public-facing project repos which might be used by several folks? I spotted an issue[1] to fix this in Expo EAS CLI and I’m guessing there are many more. The payload I saw from the report only seems to dump things to stdout but I guess analysis is still in progress and IDK if it’s the same payload for all the tags. [1]: https://github.com/expo/eas-cli/pull/2948/files https://github.com/expo/eas-cli/pull/2948/files
• mdaniel 2y ago
  The semgrep URL about this seems to have won the submission lottery: https://news.ycombinator.com/item?id=43368870 https://news.ycombinator.com/item?id=43368870
• mellosouls 2y ago
  A list of projects claimed to be using it from the GitHub page: https://github.com/tj-actions/changed-files?tab=readme-ov-file#open-source-projects- https://github.com/tj-actions/changed-files?tab=readme-ov-fi...
  • jeeyoungk 2y ago
    Looks like a lot of them are pinning to tags (which are not guaranteed to change) or SHA (https://github.com/vitejs/vite/blob/8da04227d6f818a8ad9efc0056101968037c2e36/.github/workflows/ci.yml#L61 https://github.com/vitejs/vite/blob/8da04227d6f818a8ad9efc00...) which is more hermetic.
    • gizzlon 2y ago
      tags do not help, they're just a labelor a pointer
• oefrha 2y ago
  > https://github.com/tj-actions/changed-files/pull/2460 https://github.com/tj-actions/changed-files/pull/2460 This kind of auto dependency bump bots are more trouble than their worth. If your app works today, bumping random deps won’t make it work better in any meaningful sense in 95% of cases. With such a small upside, the downside of introducing larger attack surfaces, subtle breakages (despite semver), major breakages, and in the worst cases, compromises (whether it’s a compromised dep, or fake bot commits that people are trained to ignore) just completely outweighs the upside. You’re on the fast lane to compromises by using this kind of crap. People should really learn from Go’s minimum version selection strategy.
  • lostmsu 2y ago
    Your app will have unpatched vulnerabilities.
    • oefrha 2y ago
      As long as you subscribe to security advisories, it’s a lot more likely that new vulnerabilities are introduced than old undiscovered vulnerabilities are accidentally patched. In fact barring rewrites (which usually won’t be picked up by semver-respecting auto bumps anyway) I can hardly think of an example of the latter.
• rajbot 2y ago
  This GitHub Action is still compromised, leaving thousands of repositories vulnerable. Is there anyone here from GitHub that can help get this fixed?
• captn3m0 2y ago
  Helpful update: The gist author has deleted the gist, so https://gist.githubusercontent.com/nikitastupin/30e525b776c409e03c2d6f328f254965/raw/memdump.py https://gist.githubusercontent.com/nikitastupin/30e525b776c4... now results in a 404, and stops the action from any further secrets being leaked. This means you're impacted only if you used the action, and had a build triggered in the last 6 hours or so.
  • lmc 2y ago
    Beautiful... 3rd party dependency exploit thwarted by its own 3rd party dependency.
  • deleted 2y ago
    [deleted]
• Sytten 2y ago
  I am surprised nobody here mentionned immutable github actions that are coming [1]. Been waiting for them since the issue was open in 2022. This would have significantly reduce impact and hopefully github will get it over the finish line. I always fork my actions or at least use a commit hash. [1] https://github.com/features/preview/immutable-actions https://github.com/features/preview/immutable-actions
  • ricardobeat 2y ago
    I thought actions were already immutable and published to a registry, not fetched directly from their repo. TIL. Go also uses tags for module versioning, and while go.mod or package-lock.json stop this attack from reaching existing consumers, allowing remapping of all versions to the compromised one still expands the impact surface a lot. GitHub should offer a “immutable tags” setting for repos like these.
  • deleted 2y ago
    [deleted]
• samschooler 2y ago
  https://gist.github.com/gmatuz/7186f583df5f28196cc1f402af3bfa1b https://gist.github.com/gmatuz/7186f583df5f28196cc1f402af3bf... This gist is pretty much the exact code, from the base64 encoded stuff. Looks like who ever put this in at least neafed the shell script.
• gbraad 2y ago
  Am I seeing this correctly, that a (fake/impersonation?) Renovate bot actually proposed the fix... and then other repositories trickled that fix in, also suggested by Renovate or Dependabot, as the dependency updated? I usually fork (or create my own) actions, as I do not trust the whole chain on GitHub. The marketplace does no enforcement. It is really based on trust you have in the 3rd-party... and I do not have this; as many actions have side-effects, or only operate on a specific runner OS, etc.
  • brown9-2 2y ago
    Renovate sending PRs to projects to upgrade their action version is unrelated to the original comment having a spoofed author of Renovate
• Everdred2dx 2y ago
  How does this siphon the secrets away? It looks like it just dumps them out to stdout and stops there.
  • varunsharma07 2y ago
    Yes, just prints to the build log, so the risk is higher for public repos. Lot of public repos have creds printed in their build logs due to this compromised action.
• stephenr 2y ago
  I'm sorry I must be missing something. The examples from the repo itself aren't helping to explain. Why would anyone use this whole convoluted nodejs thing when `git diff-tree` exists? I'm struggling to see a scenario where this isn't just part of some deliberately over complicated rube Goldberg setup.
• globular-toast 2y ago
  Shocked Pikachu anyone?
• eacapeisfutuile 2y ago
  What is the current state? Tags reverted or still poisoned?
  • phantomathkg 2y ago
    User and repo all 404 now. https://github.com/tj-actions https://github.com/tj-actions https://github.com/tj-actions/changed-files https://github.com/tj-actions/changed-files
    • eacapeisfutuile 2y ago
      Damn, so fully pulled, by authors, GitHub or adversary?
      • eacapeisfutuile 2y ago
        Hey, please tell me how my comment was bad in some way instead of just downvoting. It was a genuine question if anyone might have insight.
• deleted 2y ago
  [deleted]
• vimgrinder 2y ago
  with all the AI stuff going around, can't github just scan repos for such malicious code?
• prmph 2y ago
  To me the only solution is that we need a security in depth approach: - Create a trusted packages program, and mark trusted packages with a prominent badge. Package authors can apply to join the program, which will involve a review of their package and any subsequent updates. Ensure trusted packages can only depend on other trusted packages. - Implement a capabilities model for package managers. I hear Deno is better in that respect. - Have the package manager back-end use AI to continually review the packages. If anything suspicious is found, flag it and investigate manually. - Require all packages to be name-spaced
  • carols10cents 2y ago
    who is going to pay for the review of packages and updates? how do we know we can trust the reviewers? github actions are name-spaced and that didn't help anything here...
• ebfe1 2y ago
  Doing a bit of investigation with github_events in clickhouse, it is quite clear that the accounts used to perform the attack was "2ft2dKo28UazTZ", "mmvojwip" also seems suspicious: https://play.clickhouse.com/play?user=play#c2VsZWN0ICogZnJvbSBnaXRodWJfZXZlbnRzIHdoZXJlIGV2ZW50X3R5cGU9J0ZvcmtFdmVudCcgYW5kIHJlcG9fbmFtZSA9ICd0ai1hY3Rpb25zL2NoYW5nZWQtZmlsZXMnIGFuZCBjcmVhdGVkX2F0ID4gbm93KCkgLSBpbnRlcnZhbCAzMCBkYXkgYW5kIGNyZWF0ZWRfYXQgPCBub3coKSAtIGludGVydmFsIDEgZGF5IDsK https://play.clickhouse.com/play?user=play#c2VsZWN0ICogZnJvb... Actions taken by the threat actor at the time can be seen here: https://play.clickhouse.com/play?user=play#c2VsZWN0ICogZnJvbSBnaXRodWJfZXZlbnRzIHdoZXJlIGFjdG9yX2xvZ2luIGluICgnMmZ0MmRLbzI4VWF6VFonLCAnbW12b2p3aXAnKQ== https://play.clickhouse.com/play?user=play#c2VsZWN0ICogZnJvb...
  • ebfe1 2y ago
    Note that these account seems to be deleted now - 2ft2dKo28UazTZ clearly did more than just changed-files and also seem to target coinbase/agentkit as well (Actually .. they might be targeted by the threat actor)
    • rognjen 2y ago
      That is a really very likely scenario. The attacker was trying to compromise agentkit and found changed-files used in the repo so looked around. Found that it was using a bot with a PAT to release. Totally possible the bot account had a weak password, and the maintainer said it didn't have 2FA. They got the release bot PAT so they tried possibly quite an obvious vector that. They didn't need anything sophisticated or to exfil the credentials because agentkit is public. It just so happened that it was detected before agentkit updated dependencies. It's possible that with if thye had checked the dependabot config they could've timed it a bit better so that it's picked up in agentkit before being detected. edit: Although, I don't think PATs are visible after they're generated?
  • ebfe1 2y ago
    It seems i forgot to cater for the quota applied to free "play" user in ClickHouse in my previous query... In fact, the threat actor did a lot more... this should give a better list of actions that was performed - Clearly showed he was testing his payload: https://play.clickhouse.com/play?user=play#c2VsZWN0ICogZnJvbSBnaXRodWJfZXZlbnRzIHdoZXJlIGFjdG9yX2xvZ2luIGluICgnMmZ0MmRLbzI4VWF6VFonLCAnbW12b2p3aXAnKSBhbmQgY3JlYXRlZF9hdCA+ICAnMjAyNS0wMy0xMSAwMDowMDowMCcgYW5kIGNyZWF0ZWRfYXQgPCAgJzIwMjUtMDMtMTUgMDA6MDA6MDAnICAgb3JkZXIgYnkgY3JlYXRlZF9hdCBBU0MK https://play.clickhouse.com/play?user=play#c2VsZWN0ICogZnJvb...
• skwashd 2y ago
  The GitHub repo and org disappeared while I was poking around. Both https://github.com/tj-actions https://github.com/tj-actions and https://github.com/tj-actions/changed-files https://github.com/tj-actions/changed-files return 404.
  • jtwaleson 2y ago
    Is there a safe fork somewhere?
    • kareemery 2y ago
      Looking for one as well. My gh actions broke and looking for a drop in replacement. Got this working with claude, would love if someone with more knowledge had thoughts: ```bash # Extract branch name from GITHUB_REF or GITHUB_HEAD_REF echo "current_branch=${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}" >> $GITHUB_OUTPUT # Determine if this is the default branch if [[ "${GITHUB_REF#refs/heads/}" == "${GITHUB_REF_NAME}" && "${GITHUB_REF_NAME}" == "${GITHUB_BASE_REF:-${GITHUB_REF_NAME}}" ]]; then echo "is_default=true" >> $GITHUB_OUTPUT else echo "is_default=false" >> $GITHUB_OUTPUT fi # Set default branch name echo "default_branch=${GITHUB_BASE_REF:-${GITHUB_REF_NAME}}" >> $GITHUB_OUTPUT ```
    • isamu_2000 2y ago
      https://github.com/dorny/paths-filter https://github.com/dorny/paths-filter ? looking into it. Definitely going through and pinning all my 3rd party actions to specific commits e.g. party/package@<commitsha>.
    • jtwaleson 2y ago
      A colleague sent this: https://github.com/trmlabs/changed-files https://github.com/trmlabs/changed-files
• hn8726 2y ago
  I wish Github didn't just purge entire repositories/accounts in the event like this, although I know why they do this. But there's now no way to analyze the repository/exploit anymore
• sathyabhat 2y ago
  See also - https://news.ycombinator.com/item?id=43368870 https://news.ycombinator.com/item?id=43368870
• eacapeisfutuile 2y ago
  Was the entire repo just deleted from GitHub?
• mmsc 2y ago
  Paid Github organizations have a policy to block third-party actions. Would be nice if there was a way to allow third-party actions as long as they are referenced by hash, not version.
• m4rtink 2y ago
  Another reason why you should be getting software via distro, with searate maintainers taking care of it there rather than directly from the developers that can inject malware via the very next version you mindlessly pull in without checking. Also due to here being usually more than one distro, more people will look at the code & can spot the usptream getting rogue or getting compromised.
  • SAI_Peregrinus 2y ago
    Adding MITM to your chain of trust doesn't make it more trustworthy, it adds an extra possible attacker.
    • m4rtink 2y ago
      Often there is vetting before one becomes a distro maintainer & even if one of them gets compromised, the blast radius is at least limitted to that one distro, rather than "everyone" like in case of NPM & co. Non rolling distros aslo have various policies for package updates, making it much harder to get a compromised package to all supported distro versions before it is eventually discovered.
    • imoreno 2y ago
      Actually it does. The repo maintainer is on the user's side, so they are doing MITM on the attack vector. This makes it harder to get your malicious code in, because MITM might intercept it. Yes now you have to trust the maintainer but that's sometimes easier.
• rarkins 2y ago
  Hi, Renovate author/maintainer here. The affected repo has now been taken down, so I am writing this partly from memory, but I believe the scenario is: 1. An attacker had write access to the tj-actions/changed-files repo 2. The attacker chose to spoof a Renovate commit, in fact they spoofed the most recent commit in the same repo, which came from Renovate 3. Important: this spoofing of commits wasn't done to "trick" a maintainer into accepting any PR, instead it was just to obfuscate it a little. It was an orphan commit and not on top of main or any other branch 4. As you'd expect, the commit showed up as Unverified, although if we're being realistic, most people don't look at that or enforce signed commits only (the real bot signs its commits) 5. Kind of unrelated, but the "real" Renovate Bot - just like Dependabot presumably - then started proposing PRs to update the action, like it does any other outdated dependency 6. Some people had automerging of such updates enabled, but this is not Renovate's default behavior. Even without automerging, an action like this might be able to achieve its aim only with a PR, if it's run as part of PR builds 7. This incident has reminded that many people mistakenly assume that git tags are immutable, especially if they are in semver format. Although it's rare for such tags to be changed, they are not immutable by design
  • mlor 2y ago
    Thanks for taking the time to comment. Not that it wasn't there before this, but this incident highlights a lot to take into consideration with respect to securing one's supply chain going forward.
  • diggan 2y ago
    > 6. Some people had automerging of such updates enabled, but this is not Renovate's default behavior. Even without automerging, an action like this might be able to achieve its aim only with a PR, if it's run as part of PR builds I'm not sure how this could exploited by just making a PR, unless you for some reason have secrets enabled for builds by unknown contributors, which obviously would be a mistake. Usually, only builds using secrets only run on certain branches which has a known contributor approving the code before it gets there. > people mistakenly assume that git tags are immutable If you're distributing a library on GitHub used by many other people/projects, then you really need to setup `protected branches` and `protected tags`, where you can prevent changes somewhat.
• greatgib 2y ago
  Honestly now most people doing "modern software engineering" are retards with no concept of real software engineering concepts. Shitload of cardboard cto are pushing for "modern practice" to use whatever new version of whatever random dependency downloaded straight from internet. Some persons asked why I don't like Ruff or UV for Python for example? You start a new job, first thing you have to do after installing a serious and safe Linux distribution like debian: Curl whateverwebsite.com/ruff/download/latest | bash --blindly-execute --like-an-idiot -> retrieving-random-dependancy1.tgz -> executing-random-code... And I don't speak about the current trend with "pre-commit" where a lot of persons are ok to have automatic downloads and execution on dev machines and ci, at each commit, of hundred of really totally random plugins from random places. But this is a cto enforced decision to have this pre-commit for software quality...
  • trallnag 2y ago
    There are usually several different ways to install tools like uv. For example, uv is available as a Homebrew formula. And regarding pre-commit: It usually relies on external code, but it is not a requirement. Pre-commit can also be used to just run arbitrary commands and scripts defined within the pre-commit config using the values "system" or "script" for the "language" key.
• imglorp 2y ago
  Anybody have a snapshot of the good one, or maybe a drop in replacement? The repo is gone now.
  • thinkplay11 2y ago
    Yup this would be super useful. Scouring about now trying to find
    • imglorp 2y ago
      This looks like it's three years old but it's a start. https://github.com/actions-marketplace-validations/tj-actions_changed-files https://github.com/actions-marketplace-validations/tj-action...
• v1sionSec 2y ago
  As the repo is was taken down is someone able to tell me when was the malicious commit pushed. Trying to get a timeline to see if any workflows using this action were trigger in that timeframe. Thank you
  • imnotjames 2y ago
    Your secrets will be published to the CI log if you were affected. I believe it's everything since around 10pm ET last night. I would consider any runs in the past 24 hours to be suspect.
    • v1sionSec 2y ago
      Thank you, unfortunately we have a multiple of repositories with multiple runs that use this action so checking the logs one by one will be hard. Any idea how to get all logs? Thank you
      • mixologic 2y ago
        also the secrets will be published as double base 64 encoded, so it will just look like a string of random chars at the end of the changed-files action in the log.
      • mongrelion 2y ago
        I think your best bet is to traverse all the pipeline logs that make use of the action using Github's REST API. It should be easy to do with thr Github CLI tool and some bash scripting. Not sure how easy it'll be to parse the logs to look for a base64 string but it shouldn't be that complicated either.
      • imnotjames 2y ago
        If you are using the action and were as of 10p ET last night I would assume everything is compromised, remove the action, and rotate secrets.
  • alper 2y ago
    Somewhere after 18:00 CET Friday.
• MotiBanana 2y ago
  Does anyone know if the secrets compromised were sent out, or just printed to stdout? We don't have any public repos using this action.
  • varunsharma07 2y ago
    They were only printed to stdout and not sent out
• alper 2y ago
  GitHub’s incident response to this took 17 hours give or take. Actions is a paid service but Microsoft probably replaced all the security teams with AI.
• onnimonni 2y ago
  It seems pretty awful that the de-facto way to use GitHub Actions is using git tags which are not immutable. For example to checkout code [1]: - uses: actions/checkout@v4 Github does advise people to harden their actions by referring to git commit hashes [2] but Github currently only supports SHA-1 as hashing algorithm. Creating collisions with this hashing algo will be more and more affordable and I'm afraid that we will see attacks using the hash collisions during my lifetime. I wish that they will add support for SHA-256 soon and wrote product feedback regarding it here: https://github.com/orgs/community/discussions/154056 https://github.com/orgs/community/discussions/154056 If this resonates with you please go and give it a thumbs up :) [1]: https://github.com/actions/checkout?tab=readme-ov-file#usage https://github.com/actions/checkout?tab=readme-ov-file#usage [2]: https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions#using-third-party-actions https://docs.github.com/en/actions/security-for-github-actio...
  • onnimonni 2y ago
    I wasn't aware of the already existing SHA-1 collision support created by Github. It's very interesting read and AFAIK it seems that using SHA-1 collisions is not possible: https://github.blog/news-insights/company-news/sha-1-collision-detection-on-github-com/ https://github.blog/news-insights/company-news/sha-1-collisi... Is anyone aware of a git hook I could use to analyse my .github/workflows/*.yml files and replace git tags like "v4" with the current git commit hashes? I think this would make it much safer to use 3rd party GitHub Actions.
    • Y_Y 2y ago
      That's the sort of hook you should be able to write yourself pretty quickly. So I threw your comment into o3-mini-high and it gave me a decent-looking solution. Decent but wrong, since it thought "current git commit" referred to the project repo, rather than the referenced dependency. Anyway here's the gist of a solution without any of the necessary checking that the files actually exist etc. #!/bin/sh for file in .github/workflows/*.yml; do grep -E "uses:[[:space:]]+[A-Za-z0-9._-]+/[A-Za-z0-9._-]+@v[0-9]+" "$file" | while read -r line; do repo=$(echo "$line" | sed -E 's/.*uses:[[:space:]]+([A-Za-z0-9._-]+\/[A-Za-z0-9._-]+)@v[0-9]+.*/\1/') tag=$(echo "$line" | sed -E 's/.*@((v[0-9]+)).*/\1/') commit_hash=$(git ls-remote "https://github.com/$repo.git" "refs/tags/$tag" | awk '{print $1}') [ -n "$commit_hash" ] && sed -i.bak -E "s|(uses:[[:space:]]+$repo@)$tag|\1$commit_hash|g" "$file" && git add "$file" && rm -f "$file.bak" done done exit 0
• deathanatos 2y ago
  I've said this before, but in my mind the central problem in supply chain issues is this. Choose one: 1. You fix what version you're using to a fixed, immutable package. You receive no updates, no bug fixes, no security patches. 2. You follow a pointer to something like a API-compatible version, "latest" (#yolo) or ^5.0.0. You get bug fixes, security patches, but someone can push malicious updates. Security types, IME, invariably want both: fix that package to a hash, so that we can't have a take over attack. But also we need to stay on top of updates, because we don't want to find out we have a decades old struct4j CVE buried in our codebase just waiting to be exploited. So to accomplish "both", then we get into schemes like "fix the hashes … but we'll have a bot¹ update our dependency tree automatically". So like, #2, with more steps. Is anyone actually vetting that that update hash isn't going to compromise stuff? Hell no, no company is hiring that level of engineers; I'm lucky to have decent staffing for our primary concerns, reading the code in the dependency tree is out of the question. And I'm sure in the coming days, security minded people will stampede in the general direction of #1. Stuff'll get fixed to hash, and stuff'll stop getting security patches. IDK what the answer is, these seem pretty like fundamentally opposed forces of nature. The staffing problems aren't a technical problem, that's a capitalism problem, mostly in that there is very little to no penalty for a breach, so why would anyone hire the eng required to ensure the software works. There was hardly regulation in 2024, and any fines I did see regulatory bodies award are pittances, without fail. And, what regulation there was is now being actively dismantled. There is some discussion of signed packages in this thread, and that's a helpful idea, I think, though I don't think it completely eliminates the problem: if the signing key is compromised, we're back to square one. The lay eng struggles with PKI. ¹While there is a bot of such nature (the renovate bot) somewhat tied up in this particular instance, I wouldn't over-focus on that bot, specifically; renovate, in particular, is not that relevant to the point I'm trying to make.
  • HALtheWise 2y ago
    It seems like a 24hr delay for auto upgrades would mitigate a lot of this, maybe with some way that a trusted third-party could skip the delay for big-ticket zero day patches?
    • 2y ago
• rahulr0609 2y ago
  For folks looking for a drop-in replacement for v45 (latest major version), we have a patched mirror here: https://github.com/trmlabs/changed-files https://github.com/trmlabs/changed-files 1] We took the public mirror from: https://code.forgejo.org/tj-actions/changed-files/src/tag/v45.0.7 https://code.forgejo.org/tj-actions/changed-files/src/tag/v4... 2] Undid the malicious code change: https://code.forgejo.org/tj-actions/changed-files/commit/0e58ed8671d6b60d0890c21b07f8835ace038e67 https://code.forgejo.org/tj-actions/changed-files/commit/0e5... - You can see the change here: https://github.com/trmlabs/changed-files/commit/8567847ee196216df6a277444531c8f5e18478ed https://github.com/trmlabs/changed-files/commit/8567847ee196... 3] Published under a v1 tag (since we can't vet historical releases and changes and didn't want folks to get confused) If you want to contribute or report an issue, file a GH Issue or ping us at security@trmlabs.com
• kurmiashish 2y ago
  Due to the ongoing security incident involving the tj-actions/changed-files Action, we at StepSecurity have provided a secure, drop-in replacement: step-security/changed-files. We strongly advise replacing all instances of tj-actions/changed-files in your workflows with our secure alternative: https://github.com/step-security/changed-files https://github.com/step-security/changed-files
  • deleted 2y ago
    [deleted]
  • joonas 2y ago
    Kudos for making this freely available, I was initially delighted to find out that there was a StepSecurity maintained alternative for the dorny/paths-filter action[1] as that seemed like a reasonable alternative to migrate to, but ended up being disappointed once I realized that it requires a subscription to use[2] [1]: https://github.com/step-security/paths-filter https://github.com/step-security/paths-filter [2]: https://github.com/step-security/paths-filter/blob/b251c10d0394d09468fa1d97d2551522ce4a0593/src/main.ts#L22-L27 https://github.com/step-security/paths-filter/blob/b251c10d0...
    • deleted 2y ago
      [deleted]
  • rahulr0609 2y ago
    @kurmiashish - If you and team are willing share your version without requiring a Step Security subscription today or in the future, happy to archive our repo and redirect users to Step Thanks again for your timely detection and reporting!
    • kurmiashish 2y ago
      @rahulr0609 https://github.com/step-security/changed-files https://github.com/step-security/changed-files will forever remain free, and the community can use it without requiring a StepSecurity subscription.
• v1sionSec 2y ago
  [dead]
• frenchtoast8 2y ago
  The repository is back online, with this explanation from the developer: > This attack appears to have been conducted from a PAT token linked to @tj-actions-bot account to which "GitHub is not able to determine how this PAT was compromised." > Account Security Enhancements > * The password for the tj-actions-bot account has been updated. > * Authentication has been upgraded to use a passkey for enhanced security. > * The tj-actions-bot account role has been updated to ensure it has only the minimum necessary permissions. > * GitHub proactively revoked the compromised Personal Access Token (PAT) and flagged the organization to prevent further exploitation. https://github.com/tj-actions/changed-files/issues/2464#issuecomment-2727020537 https://github.com/tj-actions/changed-files/issues/2464#issu...
  • deleted 2y ago
    [deleted]
  • frenchtoast8 2y ago
    Editing to add: the developer has locked further discussion about this. Very concerning as I believe their explanations are raising more questions than they are answering. First of all, clearly Github can't answer for the developer how their bot's token was compromised, that's something the developer needs to find out. Instead they are repeating this statement like it's out of their hands. But more concerningly, I don't believe the explanation is supported by the Github history which says the compromised commit was "authored" by Renovate and "pushed" by @jackton1. It's obvious how the first part was spoofed, but the second part is concerning as it indicates the @jackton1 account was compromised not @tj-actions-bot. If I'm missing something please let me know.
    • thj4742 2y ago
      Check the timestamp on that commit push. It was from today, an hour or two before the repo was restored, not yesterday when the attack happened. The push actor != the committor or even the actual commit author, and there can be multiple push actors if the commit is pushed multiple times by different actors. He probably just re-pushed the bad commit while trying to figure out how to fix this. I find it very plausible that the bot token was compromised, not his user account token, as the attack was simply to push over the tags (which is something the automation bot would have access to do, as tag management is one of its functions)
• mgiladi 2y ago
  We've recently released open-source tools that would have easily prevented this: 1. The maintainers could have used PRevent to immediately alert and block any PR containing malicious code, or easily configured it for detection in case of a direct push: https://github.com/apiiro/PRevent https://github.com/apiiro/PRevent 2. Users could have used our malicious code detection ruleset to immediately detect and block it when scanning updates in all relevant CI/CD stages: https://github.com/apiiro/malicious-code-ruleset https://github.com/apiiro/malicious-code-ruleset 3. For a better understanding of the detection, the malicious code falls precisely into the patterns presented in our research: https://apiiro.com/blog/guard-your-codebase-practical-steps-and-tools-to-prevent-malicious-code/ https://apiiro.com/blog/guard-your-codebase-practical-steps-...
  • klysm 2y ago
    Malicious code detection is a fools errand. The attacker would’ve known that was installed and obfuscated it to avoid detection. It’s trivial to do so because they know what the detection is. There is no way this would’ve prevented anything.
    • mgiladi 2y ago
      They have no way of knowing unless they have admin access, in which case they can do whatever they want anyway. If the tool produces any visible outputs, just configure it to block silently. That's on the maintainer side. On the consumer side, not even that discloses such use. Isn't it so? Add behavioral detection, and you get a strong layer of defense, even if attackers know about it. You still want defense in depth as always, of course.
      • klysm 2y ago
        I just went to the readme of PRevent, and looked at it when writing my comment. It puts a comment explaining specifically which code triggered the problem. The attacker can easily see this happening and then test locally with PRevent to make sure their obfuscated code will bypass your detection no?
• varunsharma07 2y ago
  I’m Varun, CEO & Co-Founder of StepSecurity. StepSecurity detected and reported the tj-actions/changed-files compromise and has been actively helping the community recover from this incident. To support you in understanding what happened and recovering swiftly, we’re hosting an Office Hour: Date: March 17, 2025 Time: 10:00 AM Pacific Time (PT) Add to your calendar: https://www.addevent.com/event/Tf25207322 https://www.addevent.com/event/Tf25207322
  • klysm 2y ago
    You can rename office hour to sales pitch