6 ms·

While GitHub obsess over shoving AI into everything, the rest of the platform is genuinely crumbling and its security flaws are being abused to cause massive da

by cedws 7mo ago

While GitHub obsess over shoving AI into everything, the rest of the platform is genuinely crumbling and its security flaws are being abused to cause massive damage. Last week Aqua Security was breached and a few repositories it owns were infected. The threat actors abused widespread use of mutable references in GitHub Actions, which the community has been screaming about for years, to infect potentially thousands of CI runs. They also abused an issue GitHub has acknowledged but refused to fix that allows smuggling malicious Action references into workflows that look harmless. GHA can’t even be called Swiss cheese anymore, it’s so much worse than that. Major overhauls are needed. The best we’ve got is Immutable Releases which are opt in on a per-repository basis.

ljm 7mo ago
I worry that CI just got overcomplicated by default when providers started rocking up with templated YAML and various abstractions over it to add dynamic behaviour, dependencies, and so on. Perhaps mixing the CI with the CD made that worse because usually deployment and delivery has complexities of its own. Back in the day you'd probably use Jenkins for the delivery piece, and the E2E nightlies, and use something more lightweight for running your tests and linters. For that part I feel like all you need, really, is to be able to run a suite of well structured shell scripts. Maybe if you're in git you follow its hooks convention to execute scripts in a directory named after the repo event or something. Forget about creating reusable 'actions' which depend on running untrusted code. Provide some baked in utilities to help with reporting status, caching, saving junit files and what have you. The only thing that remains is setting up a base image with all your tooling in it. Docker does that, and is probably the only bit where you'd have to accept relying on untrusted third parties, unless you can scan them and store your own cached version of it. I make it sound simpler than it is but for some reason we accepted distributed YAML-based balls of mud for the system that is critical to deploying our code, that has unsupervised access to almost everything. And people are now hooking AI agents into it.
- Hasnep 7mo ago
  I'm trying out SelfCI [1] for one of my projects and it's similar to what you were describing. My whole CI pipeline is just a shell script that runs the actual build and test commands, I can write a script in another language like python if I need more complexity and I can run it all locally at any time to debug. [1] https://app.radicle.xyz/nodes/radicle.dpc.pw/rad%3Az2tDzYbAXxTQEKTGFVwiJPajkbeDU https://app.radicle.xyz/nodes/radicle.dpc.pw/rad%3Az2tDzYbAX...
- jmaw 7mo ago
  You could use these shell script versions of pipelines in GHA though, right? There is nothing stopping you from triggering a bash script via a "run" step in YAML. These reusable actions are nothing but a convenience feature. This discussion isn't much different than any other supply chain, dependency, or packaging system vulnerability such as NPM, etc. One slight disclaimer here is the ability of someone to run their own updated copy of an action when making a PR. Which could be used to exfil secrets. This one is NOT related to being dependent on unverified actions though. (re-reading this came across as more harsh than I intended.. my bad on that. But am I missing something or is this the same issue that every open-source user-submitted package repository runs in to?)
a-french-anon 7mo ago
If you want more ammo for your ranting (no offense meant, I also rant): an issue as massive as https://github.com/orgs/community/discussions/142308 https://github.com/orgs/community/discussions/142308 lingering for years should do the trick.
- llimllib 7mo ago
  It really feels like Firefox is not a supported browser on GitHub, I hit this and also find that much of the time the commit message is not correctly pulled from the PR description when that setting is enabled
  - mrguyorama 7mo ago
    Meanwhile, bitbucket has never given us problems with several of our team using Firefox
  - noir_lord 7mo ago
    Given I get "Too Many Requests" if I try and search for a repo from FF logged out and have done for at least a year, I'd say so. Works fine on Chrome/Edge.
- NewJazz 7mo ago
  * no ipv6 support * no ff merge support * no sha2 commit hash support
  - bigfishrunning 7mo ago
    to be fair, git itself only supports sha256 (as opposed to all of sha2) and even then it's marked "THIS OPTION IS EXPERIMENTAL" so many are afraid to use it ff merge support though....what a world that would be
- dijksterhuis 7mo ago
  i had something similar with PRs last year. 2x PRs of mine disappeared for me. they were still counted in the total number of PRs and everyone else could see them. 0_o
tasuki 7mo ago
> GHA can’t even be called Swiss cheese anymore, it’s so much worse than that. That's a high bar though. Few things are better than Swiss cheese.
- jjgreen 7mo ago
  Salers (if you can find it)
imglorp 7mo ago
Public service announcement You can pin actions versions to their hash. Some might say this is a best practice for now. It looks like this, where the comment says where the hash is supposed to point. Old --> uses: actions/checkout@v4 New --> uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4 There is a tool to sweep through your repo and automate this: https://github.com/mheap/pin-github-action https://github.com/mheap/pin-github-action
- lijok 7mo ago
  The problem is actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 probably doesn’t do this same pinning, and the actions ecosystem is such an intertwined mess that any single compromised action can propagate to the rest
  - imglorp 7mo ago
    Yes, true, but at least the fire won't spread through this one point. Hopefully all of your upstreams can be persuaded to pin also.
    - franktankbank 7mo ago
      Doesn't a single compromised action in the chain cause the whole to be fucked? Pinning the top level doesn't prevent any spread.
      - teaearlgraycold 7mo ago
        Might want to vendor everything?
        franktankbank 7mo ago
        I don't know what this means in this context.
        7mo ago

by cedws 7mo ago

ljm 7mo ago

I worry that CI just got overcomplicated by default when providers started rocking up with templated YAML and various abstractions over it to add dynamic behaviour, dependencies, and so on. Perhaps mixing the CI with the CD made that worse because usually deployment and delivery has complexities of its own. Back in the day you'd probably use Jenkins for the delivery piece, and the E2E nightlies, and use something more lightweight for running your tests and linters. For that part I feel like all you need, really, is to be able to run a suite of well structured shell scripts. Maybe if you're in git you follow its hooks convention to execute scripts in a directory named after the repo event or something. Forget about creating reusable 'actions' which depend on running untrusted code. Provide some baked in utilities to help with reporting status, caching, saving junit files and what have you. The only thing that remains is setting up a base image with all your tooling in it. Docker does that, and is probably the only bit where you'd have to accept relying on untrusted third parties, unless you can scan them and store your own cached version of it. I make it sound simpler than it is but for some reason we accepted distributed YAML-based balls of mud for the system that is critical to deploying our code, that has unsupervised access to almost everything. And people are now hooking AI agents into it.

Hasnep 7mo ago
I'm trying out SelfCI [1] for one of my projects and it's similar to what you were describing. My whole CI pipeline is just a shell script that runs the actual build and test commands, I can write a script in another language like python if I need more complexity and I can run it all locally at any time to debug. [1] https://app.radicle.xyz/nodes/radicle.dpc.pw/rad%3Az2tDzYbAXxTQEKTGFVwiJPajkbeDU https://app.radicle.xyz/nodes/radicle.dpc.pw/rad%3Az2tDzYbAX...
jmaw 7mo ago
You could use these shell script versions of pipelines in GHA though, right? There is nothing stopping you from triggering a bash script via a "run" step in YAML. These reusable actions are nothing but a convenience feature. This discussion isn't much different than any other supply chain, dependency, or packaging system vulnerability such as NPM, etc. One slight disclaimer here is the ability of someone to run their own updated copy of an action when making a PR. Which could be used to exfil secrets. This one is NOT related to being dependent on unverified actions though. (re-reading this came across as more harsh than I intended.. my bad on that. But am I missing something or is this the same issue that every open-source user-submitted package repository runs in to?)

a-french-anon 7mo ago

If you want more ammo for your ranting (no offense meant, I also rant): an issue as massive as https://github.com/orgs/community/discussions/142308 https://github.com/orgs/community/discussions/142308 lingering for years should do the trick.

llimllib 7mo ago
It really feels like Firefox is not a supported browser on GitHub, I hit this and also find that much of the time the commit message is not correctly pulled from the PR description when that setting is enabled
- mrguyorama 7mo ago
  Meanwhile, bitbucket has never given us problems with several of our team using Firefox
- noir_lord 7mo ago
  Given I get "Too Many Requests" if I try and search for a repo from FF logged out and have done for at least a year, I'd say so. Works fine on Chrome/Edge.
NewJazz 7mo ago
* no ipv6 support * no ff merge support * no sha2 commit hash support
- bigfishrunning 7mo ago
  to be fair, git itself only supports sha256 (as opposed to all of sha2) and even then it's marked "THIS OPTION IS EXPERIMENTAL" so many are afraid to use it ff merge support though....what a world that would be
dijksterhuis 7mo ago
i had something similar with PRs last year. 2x PRs of mine disappeared for me. they were still counted in the total number of PRs and everyone else could see them. 0_o

tasuki 7mo ago

> GHA can’t even be called Swiss cheese anymore, it’s so much worse than that. That's a high bar though. Few things are better than Swiss cheese.

jjgreen 7mo ago
Salers (if you can find it)

imglorp 7mo ago

Public service announcement You can pin actions versions to their hash. Some might say this is a best practice for now. It looks like this, where the comment says where the hash is supposed to point. Old --> uses: actions/checkout@v4 New --> uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4 There is a tool to sweep through your repo and automate this: https://github.com/mheap/pin-github-action https://github.com/mheap/pin-github-action

lijok 7mo ago
The problem is actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 probably doesn’t do this same pinning, and the actions ecosystem is such an intertwined mess that any single compromised action can propagate to the rest
- imglorp 7mo ago
  Yes, true, but at least the fire won't spread through this one point. Hopefully all of your upstreams can be persuaded to pin also.
  - franktankbank 7mo ago
    Doesn't a single compromised action in the chain cause the whole to be fucked? Pinning the top level doesn't prevent any spread.
    - teaearlgraycold 7mo ago
      Might want to vendor everything?
      - franktankbank 7mo ago
        I don't know what this means in this context.
        7mo ago