9 ms·
Any program can be a GitHub Actions shell
- mkoubaa 2y agoThe next DOOM port
- leonheld 2y agoHas been done: https://youtu.be/Z1Nf8KcG4ro?t=1107 https://youtu.be/Z1Nf8KcG4ro?t=1107
- pseufaux 2y agouv --script
- digianarchist 2y agoOne cool undocumented GitHub Actions trick I spotted at work was the ability to use wildcards to match repository_dispatch event names: on: repository_dispatch: - security_scan - security_scan::* Why would you want to do this? We centralize our release pipelines as it's the only way to force repositories through a defined reusable workflow (we don't want our product teams to have to maintain them). This allows us to dispatch an event like so: { "event_type": "security_scan::$product_name::$version", "client_payload": { "field": "value" } } Then it is far easier to identify which product and version a workflow is running when looking in the Actions tab of our central release repository.
- larusso 2y agoNice trick!
- c0wb0yc0d3r 2y agoHow have you found this centralized approach to work for you? Does your org require everything be built in exactly the same way? I’ve just joined an organization that is trying to do similar, but in practice it seems nearly worthless. Templates are frequently broken, or written in ways that expect code to be built in a particular manner without any supporting docs describing the requirements.
- digianarchist 2y agoWe centralize releases but we don't centralize builds. That would remove too much autonomy from teams. We don't use templates for any of this. Our interface is the payload sent with repository_dispatch, a few metadata files in the repository (which we fetch) and a GitHub application that allows us to update the PRs with the release status checks. GitHub doesn't have a great story here, ideally we would want to listen to CI events emitting from a repo and run workflows as a reaction. The reusable story on a modular basis is better but here we're missing features that we would need to move some of workflows into Actions. Notably - Action repos need to be public.
- connorgurney 2y agoOnly to use the actions within on public repositories. https://docs.github.com/en/actions/sharing-automations/sharing-actions-and-workflows-from-your-private-repository https://docs.github.com/en/actions/sharing-automations/shari...
- digianarchist 2y agoTo clarify my previous comment. Actions cannot be private for public repositories. Most of our repos are public.
- markus_zhang 2y agoWait I can finally write C for our CI/CD in production and call it a low level system job. Probably can write assembly too.
- wilg 2y agoput the C in CI
- Tabular-Iceberg 2y agoAnd here’s the library for it: https://github.com/tsoding/nob.h https://github.com/tsoding/nob.h
- throw10920 2y agoAs long as other readers of the action are aware of what's happening, this seems pretty useful. There's been many adventures where my shell script, starting out as a few lines basically mirroring what I typed in by hand, has grown to a hundred-line-plus monster where I wish that I had real arrays and types and the included batteries in the Python stdlib. I'm definitely not going to use this to implement my company's build actions in elisp.
- deleted 2y ago[deleted]
- hulitu 2y ago> Any program can be a GitHub Actions shell systemd, echo "1" > /proc/sys/kernel/panic, echo > /bin/bash, etc.
- pjerem 2y agoDoesn’t mean that the shell is given more permissions than other commands.
- jstrieb 2y agoI've used this in the past to force bash to print every command it runs (using the -x flag) in the Actions workflow. This can be very helpful for debugging. https://github.com/jstrieb/just.sh/blob/2da1e2a3bfb51d583be0b14969bd81fea696d039/.github/workflows/test.yml#L20 https://github.com/jstrieb/just.sh/blob/2da1e2a3bfb51d583be0...
- 0xbadcafebee 2y agoFYI with pipefail enabled, if one of the pipes fails, your step will fail, there will be no error output, you won't know why it failed. Pipefail also doesn't prevent more complex error states. For example this step from your config: curl -L "https://github.com/casey/just/releases/download/${{ matrix.just-version }}/just-${{ matrix.just-version }}-x86_64-apple-darwin.tar.gz" \ | sudo tar -C /usr/local/bin -xzv just Here's the different error conditions you will run into: 1. curl succeeds, sudo succeeds, tar succeeds, but just fails to extract from the tarball. Tar reports error, step fails. 2. curl succeeds, sudo succeeds, tar fails. Sudo reports error, step fails. 3. curl succeeds, sudo fails. Shell reports error, step fails. 4. curl begins running. sudo begins running in a subshell/pipe. tar begins running under the sudo pipe, extracting half of the just binary. curl fails due to network error. Due to pipefail being enabled, shell exits immediately. There is no error message. A corrupt executable is left on-disk (which will be attempted to run if your step had failure-skipping enabled)
- jchw 2y ago> there will be no error output, you won't know why it failed That's probably why the -x is there. (Well, that and if something like curl or sudo fails it tends to output something to stderr...) > Pipefail also doesn't prevent more complex error states ... A corrupt executable is left on-disk (which will be attempted to run if your step had failure-skipping enabled) If I'm reading right it seems like you're suggesting is that the case pipefail doesn't handle is if you explicitly ignore the exit code. That doesn't exactly seem like the most concerning catch 22, to be honest.
- 0xbadcafebee 2y ago
- ZiiS 2y agoYou can also trick the default shell 'bash' into running any program.
- cdata 2y agoI wonder if you could pair this with nix e.g.,: - shell: nix develop --command {0} run: ...
- _cenw 2y agoIn my experience, the default VM size is so slow, you probably don't want Nix on a workflow that doesn't already take minutes. Even with a binary cache (we used R2), installing Lix, Devbox and some common tools costs us 2 1/2 minutes. Just evaluating the derivation takes ~20-30 seconds.
- manx 2y agoIs there a way to cache the derivation evaluation?
- chuckadams 2y agoYou can cache arbitrary directories in github actions, but the nix package cache is enormous and probably bigger than GH's cache system will allow. Restoring multi-gig caches is also not instant, though it still beats doing everything from scratch. Might be more feasible to bake the cache into a container image instead. I think any nix enthusiast is still likely to go for self-hosted runners though.
- _cenw 2y agoThe default cache action also has issued with anything that isn't owned by the runner user, and caches are per-repository, so you can't just have one cache like you do for binary caches.
- adobrawy 2y agoYou can use a self-hosted runner with an image that has anything pre-loaded.
- tadfisher 2y agoYes, we do this, although you need to do `nix develop --command bash -- {0}` to make it behave as a shell.
- emilfihlman 2y agoI mean, exec already exists, so you can become anything anyways.
- PhilipRoman 2y agoI've always been confused by how the CI script is evaluated in these yaml based systems. It is written as an array of lines, but seems to be running in a single context (local variables, etc.), but also there is some special handling for exit code of each line. I'm not sure how exec would work.
- Arnavion 2y ago>It is written as an array of lines, but seems to be running in a single context (local variables, etc.) [...] It's documented for GH at https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsshell https://docs.github.com/en/actions/writing-workflows/workflo... >>The shell command that is run internally executes a temporary file that contains the commands specified in the run keyword. ... and also mentioned in the article submitted here: >>If the command doesn't already take a single file as input, you need to pass it the special {0} argument, which GitHub replaces with the temporary file that it generates the template-expanded run block into. --- >[...] but also there is some special handling for exit code of each line. As you can see in the defaults in the first link, the Linux default is to run bash with `-e`
- emilfihlman 2y agoI mean technically GitHub could intercept the exec syscall, but given that the shell allows executing other programs that call fork and exec, you can't really block it (and why would you, really). Highly likely that exec would work.
- cturner 2y agoOur generation shuddered in terror when we were asked to translate a spreadsheet to code while the spreadsheet continued to evolve. This generation will shudder when they are asked to bring discipline to deployments built from github actions.
- adastra22 2y agoWhat is undisciplined about this?
- TheDong 2y agoyaml is roughly as disciplined as malbolge. Writing any meaningful amount of logic or configuration in yaml will inevitably lead to the future super-sentient yaml-based AI torturing you for all eternity for having taken any part in cursing it to a yaml-based existence. The thought-experiment of "Roko's typed configuration language" is hopefully enough for you to realize how this blog post needs to be deleted from the internet for our own safety.
- adastra22 2y agoI literally have no idea what you are talking about. Declarative languages are great for specifying these sorts of things.
- TheDong 2y ago> Declarative languages are great for specifying these sorts of things Yes, good declarative languages are. I'm a happy nix user. I like dhall. cel is a cool experiment. jsonnet has its place. I stan XML. A language with byzantine type rules, like 'on: yes' parses the same as "true: true", but only in some languages (like ruby's built-in yaml parser for example), but not others, and only with some settings, is not it chief. It isn't even one language, since most yaml parsers only have like 90% coverage of the spec, and it's a different percent, so the same yaml document often won't be parsed the same even by two libraries in the same programming language. It's really like 20 subtly incompatbile languages that are all called "yaml". It is indefensible in any context. Github actions should have been in starlark, xml, or even lisp or lua.
- ViperCode 2y agohow many workflows could be simplified this way without sacrificing debuggability or security."
- z3t4 2y agoWhat are the advantages with Github CI yaml over just a bash script, eg run: pipeline.sh ?
- hmhhashem 2y ago- You get an overview in the Github UI for each step and can expand/collapse each step to inspect its output. - You can easily utilize Github actions that others have contributed in your pipeline. - You can modularize workflows and specify dependencies between them and control parallel executions. I'm sure there are more. But the main advantage is you don't need to implement all these things yourself.
- PhilipRoman 2y agoFor #1, you can output section markers from any software: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#grouping-log-lines https://docs.github.com/en/actions/writing-workflows/choosin... (I've only used this feature with GitLab)
- hmhhashem 2y agoThanks, I didn't know about this!
- noirscape 2y agoThat second one sounds more like a security risk to me than a feature.
- dharmab 2y ago- you have an automatic, managed GitHub API token which is useful for automating releases and publishing artifacts and containers - if your software is cross platform you can run jobs across a variety of OSes and CPU architectures concurrently, e.g. building and testing natively on all platforms - you have access to a lot of contextual information about what triggered the job and the current state of the repo, which is handy for automating per-PR chores or release automation - You can integrate some things into the GitHub Web UI, such as having your linter annotate the PR line-by-line with flagged problems, or rendering test failures in the web page so you don't have to scan through a long log for them - You have a small cache you can use to avoid redownloading/rebuilding files that have not changed between builds Ideally you do as much as possible in a regular tool that runs locally (make/scripts/whatever) and you use the GitHub CI config for the little bit of glue that you need for the triggers, caching and GitHub integrations
- dolmen 2y agoThis gives me hope to ease running Go code for CI jobs directly from GitHub workflow YAML files using goeval [1]. However goeval doesn't yet have direct support for file input (only stdin), so shell tricks are needed. So far the way is: run: | go run github.com/dolmen-go/goeval@v1 - <<'EOF' fmt.Println("Hello") EOF but this requires a bit of boilerplate. Disclaimer: I'm the author of goeval. [1] https://github.com/dolmen-go/goeval https://github.com/dolmen-go/goeval
- michaelmior 2y agoWhy do you need any shell "tricks"? Wouldn't this work? go run github.com/dolmen-go/goeval@v1 - < file.go
- mdaniel 2y agoI believe the difference is that in your example file.go needs to live in the repo (or created by a previous step:) whereas reading from stdin allows writing the logic in the .yaml itself, and thus is subject to uses: or other reuse tricks
- michaelmior 2y agoSure. But the previous post seemed to suggest that tricks were needed because goeval doesn't support reading from files.
- dolmen 2y agoI'm trying to use goeval as a GitHub workflow shell, as this is the subject of the blog post. GitHub requreq {0} to be present on the shell line.
- mdaniel 2y agoI went to the repo to find out if your program automatically converted spaces into tabs but it seems it is more oriented toward oneliner expressions If you're going to plug your toy in this context, showing something more relevant than "print hello" would have saved me the click
- _def 2y agoAh I didn't know of the shell directive. Basically an equivalent to #! in shell scripts I guess: https://en.wikipedia.org/wiki/Shebang_%28Unix%29 https://en.wikipedia.org/wiki/Shebang_%28Unix%29
- latexr 2y agoNot exactly. > If the command doesn't already take a single file as input, you need to pass it the special `{0}` argument, which GitHub replaces with the temporary file that it generates the template-expanded `run` block into. It seems to be writing your script to a file, then using an executable to run your file. That ignores any shebang.
- nickysielicki 2y agoWould be cool to use this with nix shell shebangs
- greener_grass 2y agoMy experience is that the less done in GitHub actions the better. I tend to prefer either: - Using a build-system (e.g. Make) to encode logic and just invoke that from GitHub Actions; or - Writing a small CLI program and then invoke that from GitHub Actions It's so much easier to debug this stuff locally than in CI. So an interesting trick, but I don't see where it would be useful.
- latexr 2y agoI don’t see the post as the author suggesting you do this, but informing that it can be done. There’s a large difference. Knowing the possibilities of a system, even if it’s things you never plan on using, is useful for security and debugging.
- 6LLvveMx2koXfwn 2y agoMy take was that it is not useful, definitely, categorically not useful. It is a potential security hazard though. Especially for 'exploring' self-hosted runners.
- donatj 2y agoOur workflows amount to essentially - make build - make test We got bought out, and the company that bought us' workflow files are hundreds and hundreds of lines long, often with repeated sections. Call me old school, but I want to leave YAML town as soon as possible.
- lucasyvas 2y agoI may be targeted for calling this the “correct” way, but it is - it’s the only correct way. Otherwise you need complicated setups to test any of the stuff you put up there since none of it can be run locally / normally. GitHub Actions, like any CI/CD product, is for automating in ways you cannot with scripting - like parallelizing and joining pipelines across multiple machines, modelling the workflow. That’s it. I would really appreciate an agnostic templating language for this so these workflows can be modelled generically and have different executors, so you could port them to run them locally or across different products. Maybe there is an answer to this that I’ve just not bothered to look for yet.
- nonethewiser 2y agoWho would the potential bad actor be here? Someone whose committing to your repo right? I guess the risk is that they add something malicious in the commit and you dont see it. Which is maybe obfuscated to some extent by this little known fact. But all the malicious code would be there in the open just like any other commit. I mean, it seems like it would either take not noticing the malicious code which is always a threat vector. Or seeing it and mistakenly thinking "aha, but you arent actually running it!" and then letting it through based on that (which is of course ridiculous). Or there is some other way to exploit this that I'm unaware of. Edit: OK, maybe this is a little better. Re-write some malicious bash look alike somewhere outside the repo, install it from github actions (make it look like you are updating bash or something) and then its doing the bad thing.
- woodruffw 2y agoNot committing necessarily; there are plenty of GitHub Action triggers that a workflow can use that allow third-party use. I don’t think there’s a really direct security risk here, per se: it’s just another way (there were plenty already) in which write implies execute in GHA.
- donatj 2y agoI mean it's just a shell script jammed into YAML for reasons. The shell is just the shebang of said script
- aljarry 2y agoGithub Actions Runner code is pretty easy to read, here's a specific place that define default arguments for popular shells / binaries: https://github.com/actions/runner/blob/main/src/Runner.Worker/Handlers/ScriptHandlerHelpers.cs#L18 https://github.com/actions/runner/blob/main/src/Runner.Worke..., it is exported through a method ScriptHandlerHelpers.GetScriptArgumentsFormat. In ScriptHandler.cs there's all the code for preparing process environment, arguments, etc. but specifically here's actual code to start the process: https://github.com/actions/runner/blob/main/src/Runner.Worker/Handlers/ScriptHandler.cs#L331 https://github.com/actions/runner/blob/main/src/Runner.Worke... Overall I was positively surprised at simplicity of this code. It's very procedural, it handles a ton of edge cases, but it seems to be easy to understand and debug.