15 ms·

GitHub Actions could be so much better

412 points by woodruffw 3y ago

Cub3 3y ago
Just give me yaml anchors! I'm so sick of having to duplicate pieces of my actions constantly
darepublic 3y ago
I don't see how you love something that makes you jump through these hoops: > In this particular case, it took me 4 separate commits (and 4 failed releases) to debug the various small errors I made: not using ${{ ... }}5 where I needed to, forgetting a needs: relationship, &c
- vegardx 3y ago
  There's tools like Act[0] that tries to solve this, but this has been an issue with CI systems since they were invented. [0] https://github.com/nektos/act https://github.com/nektos/act
  - 5e92cb50239222b 3y ago
    Also reused by gitea for their CI runner. I was quite impressed by that feat, pretty neat. https://gitea.com/gitea/act_runner https://gitea.com/gitea/act_runner
- hanniabu 3y ago
  Isn't that just programming? That's not much different than saying you forgot a bracket and had to make another commit to make it work. Granted, it would be nice if they had some linter.
  - matthewaveryusa 3y ago
    This is a usecase where chatgpt works great! I usually pass this kind of ops DSL in gpt and it will find my mistakes. Github actions and graphanaQL code, oof, they just click for me.
    - chc 3y ago
      I'm not gonna post my employer's proprietary code into ChatGPT.
      - matthewaveryusa 3y ago
        Don't fixate on the details, fixate on the larger point. your specific situation of having code self-hosted and an employer that cares deeply about their proprietary code is a detail beside the point. LLMs shine at fixing up DSLs that you're unfamiliar with. chatgpt to llm is kleenex to facial tissue.
  - 5e92cb50239222b 3y ago
    Generally programming these days does not require you to submit build jobs or jump through other hoops to catch trivial mistakes. Although it depends on what you do, embedded developers and game programmers writing for consoles might disagree. It really feels like a throwback to the punch card era. I too don't enjoy writing CI scripts (despite knowing Linux administration and shell scripting quite well), it always takes an inordinate amount of time, but it also saves much more over the long term.
- garciasn 3y ago
  We use Github Actions and we just don't have any issues with it outside the first time we set it up for each repo. Then we make 100s of commits a week and it does its thing and our work goes live a few seconds later. That's why I love it. Could things be better? Of course; that's how software is--and this should resonate with most folks on this site. But just because some product isn't infallible doesn't mean we can't love it too.
  - moralestapia 3y ago
    >Then we make 100s of commits a week and it does its thing and our work goes live a few seconds later. Wow, computers doing what they're supposed to do. Pretty impressive ... The UX on GH actions is crap, other CI/CD solutions give you way more control and tooling. GH really needs to step up their game on this one.
    - garciasn 3y ago
      I am not sure why you’re being so snarky, but I did address that it could be better. The thing for us is that we’re already paying for GitHub and Actions are included. I know, for a fact, GH is/was working with stakeholders using their software to get feedback on what could be better. So they are trying! I’m not here being a fanboy by any means; I’m just saying it does what I need it to do and I’m an active user who is pleased with the service provided. YMMV.
  - neuromanser 3y ago
    > we just don't have any issues with it outside the first time we set it up for each repo But that's exactly what a lot of commenters complain about.
- layer8 3y ago
  Yeah, I wouldn’t want to use any automation that you can’t also easily and quickly test locally.
  - Pxtl 3y ago
    Fun fact: Microsoft had a plan to provide that! They canned it. https://github.com/microsoft/azure-pipelines-agent/pull/2687/files#diff-570b52bcb927a5365c22d17a21e8e19e1ba3427c7f3ec16ea64b7b3d14e4ee20 https://github.com/microsoft/azure-pipelines-agent/pull/2687...
  - bdcravens 3y ago
    Act can do most of it locally https://github.com/nektos/act https://github.com/nektos/act
deleted 3y ago
[deleted]
tao_at_garden 3y ago
The git commit, push, wait loop is terrible UX. Users deserve portable pipelines that run anywhere, including their local machines. I understand Act [1] goes some way to solving this headache but it's by and large not a true representation. There are many pipelines you can't run locally, because they're production, for example, but there's no reason why we can't capture these workflows to run them locally at less-critical stages of development. Garden offers portable pipelines and then adds caching across your entire web of dependencies. Some of our customers see 80% or higher reductions in run times plus devs get that immediate feedback on what tests are failing or passing without pushing to git first using our Garden Workflows. We're OSS. [2] [1] https://github.com/nektos/act https://github.com/nektos/act [2] https://docs.garden.io https://docs.garden.io
- baggachipz 3y ago
  Act's incompleteness has had me barking up the wrong tree many times. At this point I've temporarily abandoned using it in favor of the old cycle. I'm hoping it gets better in time!
  - frodowtf 3y ago
    I don't get why GitHub doesn't adopt it and make it a standard. Especially the lack of caches is annoying.
- andrewstuart2 3y ago
  Yeah, it seems like we lost a lot of the "CI shouldn't be a snowflake" when we started creating teams that specialize in "DevOps" and "DevOps tools." Once something becomes a career, I think you've hit the turning point of "this thing is going to become too complicated." I see the same thing with capital-A Agile and all the career scrum masters needing something to do with their time.
- candiddevmike 3y ago
  If folks just had actions target make or bash scripts instead of turning actions into bash scripts none of this would be an issue. Your CI/CD and your devs should all use the same targets/commands like `make release`.
numbsafari 3y ago
> Because GitHub fails to distinguish between fork and non-fork SHA references, forks can bypass security settings on GitHub Actions that would otherwise restrict actions to only “trusted” sources (such as GitHub themselves or the repository’s own organization). How is this not resolved? Easily bypassing security controls is a major security issue. Yes, you need to convince someone to use your SHA, but social engineering is usually the easy part.
nthState 3y ago
Most of my problems would vanish if there was an official way to run workflows locally, something like: shell: run_workflow name=MyJob in=MyWorkflow.yml params={}
- avtar 3y ago
  I guess you have to ask yourself what's in MyWorkflow.yml that can't be in a script and run locally?
  - mason55 3y ago
    Well, that's why the word "official" is in there. Obviously you could mock up the entire github actions scaffolding locally and have it inject environment variables and support all the actions and everything, but keeping that up to date will be a nightmare if you're doing it on your own. Obviously you could put the whole CI/CD into a bash script and not use any features or functionality provided by github actions but there are plenty of nice things that it does and it would be a shame to not use any of them.
    - avtar 3y ago
      I didn't mention it in my reply but I was referring to the complaint in the article that the author wants to be able to debug locally without having to push changes to GitHub first. The top comment by MoreQARespect and others highlight the benefits of scripting as much as possible so that processes like builds and cutting releases can be run and tested locally. > Obviously you could mock up the entire github actions scaffolding locally Not sure if anyone's advocating for implementing GA entirely. You can at least automate project-specific bits as much as possible using scripts, and then have the CI environment use the same automation. That allows for more local debugging than overusing GA.
nafizh 3y ago
What's the technology behind the blog? It's clean, minimal and beautiful.
- junke 3y ago
  Jekyll
- woodruffw 3y ago
  It's a Jekyll site. I originally built it off of a popular theme (back in 2014 or so), but these days the theme is just a custom thing that I've cobbled together. (So, in a sense, there's no real technology "behind" it. It's just Markdown with a little HTML templating, with Jekyll as the SSG.)
  - nafizh 3y ago
    Thanks for the reply. Is the code available by any chance?
    - woodruffw 3y ago
      Unfortunately not -- it used to be in a public repo, but I also track drafts and other things now that I don't want to be public until they're ready. I'll see if I can clean it up into a public repository, but no guarantees :-)
    - fooker 3y ago
      Right click -> view page source.
Pxtl 3y ago
I mean GH Actions is basically a re-brand of Microsoft's "Azure Pipelines". As somebody who used all previous incarnations of TFS/VSTS/AzDO build and release pipelines: they are not good at this. This is not a team with a record of success. That Azure Pipelines is moderately usable only happened because they failed literally every other approach they tried. There was a project to allow you to run the pipelines locally so you could do the edit-run-debug loop on your own private environment without committing. It was, of course, canned. https://github.com/microsoft/azure-pipelines-agent/pull/2687/files#diff-570b52bcb927a5365c22d17a21e8e19e1ba3427c7f3ec16ea64b7b3d14e4ee20 https://github.com/microsoft/azure-pipelines-agent/pull/2687... However, there are tools to improve QOL. For example: https://marketplace.visualstudio.com/items?itemName=ms-azure-devops.azure-pipelines https://marketplace.visualstudio.com/items?itemName=ms-azure... A vscode extension that's syntax-aware. Now, I'll be a bit controversial: if they'd used XML instead of YAML, you could have an xmlns declaration up-top that would give you validation in most decent code editors without user intervention. XML is awful, but it has a lot of useful features that we gave up when we threw the baby out with the bathwater.
- woodruffw 3y ago
  > I mean GH Actions is basically a re-brand of Microsoft's "Azure Pipelines". As somebody who used all previous incarnations of TFS/VSTS/AzDO build and release pipelines: they are not good at this. This is not a team with a record of success. That Azure Pipelines is moderately usable only happened because they failed literally every other approach they tried. I was under the impression (which might be wrong!) that GHA was an independent project within GitHub that was well underway before the acquisition. Are you saying that GHA was rebuilt on top of AzP, that it's just a relabeling of AzP, or something else? (I have no particular dog in it being one way or the other, but I'm curious about the history here.)
  - Pxtl 3y ago
    They share a lot of code. My understanding is that it was an MS project first, but I might have that backwards. > GitHub Workflows execute on runners. The runner code is essentially a fork of the Azure Pipelines code, so it's very similar. It's also cross-platform and you can also use hosted or self-hosted runners. https://learn.microsoft.com/en-us/dotnet/architecture/devops-for-aspnet-developers/actions-vs-pipelines https://learn.microsoft.com/en-us/dotnet/architecture/devops...
riv991 3y ago
The feature I'd love is history. I'd like to know if my builds are getting slower over time. I'd be able to detect flaky tests automatically. It seems basic, but I know third-party solutions exist for this. It's out of the box in Circle CI and Buildkite and feels like it should be here.
- atomicfiredoll 3y ago
  I was with a team that moved from Jenkins to Actions. Jenkins has a lot of issues, but if you knew what you were doing, at least it was easy to see the history and at a glance know the project pipelines were green/healthy. It was super useful for seeing where flake was in tests or observing changes to coverage over time. After the team migrating to actions swept though, we ran without this basic stuff for years because nobody had the time to figure out what 3rd party tool to use, money to pay for it, or capacity to re-implement the functionality. It made dealing with test failures or flake that crept in awful.
simse 3y ago
A linter can catch some obvious errors: https://github.com/rhysd/actionlint https://github.com/rhysd/actionlint. But yes, I agree, it's not a fun debugging experience.
- woodruffw 3y ago
  Yep, actionlint is great! I've used it successfully both to lint my own workflows, and to lint third-party workflows for (basic) security issues. Unfortunately, it can't lint actions themselves, only workflows that call actions[1]. This is a substantial deficiency, especially for users (like me) who write and maintain a decent number of actions. [1]: https://github.com/rhysd/actionlint/issues/46 https://github.com/rhysd/actionlint/issues/46
- bastardoperator 3y ago
  act would also be helpful here in terms of debugging and development of Actions workflows. https://github.com/nektos/act https://github.com/nektos/act
theideaofcoffee 3y ago
As someone who has recently sunk a considerable amount of time in modernizing some pipelines, workflows, what have you, I feel this deep in my bones. The commit-test-fail-recommit test loop is painful and something I’ve experienced all too well. Trying to work around product limitations brought about by whoever concocted the system on GitHub’s side to save customers from themselves and declared it good is painful. Trying to debug simple issues is painful. I’m glad I’m not alone.
MoreQARespect 3y ago
There are two types of github actions workflows you can build. 1) Program with github actions. Google "how can I send an email with github actions?" and then plug in some marketplace tool to do it. Your workflows grow to 500-1000 lines and start having all sorts of nonsense like conditionals and the YAML becomes disgusting and hard to understand. Github actions becomes a nightmare and you've invited vendor lock in. 2) Configure with github actions. Always ask yourself "can I push this YAML complexity into a script?" and do it if you can. Send an email? Yes, that can go in a script. Your workflow ends up being about 50-60 lines as a result and very rarely needs to be changed once you've set up. Github actions is suddenly fine and you rarely have to do that stupid push-debug-commit loop because you can debug the script locally. Every time I join a new team I tell them that 1 is the way to madness and 2 is the sensible approach and they always tepidly agree with me and yet about half of the time they still do 1. The thing is, the lack of debugging tools provided by Microsoft is also really not much of a problem if you do 2, vendor lock in is lower if you do 2, debugging is easier if you do 2 but still nobody does 2.
- woodruffw 3y ago
  This is a great perspective, and one I agree with -- many of the woes associated with GitHub Actions can be eliminated by treating it just as a task substrate, and not trying to program in YAML. At the same time, I've found that it often isn't sufficient to push everything into a proper programming language: I do sometimes (even frequently) need to use vendor-specific functionality in GHA, mark dependencies between jobs, invoke REST APIs that are already well abstracted as actions, etc. Re-implementing those things in a programming language of my choice is possible, but doesn't break the vendor dependency and is (IME) still brittle. Essentially: the vendor lock-in value proposition for GHA is very, very strong. Convincing people that they should take option (2) means making a stronger value proposition, which is pretty hard!
  - MoreQARespect 3y ago
    No, you're right it's not necessarily a good idea to be anal about this rule. E.g. If an action is simple to use and already built I use it - I won't necessarily try to reimplement e.g. upload artifacts step in code. Another thing I noticed is that if you do 1 sophisticated features like build caching and parallelization often becomes completely impractical whereas if you default to 2 you can probably do it with only a moderate amount of commit-push-debug.
syndicatedjelly 3y ago
gitlab-runner allows you to run scripts locally for GitLab, but even that is lacking (any env secrets stored in GitLab? Script is going to fail). I really, really wish there was a way to clone the GitHub Actions or GitLab runner environment, spin up a runner locally and test. That would shave off 95% of the wait time
rurp 3y ago
It's hard for me to articulate exactly why but I really dislike the information layout in GH Actions. Compared to other CI/CD tools, it's harder to debug problems or get a sense of the state of a pipeline. Part of the problem is the number of clicks it takes to look at various step logs. Some of the controls are unintuitive, but maybe that's just me.
- whalesalad 3y ago
  This could be said for a lot of GitHub. The fact that you need to open a disclosure menu to edit a PR is wild.
  - mock-possum 3y ago
    Their comment system just doesn’t feel right compared to Bitbucket’s either - I think it’s the distinction between comments that are part of a review, versus comments that are ‘just’ comments
atomicnature 3y ago
This is a real problem. I think the problem stems from the unasserted assumption that declarative YAML is not really coding/debugging. The root problem is the assumption that infrastructure specifications are second-class citizens when it comes to managing a software ecosystem. Due to this, one rarely sees any sort of strong tooling support for creating, updating, debugging and extending various infrastructure activities. The unpleasant truth is that we live in the dark primitive days of infrastructure management.
cwales95 3y ago
This is my normal CI/CD workflow -- even for GitLab. Debugging why CI/CD doesn't work is one of the worst headaches you can imagine.
joshstrange 3y ago
I couldn't agree more with the pain of debugging a GH Actions run. The /only/ tool you have is the ability to re-run with debug on. That's it. I have so many "trash" commits trying to fix or debug a pipeline and so much of it's just throwing stuff at the wall to see if it sticks. Very basic things, like having reusable logic, is needlessly complex or poorly documented. Once I figured out how to do it it was fairly easy but GitHub's docs were terrible for this. They made it seem like I had to publish an action to get any reusability or put it in a different repo. Turns out you can create new yaml files with reusable logic but make sure you put them in the root of the workflows folder or they won't work, go figure. It's just incredibly painful to work on GH Actions but once you have them working they are such a joy. I really wish there was some kind of local runner or way to test your actions before committing and pushing.
- TeeMassive 3y ago
  I've tried running the GitHub runner image (or maybe an imitation) and it was really painful to setup and to get some things working. I just let it go after 2 days. And it's not just Github. The others big CI platform are not really better in terms of workflow and integration. Now I just script everything to the maximum.
- giobox 3y ago
  > I have so many "trash" commits trying to fix or debug a pipeline and so much of it's just throwing stuff at the wall to see if it sticks. One tool is to use draft PRs for this - you can run changes to your action YAML from the draft PR. When you are happy just squash the commits as you see fit on a "real" PR to merge the changes in without the mess. I've found draft PRs for debugging/developing GH action logic to be pretty reasonable.
  - lambda_garden 3y ago
    Since some action depend on the branch / tag you are on this is not always possible.
    - 3y ago
he0001 3y ago
We recently converted from Jenkins to GitHub Actions (complete rewrite) build pipeline. Jenkins had its issues and warts, but holy Beelzebub what a monster GitHub Actions is. It pure downright evil and creates all sort of headaches. Opaque, zero clues on what broke. Glacial slow even if you throw the biggest instance at it. It’s all enterprise and 1000s of repositories big company. I’m spending more time finding bugs in the build pipeline than the time I spend doing other things. And it’s all GitHub actions fault. I hate it with a passion.
sitzkrieg 3y ago
one thing to be aware of, if you use windows, and submodules in a repo, ci will fail to clone over 50% of the time (with internal retries) with early eof errors it's apparently a fixed problem in upstream openssl beta that isnt bundled yet, but i think a custom runner would be better too (but not an option at my org) still, some days it gets in a rut and quickly fails for 10+ runs all billed at full rate. the entire eng group is severely annoyed by it because we cross test all oses on each commit so its becoming an expensive problem. basically 3x the billing on windows ci
ZeWaka 3y ago
re: `pull_request_target` and fork PRs - You can get around the !Even More Fun! limitations this has (GITHUB_TOKEN being read-only) by having a workflow called by another workflow. For example, I wrote a set of workflows[1] to automatically apply a label after 2 'review approvals'. First, a 'dummy' workflow triggering off PRs (in that context) that uploads an artifact: the PR number Second, the 'real' workflow that runs in the context of the actual repository, set to be `on: workflow_run: workflows: - Final Review Labeler` - this pulls in the artifact, runs a GraphQL query, and applies the label if applicable. [1]: https://github.com/goonstation/goonstation/blob/master/.github/workflows/final_review_labeler.yml https://github.com/goonstation/goonstation/blob/master/.gith...
giobox 3y ago
My personal wish is for the ability to attach HTML reports the the action runs without having to use the current actions/upload-artifact etc. Particularly for test builds, I very often just want to quickly view the output HTML report. The current approaches I am familiar with are the aforementioned upload-artifact, or using GH pages, but GH pages is not great when you have multiple different reporting output for the same repo, or you wanna quickly view historical reporting rather than latest. I'd love a simple "attach-report" action that just put a link to an HTML report on the job summary, and clicking on it renders the html. Other automated CI/CD style systems have much richer support out of the box for dealing with HTML report capturing and viewing.
- dmart 3y ago
  Granted, it's not HTML, but you can append Markdown output directly to $GITHUB_STEP_SUMMARY without dealing with artifact uploads etc.
  - giobox 3y ago
    Thanks, I'm aware of this option too, but if you say have an HTML report with the results of 200 tests, no one has time to write some logic to parse the results to a markdown summary frankly. Just give me the link! Similar too if the reporting captures video/screenshots of a given bug - the original report UI is far easier to deal with. Many test frameworks already natively put out an HTML report UI etc, it's just harder to get to than it should be with GH actions today.
    - pests 3y ago
      > an HTML report with the results of 200 tests, no one has time to write some logic to parse the results to a markdown summary frankly. Not sure how GHA treats it, but markdown is a subset of HTML so by default all HTML pages are valid Markdown documents. No conversion required.
      - Aeolun 3y ago
        They just don’t show half of the stuff inside correctly.
andix 3y ago
I'm really not sure if we are using CI correctly. Sometime i think all those CI Templates should be replaced by just one executable that does everything, like a modern alternative to Makefiles (and there are a lot of build tools). So the CI pipeline would only call the build tool, like "./build containers push-to-registry release:1.0.0 run-tests" Those scripts can be tested and debugged everywhere. Also migrating to a different CI platform would be really easy.
- WirelessGigabit 3y ago
  While this is totally possible you lose a lot of the things that make GitHub Actions nice. Nice logs, annotations, individual steps that went wrong. Being able to see the status at a glance. Seeing 10 steps where 1 fails and 13 pass is nice.
  - andix 3y ago
    All those things the custom build tool would need to handle. It needs more or less the same functionality as GitHub actions and the marketplace. It would also need to integrate with different CI platforms to some extent for reporting and user interactions.
- pixl97 3y ago
  How does that integrate in with every else's tooling? At least in enterprise there are a ton of things like "SCA, SBOM, Compliance report, etc" that tie in with plugins and such. Also, why would a large commercial CI want to have their environment too open?
  - andix 3y ago
    They don’t want to have their environments too open, that’s why we are at this point. Those enterprise tools i know (like sonarqube for example) are mostly cli tools in their core. Can be integrated everywhere.
- gravlaks 3y ago
  https://dagger.io/ https://dagger.io/ may be an alternative. You can run it in GitHub actions somehow as well.
gchamonlive 3y ago
I am thinking about porting my ci/cd pipelines from Jenkins to GitHub actions to leverage autoscaling self-hosted runners, which I don't think there is an exact equivalent for Jenkins nodes, at least not one that is already implemented where I work. The pipelines though, are all configured using ansible. Applying the same principle I did to Jenkins, using it strictly for execution automation, logs, permission and the likes, and leaving the heavy lifting to a more trusty tool, I think I can avoid both major vendor lock-in and idiosyncrasies and nonsenses from the language. It boggles my mind how unrefined the GitHub actions flow is. It doesn't offer a good experience by a long shot.
kaikoenig 3y ago
Been through that git commit; git push; repeat cycle too much as well until i discovered https://github.com/mxschmitt/action-tmate https://github.com/mxschmitt/action-tmate which gives a shell in between steps, which does not help with all problems but sure it's makes it less painful at times.
- justinclift 3y ago
  Thanks, that looks super useful. :)
ctoth 3y ago
One thing that actually makes this slightly less horrible is the Gh tool. You can use gh run watch to tail your logs in the terminal and at least don't have to click through a million things just to see output.
- rane 3y ago
  I wrote a command-line tool that streamlines retrieving test results from GitHub Actions even further. Essentially parses jest/tsc/eslint errors in GHA jobs' logs for the current's branch PR. https://github.com/raine/ghtool https://github.com/raine/ghtool
- fijiaarone 3y ago
  Do you know a way to stream the gh run watch output?
natbennett 3y ago
For validating YAML I love CUE. It takes some work to define the schema — would def be nice if GitHub provided that. I found Nektos act pretty usable for local testing when I was doing a lot of GA work.
eviks 3y ago
Very good points, especially re. the security footguns
chrissoundz 3y ago
Struggling with this today, and numerous other days. It's so bad. Stop trying to build an operating system out of YAML. I'll always use and recommend Gitlab from now on. And what the earth is an "actions" anyway? How on earth is simple bash functions not just as suitable here? Instead you have some weird YAML scripting language. It's so bad. Why. Somebody please tell me. I'm losing my mind. It is a good reflection of the rest of the world though and why the worlds infrastructure is crumbling in many places.
- giobox 3y ago
  Executing bash statements/scripts/functions is the thing I struggle least with in GH actions personally, it's remarkably easy to execute shell steps. If you really want your entire build to be a shellscript the action executes, you can do just that with very little YAML.
  - Aeolun 3y ago
    Except it doesn’t execute quite like in bash, and every single command is it’s own little island. So exporting an environment variable doesn’t work for example.
    - trallnag 3y ago
      What are you talking about? You can put a whole Bash script into a single step.
watermelon0 3y ago
My number one complaint is that they still don't have managed ARM runners. Considering how overpriced runners are (compared to cloud instances), I'm sure they would make a ton of money here. My choices here are either to use a different CI solution (but I quite like Actions, and they are better integrated in the Github UI than alternatives), or to use custom self-hosted autoscaling runners (but there doesn't appear to be a good solution available at the moment, and I definitely don't want to managed infra for CI.)
- xjia 3y ago
  Check out https://dime.run/ https://dime.run/ they manage workers for you
jusonchan81 3y ago
We moved our CI to this orchestrator called Netflix Conductor. It’s highly customized internally with workers capable of running different kind of tasks etc. But since we did this, CI has been a wonderful experience. Things run smoothly and history can be tracked and easy to build variants by modifying or creating new workflows. We aim to open source this extension soon.
samiam_iam 3y ago
[dead]
artursapek 3y ago
GitLab’s CI is so much better than GH it’s not even funny
kitallis 3y ago
One of my biggest gripes with Actions is that it doesn't allow you to trigger workflows for a specific SHA. Only the branch HEAD. This is pretty much possible in all other CIs.
- tra3 3y ago
  You can pass a sha into your workflow and then check it out.
pxeger1 3y ago
I would like to give a strong recommendation for https://pre-commit.ci https://pre-commit.ci (no relation, just a happy user). The idea is that you write hooks (or use pre-existing ones) that check (and fix, where possible) your code before you commit, and then once you've pushed, the CI will check again, including automatically committing fixes if necessary. Anyway, it works brilliantly - unbelievably fast thanks to good automatic caching, and using the exact same setup on your local development machine to in CI negates a lot of these debugging problems. It's free for open source. It only does checks, not things like automatic releases, but it does those them well.
- globular-toast 3y ago
  This combined with tox is great for Python projects in particular. Tox automates creating virtualenvs for the right Python versions you want to test with then running your tests in them. It can also run static checks by issuing `skip_install = True` because you want to test the source code itself. You just need to run this in a Python container that has tox installed as a globally available tool and all versions of Python available in it (like https://github.com/georgek/docker-python-multiversion https://github.com/georgek/docker-python-multiversion <-- not maintained but easy to update). Here's some boilerplate to do all that: [tox] envlist = py{310,311} [testenv] passenv = PIP_CACHE_DIR deps = coverage # deps only for tox extras = testing # testing extras include pytest commands = pytest ... [testenv:check] passenv = PIP_CACHE_DIR PRE_COMMIT_HOME skip_install = True deps = pre-commit commands = pre-commit run --all-files --show-diff-on-failure
kevmo314 3y ago
I wish individual actions could be specified as Dockerfiles instead of yaml files. That would address a lot of issues that I've run into around build environments being slightly different than locally. Yes, I can wrap it with a simple docker action. It would be cool if that were automatic.
Nezteb 3y ago
This is why I want projects like Earthly to succeed: https://github.com/earthly/earthly https://github.com/earthly/earthly I want to be able to run my all of CI workflows on my local machine. I agree with other commenters that most of what CI does should be abstracted out into scripts or other non-CI tools. Unfortunately it's not easy to do that for large pre-existing CI setups, especially if a different team is the one maintaining your CI workflows.
- alexvs 3y ago
  Isn't Earthly out of the picture now? https://earthly.dev/blog/shutting-down-earthly-ci/ https://earthly.dev/blog/shutting-down-earthly-ci/
  - vladaionescu 3y ago
    Nope. Only Earthly CI is.
  - adamgordonbell 3y ago
    As @vladaionescu mentioned, Earthly is alive and well. We stopped the CI product exactly so we could focus on use cases like making Earthly in GitHub Actions better.
vladaionescu 3y ago
Founder of Earthly here - besides the build debugging difficulty, I would add that modern CI/CD repeats a lot of steps: downloading, installing and configuring dependencies, making things much slower than they should be. We built Earthly [1] to tackle these two problems specifically. We're open-source (10k stars). [1]: https://earthly.dev https://earthly.dev
jurassic 3y ago
Most of my frustrations with GHA arise when doing something useful conflicts with someone’s idea of security. For example, branch protection rules intended to stop devs from yoloing commits blocking me from pushing a version bump commit during a release workflow.
- Aeolun 3y ago
  The lack of exceptions for actions is bizarre. And then they build repository rules or whatever, and made the same mistake again! You can at least exclude github apps now, but then you need to run all your repo actions with an app installtion key :/
ris 3y ago
The single thing that I most detest about GitHub Actions is how they, by design, completely miss the point of containers. Having "actions" that are just "install language foo" is barely better than just publishing shell scripts or even the days of Travis. As a result, few GitHub workflows benefit from the immutability or reproducibility guarantees that can be provided by containers, and most workflows I interact with spend more than half their time running ridiculous installer scripts.
- brabel 3y ago
  Are you unaware that GH Actions can run jobs in containers?? https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container https://docs.github.com/en/actions/using-jobs/running-jobs-i...
ulizzle 3y ago
More official or blessed packages for sure. I agree with that.
kathlam 3y ago
It's clear we've all wrestled with GitHub actions and Jenkins in some shape or form. Debugging them can feel like being in an escape room, without any clues. I work at Trunk.io and we have a tool called CI Debugger. It helps you understand your workflows. Instead of getting lost in logfiles and commits, you can pinpoint where things are bogging down or going haywire. It can give you a clearer picture of what's happening under the hood. Like having a heat map for the CI/CD processes.
lijok 3y ago
People really need to stop writing scripts in yaml. Not to be dismissive but the issues outlined in this article, aside from that security footgun, are non-issues as soon as you start using github actions only for what it should be used - to invoke your scripts in parallel in response to an event. We've banned all but actions/checkout in our org and have a very healthy dev experience as a result. People are naturally guided towards writing scripts that they can run locally instead of for github actions specifically. Some real issues with github actions: 1. MacOS runners have ridiculously inconsistent IO performance causing 200x slowdown in some cases 2. Getting charged to the nearest minute is asinine and punishes highly parallelised (fast) workloads 3. GitHub OIDC endpoints constantly timing out
- fijiaarone 3y ago
  Yaml is the worst possible programming language, beating out even ant’s xml abomination. Unfortunately most of the work of GitHub actions is configuring the runners and setup data, which requires you to use lots of yaml to invoke unvetted public repositories (actions) written in JavaScript to do what is easier done as shell scripts, and then use shell scripts to workaround the limitations of actions. State is impossible, data is even more impossible, and caching is completely broken. And if you use your own runners, GitHub actions don’t even clean up their own garbage. I suspect it’s the same on public runners, but just statistically less likely to affect you, but a potential security vulnerability. I’ve been working 2 things: 1. A front end that crafts GHA workflow yamls from a sane configuration with common tasks baked in. 2. An event and reporting system so you can see what is happening during a workflow — not after — without having to scroll through GitHub’s hideous log output in html or download and unzip the full log.
k8svet 3y ago
I mean, it's embarrassing how bad it is. - (unrelated) build failures just randomly notifies the latest maintainer who happened to merge something? (Imagine you finding this out when your newly added maintainer pings you on Matrix and tells you 1: about this behavior, and 2: that your update/builds have been failing for a week without you knowing?!?!) - The cache action is horribly, trivially observably broken with seemingly no maintainer? - Can't tail the build log of a step if their UI poops or your tab unloaded after it started? - The complete lack of abstraction that might actually make workflows portable across worker nodes? pfft. - the default image is a travesty. I thought it was obnoxious how bloated it was and then I started digging in and realizing "Oh, some Microsoftie that didn't know Linux was put in charge of this". (saying this as a former FTE that knows). And there's no effort to allow a more slimmed down image for folks that, you know, use Nix? Or even just Docker? I'm in the process of migrating off GitHub and it's mostly because Actions shocked me to my senses. Too bad MS can't figure out how to retain virtually any Linux talent, and not just their cuffed-acqui-hires or Windows-devs-cosplaying. Even the well compensated ones head for the door. And I'll just say, I don't program in YAML because YAML is a disgrace wrought upon us by Go+Yaml enthusiasts that don't know any better fueled by senseless VC money shoveled at an overall ecosystem incognizant of modern, actually useful technology. edit: removing some of the blatantly identifying former-FTE trauma. Knowing what I know I should sell all my MSFT, but the market thinks differently.
- rijoja 3y ago
  Wasn't it obvious that something along these lines would happen when Microsoft took over Github?
becurious 3y ago
You’d think they could at least train GitHub Copilot to test run the workflow and point out immediate issues. We have to add an action to kill duplicate runs on triggers. The caching is marginal, at least on Windows runners because it takes forever to expand the tar ball. Not even sure from their docs / issues on the cache action if they did finally move it to use GNU tar. Having some way to get an interactive shell on fail would be a big step up for debugging issues. Otherwise we are back to print debugging in the actions or uploading artifacts at each step so we can inspect them.
bingemaker 3y ago
The current state of GA is much better than Jenkins. I don't like yaml, but this is still better than mucking around Jenkins configs. Fin
plonk 3y ago
The best approach is to develop completely local build scripts and add a few options to plug GitHub Actions' exclusive features into them. Make a build script that can take a cache folder's path as argument. Make it take a test report output path as another argument. etc. So your GHA workflow is: set up secrets (AWS and co.), set up Python, download cache to path X if possible, run the build in X and send reports to Y, publish reports in Y. And put as much work as possible inside the build script. We have a whole Python project for that. This is truly painless after the first development effort.
solatic 3y ago
GitHub Actions is a horrible CI/CD system. You cannot run steps in parallel on the same VM; container-based workloads are a second-class citizen. The first problem means that setting up local credentials and other environment dependencies cannot be parallelized (I'm looking at you, google-github-actions/setup-gcloud, with your 1m+ runtime... grrr), the second makes it quite difficult to put a Dockerfile in a repository to represent setup of the CI environment, and have both (a) the CI rebuild the container images when the image would change, pausing workflows depending on that image until the image is rebuilt, (b) not attempting to rebuild the image when its contents did not change, and immediately running workflows inside that image, including all dependencies already installed. No, in GitHub Actions, you will attempt to repopulate from cache on every single run. Of course, sometimes the cache isn't found, particularly because there's a 5 GB cache size limit (which cannot be enlarged, not even for payment) which cycles out FIFO. So if you go over the 5 GB cache, you might as well not have one. I deeply miss Concourse. Too bad Pivotal dismantled the team and it barely gets even maintenance work. Parallel tasks. Custom pipeline triggers. SSH into the CI environment to debug. Run one-off tasks in CI without needing to set up a pipeline. Transfer directory contents from one job to another without needing to create artifacts (which cost money to store if you're not careful about how long they should stick around for). GitHub Actions is a bastardized toy CI/CD system that only got popular because GitHub make it as simple as uploading a file to .github/workflows in any repository - no additional signup, no in-house ops required, everything you could want is just there. So let's be very clear about what GitHub Actions is good and what it's bad at - it's good at getting people to sign up, but it's not nearly powerful enough to be the "best" system once you start to outgrow the early features.
- lawnchair 3y ago
  solatic -- I have an existing solution that accounts for a lot of these issues you bring up. Would it be possible to pick your brain? Can you share your email or shoot me an email? lawnchair@lawnchair.net.
- bryandollery 3y ago
  Have you seen Tekton? (https://tekton.dev/ https://tekton.dev/)
- 3y ago
bob1029 3y ago
My experience with GitHub Actions is mostly managed by way of certain Azure Portal interactions these days. When I create a new Function app, it gives me the opportunity to enable GitHub integration. The experience for this is flawless, IMO. You provide your GH credentials, select the org/repo/branch, and then it will create the workflow file for you and push it automatically. It will also update the secrets in your repository settings to match what Azure expects on its end for deployment. By the time you get to look at your GitHub repo, the action is already running and will 100% complete successfully if you followed a standard/default project structure. The automatically-generated workflow files aren't perfect, but they're so close that it becomes trivial to tweak for additional build args or project arrangements. Just getting the secrets & related boilerplate configured makes the difference between me doing it right now vs maybe never. The consequence of always having proper CI/CD from day zero, even for the most trivial projects, seems profound to me. There exist some really happy paths now wrt GH actions, but you gotta be willing to get pretty hammered on the Microsoft koolaid to explore them.
varsketiz 3y ago
It's not really context switching. Author is debugging github actions, thats the context in which a few window changes seem minor. Context switching would be to get dragged to meeting, or getting asked by a colleague to explain how to set up github actions for their project.
remram 3y ago
Can I trigger a workflow from a different workflow yet? That seems very basic, and last time I looked I had to create a token that gives read/write access to every repo in every organization to do this. And then it didn't work because the docs for the trigger API is apparently wrong.
- Aeolun 3y ago
  As long as they’re in the same repository, yeah.
  - remram 3y ago
    Different repo same org
bysja 3y ago
Github Actions has been a pleasant introduction to CI/CD for me. I used to build locally and then git push and/or rsync everything directly to a VPS (Digitalocean Droplets in my case). But for collaboration, this workflow breaks down. Then, using Github as the "origin" repository becomes compelling, and Github Actions fit very nicely with that workflow.
elischleifer 3y ago
Another tool that tackles GitHub Action debugging head on: https://www.ci-debugger.io/ https://www.ci-debugger.io/ Wrap your GH Actions in a debugger breakpoint and connect into the live broken GH Action to inspect the machine, re-run commands etc.. At it's heart - realtime debugging for GitHub actions.
ivanjermakov 3y ago
I feel this pain. So much that now, when I need to set up new GitHub actions pipeline, I always create a new branch to push there until it's working. And then squash merge as a single commit.
Aeolun 3y ago
My biggest pain point is not being able to do proper ternary statements in workflows. And passing arrays to called workflows. My ‘runs-on’ statements need more than one label! You can fake it by passing a stringified JSON object, but that’s bizarre.
lenkite 3y ago
I wish Github Actions were Lua instead of YAML. One could use a restricted environment populated with predefined functions, a restricted set of modules and all the advantages of syntax checking and language help.
- lpapez 3y ago
  Congratulations, you just reinvented Jenkins but using Lua instead of Groovy.
  - lenkite 3y ago
    Not a bad thing. Also Lua is both smaller, saner and more performant than Groovy. (with LuaJIT)
deleted 3y ago
[deleted]
icecap12 3y ago
> And so, the question: why are there so few of them? Here is just a smattering of the official actions that don’t exist[...] This part of the article half triggered me and half made me laugh. I was recently on a GitHub webinar to listen to the pitch for CoPilot, but of course they also talked about Actions and GHAS. Anyway, during the overview of Actions, the presenters made a comment about how Actions was "backed by the community." It felt to me like a glaring admission that the product isn't mature. But in true Microsoft fashion, they charge you like it is.
krzema12 3y ago
I created https://github.com/typesafegithub/github-workflows-kt https://github.com/typesafegithub/github-workflows-kt and https://github.com/typesafegithub/github-actions-typing https://github.com/typesafegithub/github-actions-typing to address some of the mentioned issues. Some problems remain unsolved because it's just a Kotlin DSL that generates the YAML, but it does catch some issues early. Related blog post: https://dev.to/jmfayard/github-actions-a-new-hope-in-yaml-wasteland-1i9c https://dev.to/jmfayard/github-actions-a-new-hope-in-yaml-wa...
globular-toast 3y ago
A tight feedback loop is everything. I will not use tools that require long waits between tests. I want to edit, test, wait a matter of seconds to see the results. Seeing people push hundreds of commits to their CI pipeline trying to fix it by trial and error makes me cringe. I do not understand how some people accept such a workflow without thinking it could be better. When I was planning the CI set up at my job I made sure it could all be run locally and hammered this point home to developers. CI isn't magic, it's just some other computer running your shit. We use Gitlab CI which runs your script in a container. It's trivial to run the container locally with docker to test exactly what will happen in CI (although you do have to make a fresh clone of your repo for the mount, I thought about scripting this but in practice it was never needed that much). GitHub Actions seems much harder to test locally. There seems to be some projects to do it but they are too heavy for me to want to install. For this reason alone I don't like Actions.
dboreham 3y ago
I think most developers don't need to or don't have the time to dive deeply into their build systems, but having had to do so as part of a project to migrate to Gitea (which turned into a larger project to enhance/fix its act-based github actions clone), my take is: 1. Github actions embodies the wrong abstractions. I'm not 100% sure what the right ones would be but it feels all wrong. 2. It has the feel of an unfinished/undocumented project. Someone's project to get a promotion perhaps that they handed off to interns after securing said promotion. A product that forces me to expend brain cycles reverse engineering it's behavior just makes me want to design it out of my life. I may as well develop my own thing : at least I'll understand that. 3. A build system that can only be run in a proprietary SaaS environment is a terrible idea. I mean, great idea for locking in customers I suppose.
suhlig 3y ago
Concourse has it right: You can ssh into a failed container, and debug in the exact environment where the failure occurred. I don’t understand why people settle for less.
da39a3ee 3y ago
The VSCode extension is quite good and solves some of the problems mentioned concerning detecting errors without needing to go through the commit-push-run cycle.
ptx 3y ago
One serious problem not mentioned in the article is the feature that looks for magic strings in the output and treats them as special commands [0], meaning that you can't safely output anything that's not completely trusted or carefully filtered. At least it's deprecated now, but they apparently decided to postpone the removal because too many people are still using it. Maybe if we all do our part in migrating to the new way [1], they will eventually close this giant security hole. [0] https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ https://github.blog/changelog/2022-10-11-github-actions-depr... [1] https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#environment-files https://docs.github.com/en/actions/using-workflows/workflow-...

412 points by woodruffw 3y ago

Cub3 3y ago

Just give me yaml anchors! I'm so sick of having to duplicate pieces of my actions constantly

I don't see how you love something that makes you jump through these hoops: > In this particular case, it took me 4 separate commits (and 4 failed releases) to debug the various small errors I made: not using ${{ ... }}5 where I needed to, forgetting a needs: relationship, &c

vegardx 3y ago
There's tools like Act[0] that tries to solve this, but this has been an issue with CI systems since they were invented. [0] https://github.com/nektos/act https://github.com/nektos/act
- 5e92cb50239222b 3y ago
  Also reused by gitea for their CI runner. I was quite impressed by that feat, pretty neat. https://gitea.com/gitea/act_runner https://gitea.com/gitea/act_runner
hanniabu 3y ago
Isn't that just programming? That's not much different than saying you forgot a bracket and had to make another commit to make it work. Granted, it would be nice if they had some linter.
- matthewaveryusa 3y ago
  This is a usecase where chatgpt works great! I usually pass this kind of ops DSL in gpt and it will find my mistakes. Github actions and graphanaQL code, oof, they just click for me.
  - chc 3y ago
    I'm not gonna post my employer's proprietary code into ChatGPT.
    - matthewaveryusa 3y ago
      Don't fixate on the details, fixate on the larger point. your specific situation of having code self-hosted and an employer that cares deeply about their proprietary code is a detail beside the point. LLMs shine at fixing up DSLs that you're unfamiliar with. chatgpt to llm is kleenex to facial tissue.
- 5e92cb50239222b 3y ago
  Generally programming these days does not require you to submit build jobs or jump through other hoops to catch trivial mistakes. Although it depends on what you do, embedded developers and game programmers writing for consoles might disagree. It really feels like a throwback to the punch card era. I too don't enjoy writing CI scripts (despite knowing Linux administration and shell scripting quite well), it always takes an inordinate amount of time, but it also saves much more over the long term.
garciasn 3y ago
We use Github Actions and we just don't have any issues with it outside the first time we set it up for each repo. Then we make 100s of commits a week and it does its thing and our work goes live a few seconds later. That's why I love it. Could things be better? Of course; that's how software is--and this should resonate with most folks on this site. But just because some product isn't infallible doesn't mean we can't love it too.
- moralestapia 3y ago
  >Then we make 100s of commits a week and it does its thing and our work goes live a few seconds later. Wow, computers doing what they're supposed to do. Pretty impressive ... The UX on GH actions is crap, other CI/CD solutions give you way more control and tooling. GH really needs to step up their game on this one.
  - garciasn 3y ago
    I am not sure why you’re being so snarky, but I did address that it could be better. The thing for us is that we’re already paying for GitHub and Actions are included. I know, for a fact, GH is/was working with stakeholders using their software to get feedback on what could be better. So they are trying! I’m not here being a fanboy by any means; I’m just saying it does what I need it to do and I’m an active user who is pleased with the service provided. YMMV.
- neuromanser 3y ago
  > we just don't have any issues with it outside the first time we set it up for each repo But that's exactly what a lot of commenters complain about.
layer8 3y ago
Yeah, I wouldn’t want to use any automation that you can’t also easily and quickly test locally.
- Pxtl 3y ago
  Fun fact: Microsoft had a plan to provide that! They canned it. https://github.com/microsoft/azure-pipelines-agent/pull/2687/files#diff-570b52bcb927a5365c22d17a21e8e19e1ba3427c7f3ec16ea64b7b3d14e4ee20 https://github.com/microsoft/azure-pipelines-agent/pull/2687...
- bdcravens 3y ago
  Act can do most of it locally https://github.com/nektos/act https://github.com/nektos/act

deleted 3y ago

[deleted]

tao_at_garden 3y ago

The git commit, push, wait loop is terrible UX. Users deserve portable pipelines that run anywhere, including their local machines. I understand Act [1] goes some way to solving this headache but it's by and large not a true representation. There are many pipelines you can't run locally, because they're production, for example, but there's no reason why we can't capture these workflows to run them locally at less-critical stages of development. Garden offers portable pipelines and then adds caching across your entire web of dependencies. Some of our customers see 80% or higher reductions in run times plus devs get that immediate feedback on what tests are failing or passing without pushing to git first using our Garden Workflows. We're OSS. [2] [1] https://github.com/nektos/act https://github.com/nektos/act [2] https://docs.garden.io https://docs.garden.io

baggachipz 3y ago
Act's incompleteness has had me barking up the wrong tree many times. At this point I've temporarily abandoned using it in favor of the old cycle. I'm hoping it gets better in time!
- frodowtf 3y ago
  I don't get why GitHub doesn't adopt it and make it a standard. Especially the lack of caches is annoying.
andrewstuart2 3y ago
Yeah, it seems like we lost a lot of the "CI shouldn't be a snowflake" when we started creating teams that specialize in "DevOps" and "DevOps tools." Once something becomes a career, I think you've hit the turning point of "this thing is going to become too complicated." I see the same thing with capital-A Agile and all the career scrum masters needing something to do with their time.
candiddevmike 3y ago
If folks just had actions target make or bash scripts instead of turning actions into bash scripts none of this would be an issue. Your CI/CD and your devs should all use the same targets/commands like `make release`.

numbsafari 3y ago

> Because GitHub fails to distinguish between fork and non-fork SHA references, forks can bypass security settings on GitHub Actions that would otherwise restrict actions to only “trusted” sources (such as GitHub themselves or the repository’s own organization). How is this not resolved? Easily bypassing security controls is a major security issue. Yes, you need to convince someone to use your SHA, but social engineering is usually the easy part.

nthState 3y ago

Most of my problems would vanish if there was an official way to run workflows locally, something like: shell: run_workflow name=MyJob in=MyWorkflow.yml params={}

avtar 3y ago
I guess you have to ask yourself what's in MyWorkflow.yml that can't be in a script and run locally?
- mason55 3y ago
  Well, that's why the word "official" is in there. Obviously you could mock up the entire github actions scaffolding locally and have it inject environment variables and support all the actions and everything, but keeping that up to date will be a nightmare if you're doing it on your own. Obviously you could put the whole CI/CD into a bash script and not use any features or functionality provided by github actions but there are plenty of nice things that it does and it would be a shame to not use any of them.
  - avtar 3y ago
    I didn't mention it in my reply but I was referring to the complaint in the article that the author wants to be able to debug locally without having to push changes to GitHub first. The top comment by MoreQARespect and others highlight the benefits of scripting as much as possible so that processes like builds and cutting releases can be run and tested locally. > Obviously you could mock up the entire github actions scaffolding locally Not sure if anyone's advocating for implementing GA entirely. You can at least automate project-specific bits as much as possible using scripts, and then have the CI environment use the same automation. That allows for more local debugging than overusing GA.

nafizh 3y ago

What's the technology behind the blog? It's clean, minimal and beautiful.

junke 3y ago
Jekyll
woodruffw 3y ago
It's a Jekyll site. I originally built it off of a popular theme (back in 2014 or so), but these days the theme is just a custom thing that I've cobbled together. (So, in a sense, there's no real technology "behind" it. It's just Markdown with a little HTML templating, with Jekyll as the SSG.)
- nafizh 3y ago
  Thanks for the reply. Is the code available by any chance?
  - woodruffw 3y ago
    Unfortunately not -- it used to be in a public repo, but I also track drafts and other things now that I don't want to be public until they're ready. I'll see if I can clean it up into a public repository, but no guarantees :-)
  - fooker 3y ago
    Right click -> view page source.

Pxtl 3y ago

I mean GH Actions is basically a re-brand of Microsoft's "Azure Pipelines". As somebody who used all previous incarnations of TFS/VSTS/AzDO build and release pipelines: they are not good at this. This is not a team with a record of success. That Azure Pipelines is moderately usable only happened because they failed literally every other approach they tried. There was a project to allow you to run the pipelines locally so you could do the edit-run-debug loop on your own private environment without committing. It was, of course, canned. https://github.com/microsoft/azure-pipelines-agent/pull/2687/files#diff-570b52bcb927a5365c22d17a21e8e19e1ba3427c7f3ec16ea64b7b3d14e4ee20 https://github.com/microsoft/azure-pipelines-agent/pull/2687... However, there are tools to improve QOL. For example: https://marketplace.visualstudio.com/items?itemName=ms-azure-devops.azure-pipelines https://marketplace.visualstudio.com/items?itemName=ms-azure... A vscode extension that's syntax-aware. Now, I'll be a bit controversial: if they'd used XML instead of YAML, you could have an xmlns declaration up-top that would give you validation in most decent code editors without user intervention. XML is awful, but it has a lot of useful features that we gave up when we threw the baby out with the bathwater.

woodruffw 3y ago
> I mean GH Actions is basically a re-brand of Microsoft's "Azure Pipelines". As somebody who used all previous incarnations of TFS/VSTS/AzDO build and release pipelines: they are not good at this. This is not a team with a record of success. That Azure Pipelines is moderately usable only happened because they failed literally every other approach they tried. I was under the impression (which might be wrong!) that GHA was an independent project within GitHub that was well underway before the acquisition. Are you saying that GHA was rebuilt on top of AzP, that it's just a relabeling of AzP, or something else? (I have no particular dog in it being one way or the other, but I'm curious about the history here.)
- Pxtl 3y ago
  They share a lot of code. My understanding is that it was an MS project first, but I might have that backwards. > GitHub Workflows execute on runners. The runner code is essentially a fork of the Azure Pipelines code, so it's very similar. It's also cross-platform and you can also use hosted or self-hosted runners. https://learn.microsoft.com/en-us/dotnet/architecture/devops-for-aspnet-developers/actions-vs-pipelines https://learn.microsoft.com/en-us/dotnet/architecture/devops...

riv991 3y ago

The feature I'd love is history. I'd like to know if my builds are getting slower over time. I'd be able to detect flaky tests automatically. It seems basic, but I know third-party solutions exist for this. It's out of the box in Circle CI and Buildkite and feels like it should be here.

atomicfiredoll 3y ago
I was with a team that moved from Jenkins to Actions. Jenkins has a lot of issues, but if you knew what you were doing, at least it was easy to see the history and at a glance know the project pipelines were green/healthy. It was super useful for seeing where flake was in tests or observing changes to coverage over time. After the team migrating to actions swept though, we ran without this basic stuff for years because nobody had the time to figure out what 3rd party tool to use, money to pay for it, or capacity to re-implement the functionality. It made dealing with test failures or flake that crept in awful.

simse 3y ago

A linter can catch some obvious errors: https://github.com/rhysd/actionlint https://github.com/rhysd/actionlint. But yes, I agree, it's not a fun debugging experience.

woodruffw 3y ago
Yep, actionlint is great! I've used it successfully both to lint my own workflows, and to lint third-party workflows for (basic) security issues. Unfortunately, it can't lint actions themselves, only workflows that call actions[1]. This is a substantial deficiency, especially for users (like me) who write and maintain a decent number of actions. [1]: https://github.com/rhysd/actionlint/issues/46 https://github.com/rhysd/actionlint/issues/46
bastardoperator 3y ago
act would also be helpful here in terms of debugging and development of Actions workflows. https://github.com/nektos/act https://github.com/nektos/act

theideaofcoffee 3y ago

As someone who has recently sunk a considerable amount of time in modernizing some pipelines, workflows, what have you, I feel this deep in my bones. The commit-test-fail-recommit test loop is painful and something I’ve experienced all too well. Trying to work around product limitations brought about by whoever concocted the system on GitHub’s side to save customers from themselves and declared it good is painful. Trying to debug simple issues is painful. I’m glad I’m not alone.

MoreQARespect 3y ago

There are two types of github actions workflows you can build. 1) Program with github actions. Google "how can I send an email with github actions?" and then plug in some marketplace tool to do it. Your workflows grow to 500-1000 lines and start having all sorts of nonsense like conditionals and the YAML becomes disgusting and hard to understand. Github actions becomes a nightmare and you've invited vendor lock in. 2) Configure with github actions. Always ask yourself "can I push this YAML complexity into a script?" and do it if you can. Send an email? Yes, that can go in a script. Your workflow ends up being about 50-60 lines as a result and very rarely needs to be changed once you've set up. Github actions is suddenly fine and you rarely have to do that stupid push-debug-commit loop because you can debug the script locally. Every time I join a new team I tell them that 1 is the way to madness and 2 is the sensible approach and they always tepidly agree with me and yet about half of the time they still do 1. The thing is, the lack of debugging tools provided by Microsoft is also really not much of a problem if you do 2, vendor lock in is lower if you do 2, debugging is easier if you do 2 but still nobody does 2.

woodruffw 3y ago
This is a great perspective, and one I agree with -- many of the woes associated with GitHub Actions can be eliminated by treating it just as a task substrate, and not trying to program in YAML. At the same time, I've found that it often isn't sufficient to push everything into a proper programming language: I do sometimes (even frequently) need to use vendor-specific functionality in GHA, mark dependencies between jobs, invoke REST APIs that are already well abstracted as actions, etc. Re-implementing those things in a programming language of my choice is possible, but doesn't break the vendor dependency and is (IME) still brittle. Essentially: the vendor lock-in value proposition for GHA is very, very strong. Convincing people that they should take option (2) means making a stronger value proposition, which is pretty hard!
- MoreQARespect 3y ago
  No, you're right it's not necessarily a good idea to be anal about this rule. E.g. If an action is simple to use and already built I use it - I won't necessarily try to reimplement e.g. upload artifacts step in code. Another thing I noticed is that if you do 1 sophisticated features like build caching and parallelization often becomes completely impractical whereas if you default to 2 you can probably do it with only a moderate amount of commit-push-debug.

syndicatedjelly 3y ago

gitlab-runner allows you to run scripts locally for GitLab, but even that is lacking (any env secrets stored in GitLab? Script is going to fail). I really, really wish there was a way to clone the GitHub Actions or GitLab runner environment, spin up a runner locally and test. That would shave off 95% of the wait time

rurp 3y ago

It's hard for me to articulate exactly why but I really dislike the information layout in GH Actions. Compared to other CI/CD tools, it's harder to debug problems or get a sense of the state of a pipeline. Part of the problem is the number of clicks it takes to look at various step logs. Some of the controls are unintuitive, but maybe that's just me.

whalesalad 3y ago
This could be said for a lot of GitHub. The fact that you need to open a disclosure menu to edit a PR is wild.
- mock-possum 3y ago
  Their comment system just doesn’t feel right compared to Bitbucket’s either - I think it’s the distinction between comments that are part of a review, versus comments that are ‘just’ comments

atomicnature 3y ago

This is a real problem. I think the problem stems from the unasserted assumption that declarative YAML is not really coding/debugging. The root problem is the assumption that infrastructure specifications are second-class citizens when it comes to managing a software ecosystem. Due to this, one rarely sees any sort of strong tooling support for creating, updating, debugging and extending various infrastructure activities. The unpleasant truth is that we live in the dark primitive days of infrastructure management.

cwales95 3y ago

This is my normal CI/CD workflow -- even for GitLab. Debugging why CI/CD doesn't work is one of the worst headaches you can imagine.

joshstrange 3y ago

I couldn't agree more with the pain of debugging a GH Actions run. The /only/ tool you have is the ability to re-run with debug on. That's it. I have so many "trash" commits trying to fix or debug a pipeline and so much of it's just throwing stuff at the wall to see if it sticks. Very basic things, like having reusable logic, is needlessly complex or poorly documented. Once I figured out how to do it it was fairly easy but GitHub's docs were terrible for this. They made it seem like I had to publish an action to get any reusability or put it in a different repo. Turns out you can create new yaml files with reusable logic but make sure you put them in the root of the workflows folder or they won't work, go figure. It's just incredibly painful to work on GH Actions but once you have them working they are such a joy. I really wish there was some kind of local runner or way to test your actions before committing and pushing.

TeeMassive 3y ago
I've tried running the GitHub runner image (or maybe an imitation) and it was really painful to setup and to get some things working. I just let it go after 2 days. And it's not just Github. The others big CI platform are not really better in terms of workflow and integration. Now I just script everything to the maximum.
giobox 3y ago
> I have so many "trash" commits trying to fix or debug a pipeline and so much of it's just throwing stuff at the wall to see if it sticks. One tool is to use draft PRs for this - you can run changes to your action YAML from the draft PR. When you are happy just squash the commits as you see fit on a "real" PR to merge the changes in without the mess. I've found draft PRs for debugging/developing GH action logic to be pretty reasonable.
- lambda_garden 3y ago
  Since some action depend on the branch / tag you are on this is not always possible.
  - 3y ago

he0001 3y ago

We recently converted from Jenkins to GitHub Actions (complete rewrite) build pipeline. Jenkins had its issues and warts, but holy Beelzebub what a monster GitHub Actions is. It pure downright evil and creates all sort of headaches. Opaque, zero clues on what broke. Glacial slow even if you throw the biggest instance at it. It’s all enterprise and 1000s of repositories big company. I’m spending more time finding bugs in the build pipeline than the time I spend doing other things. And it’s all GitHub actions fault. I hate it with a passion.

sitzkrieg 3y ago

one thing to be aware of, if you use windows, and submodules in a repo, ci will fail to clone over 50% of the time (with internal retries) with early eof errors it's apparently a fixed problem in upstream openssl beta that isnt bundled yet, but i think a custom runner would be better too (but not an option at my org) still, some days it gets in a rut and quickly fails for 10+ runs all billed at full rate. the entire eng group is severely annoyed by it because we cross test all oses on each commit so its becoming an expensive problem. basically 3x the billing on windows ci

ZeWaka 3y ago

re: `pull_request_target` and fork PRs - You can get around the !Even More Fun! limitations this has (GITHUB_TOKEN being read-only) by having a workflow called by another workflow. For example, I wrote a set of workflows[1] to automatically apply a label after 2 'review approvals'. First, a 'dummy' workflow triggering off PRs (in that context) that uploads an artifact: the PR number Second, the 'real' workflow that runs in the context of the actual repository, set to be `on: workflow_run: workflows: - Final Review Labeler` - this pulls in the artifact, runs a GraphQL query, and applies the label if applicable. [1]: https://github.com/goonstation/goonstation/blob/master/.github/workflows/final_review_labeler.yml https://github.com/goonstation/goonstation/blob/master/.gith...

giobox 3y ago

My personal wish is for the ability to attach HTML reports the the action runs without having to use the current actions/upload-artifact etc. Particularly for test builds, I very often just want to quickly view the output HTML report. The current approaches I am familiar with are the aforementioned upload-artifact, or using GH pages, but GH pages is not great when you have multiple different reporting output for the same repo, or you wanna quickly view historical reporting rather than latest. I'd love a simple "attach-report" action that just put a link to an HTML report on the job summary, and clicking on it renders the html. Other automated CI/CD style systems have much richer support out of the box for dealing with HTML report capturing and viewing.

dmart 3y ago
Granted, it's not HTML, but you can append Markdown output directly to $GITHUB_STEP_SUMMARY without dealing with artifact uploads etc.
- giobox 3y ago
  Thanks, I'm aware of this option too, but if you say have an HTML report with the results of 200 tests, no one has time to write some logic to parse the results to a markdown summary frankly. Just give me the link! Similar too if the reporting captures video/screenshots of a given bug - the original report UI is far easier to deal with. Many test frameworks already natively put out an HTML report UI etc, it's just harder to get to than it should be with GH actions today.
  - pests 3y ago
    > an HTML report with the results of 200 tests, no one has time to write some logic to parse the results to a markdown summary frankly. Not sure how GHA treats it, but markdown is a subset of HTML so by default all HTML pages are valid Markdown documents. No conversion required.
    - Aeolun 3y ago
      They just don’t show half of the stuff inside correctly.

andix 3y ago

I'm really not sure if we are using CI correctly. Sometime i think all those CI Templates should be replaced by just one executable that does everything, like a modern alternative to Makefiles (and there are a lot of build tools). So the CI pipeline would only call the build tool, like "./build containers push-to-registry release:1.0.0 run-tests" Those scripts can be tested and debugged everywhere. Also migrating to a different CI platform would be really easy.

WirelessGigabit 3y ago
While this is totally possible you lose a lot of the things that make GitHub Actions nice. Nice logs, annotations, individual steps that went wrong. Being able to see the status at a glance. Seeing 10 steps where 1 fails and 13 pass is nice.
- andix 3y ago
  All those things the custom build tool would need to handle. It needs more or less the same functionality as GitHub actions and the marketplace. It would also need to integrate with different CI platforms to some extent for reporting and user interactions.
pixl97 3y ago
How does that integrate in with every else's tooling? At least in enterprise there are a ton of things like "SCA, SBOM, Compliance report, etc" that tie in with plugins and such. Also, why would a large commercial CI want to have their environment too open?
- andix 3y ago
  They don’t want to have their environments too open, that’s why we are at this point. Those enterprise tools i know (like sonarqube for example) are mostly cli tools in their core. Can be integrated everywhere.
gravlaks 3y ago
https://dagger.io/ https://dagger.io/ may be an alternative. You can run it in GitHub actions somehow as well.

gchamonlive 3y ago

I am thinking about porting my ci/cd pipelines from Jenkins to GitHub actions to leverage autoscaling self-hosted runners, which I don't think there is an exact equivalent for Jenkins nodes, at least not one that is already implemented where I work. The pipelines though, are all configured using ansible. Applying the same principle I did to Jenkins, using it strictly for execution automation, logs, permission and the likes, and leaving the heavy lifting to a more trusty tool, I think I can avoid both major vendor lock-in and idiosyncrasies and nonsenses from the language. It boggles my mind how unrefined the GitHub actions flow is. It doesn't offer a good experience by a long shot.

kaikoenig 3y ago

Been through that git commit; git push; repeat cycle too much as well until i discovered https://github.com/mxschmitt/action-tmate https://github.com/mxschmitt/action-tmate which gives a shell in between steps, which does not help with all problems but sure it's makes it less painful at times.

justinclift 3y ago
Thanks, that looks super useful. :)

ctoth 3y ago

One thing that actually makes this slightly less horrible is the Gh tool. You can use gh run watch to tail your logs in the terminal and at least don't have to click through a million things just to see output.

rane 3y ago
I wrote a command-line tool that streamlines retrieving test results from GitHub Actions even further. Essentially parses jest/tsc/eslint errors in GHA jobs' logs for the current's branch PR. https://github.com/raine/ghtool https://github.com/raine/ghtool
fijiaarone 3y ago
Do you know a way to stream the gh run watch output?

natbennett 3y ago

For validating YAML I love CUE. It takes some work to define the schema — would def be nice if GitHub provided that. I found Nektos act pretty usable for local testing when I was doing a lot of GA work.

eviks 3y ago

Very good points, especially re. the security footguns

chrissoundz 3y ago

Struggling with this today, and numerous other days. It's so bad. Stop trying to build an operating system out of YAML. I'll always use and recommend Gitlab from now on. And what the earth is an "actions" anyway? How on earth is simple bash functions not just as suitable here? Instead you have some weird YAML scripting language. It's so bad. Why. Somebody please tell me. I'm losing my mind. It is a good reflection of the rest of the world though and why the worlds infrastructure is crumbling in many places.

giobox 3y ago
Executing bash statements/scripts/functions is the thing I struggle least with in GH actions personally, it's remarkably easy to execute shell steps. If you really want your entire build to be a shellscript the action executes, you can do just that with very little YAML.
- Aeolun 3y ago
  Except it doesn’t execute quite like in bash, and every single command is it’s own little island. So exporting an environment variable doesn’t work for example.
  - trallnag 3y ago
    What are you talking about? You can put a whole Bash script into a single step.

watermelon0 3y ago

My number one complaint is that they still don't have managed ARM runners. Considering how overpriced runners are (compared to cloud instances), I'm sure they would make a ton of money here. My choices here are either to use a different CI solution (but I quite like Actions, and they are better integrated in the Github UI than alternatives), or to use custom self-hosted autoscaling runners (but there doesn't appear to be a good solution available at the moment, and I definitely don't want to managed infra for CI.)

xjia 3y ago
Check out https://dime.run/ https://dime.run/ they manage workers for you

jusonchan81 3y ago

We moved our CI to this orchestrator called Netflix Conductor. It’s highly customized internally with workers capable of running different kind of tasks etc. But since we did this, CI has been a wonderful experience. Things run smoothly and history can be tracked and easy to build variants by modifying or creating new workflows. We aim to open source this extension soon.

samiam_iam 3y ago

[dead]

artursapek 3y ago

GitLab’s CI is so much better than GH it’s not even funny

kitallis 3y ago

One of my biggest gripes with Actions is that it doesn't allow you to trigger workflows for a specific SHA. Only the branch HEAD. This is pretty much possible in all other CIs.

tra3 3y ago
You can pass a sha into your workflow and then check it out.

pxeger1 3y ago

I would like to give a strong recommendation for https://pre-commit.ci https://pre-commit.ci (no relation, just a happy user). The idea is that you write hooks (or use pre-existing ones) that check (and fix, where possible) your code before you commit, and then once you've pushed, the CI will check again, including automatically committing fixes if necessary. Anyway, it works brilliantly - unbelievably fast thanks to good automatic caching, and using the exact same setup on your local development machine to in CI negates a lot of these debugging problems. It's free for open source. It only does checks, not things like automatic releases, but it does those them well.

globular-toast 3y ago
This combined with tox is great for Python projects in particular. Tox automates creating virtualenvs for the right Python versions you want to test with then running your tests in them. It can also run static checks by issuing `skip_install = True` because you want to test the source code itself. You just need to run this in a Python container that has tox installed as a globally available tool and all versions of Python available in it (like https://github.com/georgek/docker-python-multiversion https://github.com/georgek/docker-python-multiversion <-- not maintained but easy to update). Here's some boilerplate to do all that: [tox] envlist = py{310,311} [testenv] passenv = PIP_CACHE_DIR deps = coverage # deps only for tox extras = testing # testing extras include pytest commands = pytest ... [testenv:check] passenv = PIP_CACHE_DIR PRE_COMMIT_HOME skip_install = True deps = pre-commit commands = pre-commit run --all-files --show-diff-on-failure

kevmo314 3y ago

I wish individual actions could be specified as Dockerfiles instead of yaml files. That would address a lot of issues that I've run into around build environments being slightly different than locally. Yes, I can wrap it with a simple docker action. It would be cool if that were automatic.

Nezteb 3y ago

This is why I want projects like Earthly to succeed: https://github.com/earthly/earthly https://github.com/earthly/earthly I want to be able to run my all of CI workflows on my local machine. I agree with other commenters that most of what CI does should be abstracted out into scripts or other non-CI tools. Unfortunately it's not easy to do that for large pre-existing CI setups, especially if a different team is the one maintaining your CI workflows.

alexvs 3y ago
Isn't Earthly out of the picture now? https://earthly.dev/blog/shutting-down-earthly-ci/ https://earthly.dev/blog/shutting-down-earthly-ci/
- vladaionescu 3y ago
  Nope. Only Earthly CI is.
- adamgordonbell 3y ago
  As @vladaionescu mentioned, Earthly is alive and well. We stopped the CI product exactly so we could focus on use cases like making Earthly in GitHub Actions better.

vladaionescu 3y ago

Founder of Earthly here - besides the build debugging difficulty, I would add that modern CI/CD repeats a lot of steps: downloading, installing and configuring dependencies, making things much slower than they should be. We built Earthly [1] to tackle these two problems specifically. We're open-source (10k stars). [1]: https://earthly.dev https://earthly.dev

jurassic 3y ago

Most of my frustrations with GHA arise when doing something useful conflicts with someone’s idea of security. For example, branch protection rules intended to stop devs from yoloing commits blocking me from pushing a version bump commit during a release workflow.

Aeolun 3y ago
The lack of exceptions for actions is bizarre. And then they build repository rules or whatever, and made the same mistake again! You can at least exclude github apps now, but then you need to run all your repo actions with an app installtion key :/

ris 3y ago

The single thing that I most detest about GitHub Actions is how they, by design, completely miss the point of containers. Having "actions" that are just "install language foo" is barely better than just publishing shell scripts or even the days of Travis. As a result, few GitHub workflows benefit from the immutability or reproducibility guarantees that can be provided by containers, and most workflows I interact with spend more than half their time running ridiculous installer scripts.

brabel 3y ago
Are you unaware that GH Actions can run jobs in containers?? https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container https://docs.github.com/en/actions/using-jobs/running-jobs-i...

ulizzle 3y ago

More official or blessed packages for sure. I agree with that.

kathlam 3y ago

It's clear we've all wrestled with GitHub actions and Jenkins in some shape or form. Debugging them can feel like being in an escape room, without any clues. I work at Trunk.io and we have a tool called CI Debugger. It helps you understand your workflows. Instead of getting lost in logfiles and commits, you can pinpoint where things are bogging down or going haywire. It can give you a clearer picture of what's happening under the hood. Like having a heat map for the CI/CD processes.

lijok 3y ago

People really need to stop writing scripts in yaml. Not to be dismissive but the issues outlined in this article, aside from that security footgun, are non-issues as soon as you start using github actions only for what it should be used - to invoke your scripts in parallel in response to an event. We've banned all but actions/checkout in our org and have a very healthy dev experience as a result. People are naturally guided towards writing scripts that they can run locally instead of for github actions specifically. Some real issues with github actions: 1. MacOS runners have ridiculously inconsistent IO performance causing 200x slowdown in some cases 2. Getting charged to the nearest minute is asinine and punishes highly parallelised (fast) workloads 3. GitHub OIDC endpoints constantly timing out

fijiaarone 3y ago
Yaml is the worst possible programming language, beating out even ant’s xml abomination. Unfortunately most of the work of GitHub actions is configuring the runners and setup data, which requires you to use lots of yaml to invoke unvetted public repositories (actions) written in JavaScript to do what is easier done as shell scripts, and then use shell scripts to workaround the limitations of actions. State is impossible, data is even more impossible, and caching is completely broken. And if you use your own runners, GitHub actions don’t even clean up their own garbage. I suspect it’s the same on public runners, but just statistically less likely to affect you, but a potential security vulnerability. I’ve been working 2 things: 1. A front end that crafts GHA workflow yamls from a sane configuration with common tasks baked in. 2. An event and reporting system so you can see what is happening during a workflow — not after — without having to scroll through GitHub’s hideous log output in html or download and unzip the full log.

k8svet 3y ago

I mean, it's embarrassing how bad it is. - (unrelated) build failures just randomly notifies the latest maintainer who happened to merge something? (Imagine you finding this out when your newly added maintainer pings you on Matrix and tells you 1: about this behavior, and 2: that your update/builds have been failing for a week without you knowing?!?!) - The cache action is horribly, trivially observably broken with seemingly no maintainer? - Can't tail the build log of a step if their UI poops or your tab unloaded after it started? - The complete lack of abstraction that might actually make workflows portable across worker nodes? pfft. - the default image is a travesty. I thought it was obnoxious how bloated it was and then I started digging in and realizing "Oh, some Microsoftie that didn't know Linux was put in charge of this". (saying this as a former FTE that knows). And there's no effort to allow a more slimmed down image for folks that, you know, use Nix? Or even just Docker? I'm in the process of migrating off GitHub and it's mostly because Actions shocked me to my senses. Too bad MS can't figure out how to retain virtually any Linux talent, and not just their cuffed-acqui-hires or Windows-devs-cosplaying. Even the well compensated ones head for the door. And I'll just say, I don't program in YAML because YAML is a disgrace wrought upon us by Go+Yaml enthusiasts that don't know any better fueled by senseless VC money shoveled at an overall ecosystem incognizant of modern, actually useful technology. edit: removing some of the blatantly identifying former-FTE trauma. Knowing what I know I should sell all my MSFT, but the market thinks differently.

rijoja 3y ago
Wasn't it obvious that something along these lines would happen when Microsoft took over Github?

becurious 3y ago

You’d think they could at least train GitHub Copilot to test run the workflow and point out immediate issues. We have to add an action to kill duplicate runs on triggers. The caching is marginal, at least on Windows runners because it takes forever to expand the tar ball. Not even sure from their docs / issues on the cache action if they did finally move it to use GNU tar. Having some way to get an interactive shell on fail would be a big step up for debugging issues. Otherwise we are back to print debugging in the actions or uploading artifacts at each step so we can inspect them.

bingemaker 3y ago

The current state of GA is much better than Jenkins. I don't like yaml, but this is still better than mucking around Jenkins configs. Fin

plonk 3y ago

The best approach is to develop completely local build scripts and add a few options to plug GitHub Actions' exclusive features into them. Make a build script that can take a cache folder's path as argument. Make it take a test report output path as another argument. etc. So your GHA workflow is: set up secrets (AWS and co.), set up Python, download cache to path X if possible, run the build in X and send reports to Y, publish reports in Y. And put as much work as possible inside the build script. We have a whole Python project for that. This is truly painless after the first development effort.

solatic 3y ago

GitHub Actions is a horrible CI/CD system. You cannot run steps in parallel on the same VM; container-based workloads are a second-class citizen. The first problem means that setting up local credentials and other environment dependencies cannot be parallelized (I'm looking at you, google-github-actions/setup-gcloud, with your 1m+ runtime... grrr), the second makes it quite difficult to put a Dockerfile in a repository to represent setup of the CI environment, and have both (a) the CI rebuild the container images when the image would change, pausing workflows depending on that image until the image is rebuilt, (b) not attempting to rebuild the image when its contents did not change, and immediately running workflows inside that image, including all dependencies already installed. No, in GitHub Actions, you will attempt to repopulate from cache on every single run. Of course, sometimes the cache isn't found, particularly because there's a 5 GB cache size limit (which cannot be enlarged, not even for payment) which cycles out FIFO. So if you go over the 5 GB cache, you might as well not have one. I deeply miss Concourse. Too bad Pivotal dismantled the team and it barely gets even maintenance work. Parallel tasks. Custom pipeline triggers. SSH into the CI environment to debug. Run one-off tasks in CI without needing to set up a pipeline. Transfer directory contents from one job to another without needing to create artifacts (which cost money to store if you're not careful about how long they should stick around for). GitHub Actions is a bastardized toy CI/CD system that only got popular because GitHub make it as simple as uploading a file to .github/workflows in any repository - no additional signup, no in-house ops required, everything you could want is just there. So let's be very clear about what GitHub Actions is good and what it's bad at - it's good at getting people to sign up, but it's not nearly powerful enough to be the "best" system once you start to outgrow the early features.

lawnchair 3y ago
solatic -- I have an existing solution that accounts for a lot of these issues you bring up. Would it be possible to pick your brain? Can you share your email or shoot me an email? lawnchair@lawnchair.net.
bryandollery 3y ago
Have you seen Tekton? (https://tekton.dev/ https://tekton.dev/)
3y ago

bob1029 3y ago

My experience with GitHub Actions is mostly managed by way of certain Azure Portal interactions these days. When I create a new Function app, it gives me the opportunity to enable GitHub integration. The experience for this is flawless, IMO. You provide your GH credentials, select the org/repo/branch, and then it will create the workflow file for you and push it automatically. It will also update the secrets in your repository settings to match what Azure expects on its end for deployment. By the time you get to look at your GitHub repo, the action is already running and will 100% complete successfully if you followed a standard/default project structure. The automatically-generated workflow files aren't perfect, but they're so close that it becomes trivial to tweak for additional build args or project arrangements. Just getting the secrets & related boilerplate configured makes the difference between me doing it right now vs maybe never. The consequence of always having proper CI/CD from day zero, even for the most trivial projects, seems profound to me. There exist some really happy paths now wrt GH actions, but you gotta be willing to get pretty hammered on the Microsoft koolaid to explore them.

varsketiz 3y ago

It's not really context switching. Author is debugging github actions, thats the context in which a few window changes seem minor. Context switching would be to get dragged to meeting, or getting asked by a colleague to explain how to set up github actions for their project.

remram 3y ago

Can I trigger a workflow from a different workflow yet? That seems very basic, and last time I looked I had to create a token that gives read/write access to every repo in every organization to do this. And then it didn't work because the docs for the trigger API is apparently wrong.

Aeolun 3y ago
As long as they’re in the same repository, yeah.
- remram 3y ago
  Different repo same org

bysja 3y ago

Github Actions has been a pleasant introduction to CI/CD for me. I used to build locally and then git push and/or rsync everything directly to a VPS (Digitalocean Droplets in my case). But for collaboration, this workflow breaks down. Then, using Github as the "origin" repository becomes compelling, and Github Actions fit very nicely with that workflow.

elischleifer 3y ago

Another tool that tackles GitHub Action debugging head on: https://www.ci-debugger.io/ https://www.ci-debugger.io/ Wrap your GH Actions in a debugger breakpoint and connect into the live broken GH Action to inspect the machine, re-run commands etc.. At it's heart - realtime debugging for GitHub actions.

ivanjermakov 3y ago

I feel this pain. So much that now, when I need to set up new GitHub actions pipeline, I always create a new branch to push there until it's working. And then squash merge as a single commit.

Aeolun 3y ago

My biggest pain point is not being able to do proper ternary statements in workflows. And passing arrays to called workflows. My ‘runs-on’ statements need more than one label! You can fake it by passing a stringified JSON object, but that’s bizarre.

lenkite 3y ago

I wish Github Actions were Lua instead of YAML. One could use a restricted environment populated with predefined functions, a restricted set of modules and all the advantages of syntax checking and language help.

lpapez 3y ago
Congratulations, you just reinvented Jenkins but using Lua instead of Groovy.
- lenkite 3y ago
  Not a bad thing. Also Lua is both smaller, saner and more performant than Groovy. (with LuaJIT)

deleted 3y ago

[deleted]

icecap12 3y ago

> And so, the question: why are there so few of them? Here is just a smattering of the official actions that don’t exist[...] This part of the article half triggered me and half made me laugh. I was recently on a GitHub webinar to listen to the pitch for CoPilot, but of course they also talked about Actions and GHAS. Anyway, during the overview of Actions, the presenters made a comment about how Actions was "backed by the community." It felt to me like a glaring admission that the product isn't mature. But in true Microsoft fashion, they charge you like it is.

krzema12 3y ago

I created https://github.com/typesafegithub/github-workflows-kt https://github.com/typesafegithub/github-workflows-kt and https://github.com/typesafegithub/github-actions-typing https://github.com/typesafegithub/github-actions-typing to address some of the mentioned issues. Some problems remain unsolved because it's just a Kotlin DSL that generates the YAML, but it does catch some issues early. Related blog post: https://dev.to/jmfayard/github-actions-a-new-hope-in-yaml-wasteland-1i9c https://dev.to/jmfayard/github-actions-a-new-hope-in-yaml-wa...

globular-toast 3y ago

A tight feedback loop is everything. I will not use tools that require long waits between tests. I want to edit, test, wait a matter of seconds to see the results. Seeing people push hundreds of commits to their CI pipeline trying to fix it by trial and error makes me cringe. I do not understand how some people accept such a workflow without thinking it could be better. When I was planning the CI set up at my job I made sure it could all be run locally and hammered this point home to developers. CI isn't magic, it's just some other computer running your shit. We use Gitlab CI which runs your script in a container. It's trivial to run the container locally with docker to test exactly what will happen in CI (although you do have to make a fresh clone of your repo for the mount, I thought about scripting this but in practice it was never needed that much). GitHub Actions seems much harder to test locally. There seems to be some projects to do it but they are too heavy for me to want to install. For this reason alone I don't like Actions.

dboreham 3y ago

I think most developers don't need to or don't have the time to dive deeply into their build systems, but having had to do so as part of a project to migrate to Gitea (which turned into a larger project to enhance/fix its act-based github actions clone), my take is: 1. Github actions embodies the wrong abstractions. I'm not 100% sure what the right ones would be but it feels all wrong. 2. It has the feel of an unfinished/undocumented project. Someone's project to get a promotion perhaps that they handed off to interns after securing said promotion. A product that forces me to expend brain cycles reverse engineering it's behavior just makes me want to design it out of my life. I may as well develop my own thing : at least I'll understand that. 3. A build system that can only be run in a proprietary SaaS environment is a terrible idea. I mean, great idea for locking in customers I suppose.

suhlig 3y ago

Concourse has it right: You can ssh into a failed container, and debug in the exact environment where the failure occurred. I don’t understand why people settle for less.

da39a3ee 3y ago

The VSCode extension is quite good and solves some of the problems mentioned concerning detecting errors without needing to go through the commit-push-run cycle.

ptx 3y ago

One serious problem not mentioned in the article is the feature that looks for magic strings in the output and treats them as special commands [0], meaning that you can't safely output anything that's not completely trusted or carefully filtered. At least it's deprecated now, but they apparently decided to postpone the removal because too many people are still using it. Maybe if we all do our part in migrating to the new way [1], they will eventually close this giant security hole. [0] https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ https://github.blog/changelog/2022-10-11-github-actions-depr... [1] https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#environment-files https://docs.github.com/en/actions/using-workflows/workflow-...