27 ms·
The Pain That Is GitHub Actions
- cresolutejw 2y ago[flagged]
- simonw 2y agoDefinitely not a skill issue if you read the details in their post - https://www.feldera.com/blog/the-pain-that-is-github-actions https://www.feldera.com/blog/the-pain-that-is-github-actions - they're clearly very experienced with using GitHub Actions and have run into legitimate challenges due to the complexity of what they're using it for.
- cresolutejw 2y agoI did read the post. The documentation of the product does explain how to use it, so it is unfair to blame the product in my opinion.
- simonw 2y agoGitHub Actions may cover this stuff in the documentation but it's still difficult to use. That means the product could be designed better (and the documentation could be easier to follow.)
- cresolutejw 2y agoWhy downvote me because there is discourse? Could we not usually say most software could be documented better. I do not think GitHub Actions ranks near the bottom in terms of documentation and user experience overall. I do understand your point though.
- SequoiaHope 2y agoIt would be more helpful for you to provide the solution rather than just boast about your intelligence.
- cresolutejw 2y agoThe solution was found by the author, and is covered in GitHub actions documentation.
- lxe 2y agoShould have been zero permissions by default. The current model is a mess of global settings, workflow permissions, and job tokens that nobody understands.
- jiggawatts 2y agoAzure DevOps is nearly identical, but with slightly different zoo of issues that are less well documented in public sources. It also has the problem of not having a local dev runner for actions. The "inner loop" is atrociously slow and involves spamming your colleagues with "build failed" about a thousand times, whether you like it or not. IMHO, a future DevOps runner system must be an open-source, local-first. Anything else is madness. Right now we're in the "mainframe era" of DevOps, where we edit text files in baroque formats with virtually no tooling assistance, "submit" that to a proprietary batch system on a remote server that puts it into a queue... then come back after our coffee to read through the log printout. I should buy a dot matrix printer to really immerse myself into the paradigm.
- quickslowdown 2y agohttps://xkcd.com/303/ https://xkcd.com/303/
- _cenw 2y agoThey are so identical, there's code in the GitHub runner to search and replace "Azure DevOps" with "GitHub Actions" in log output on the fly. The entire code is a wonderful mess. We found that when we early-adopted ephemeral runners, that the control flow is full of races and the status code you get at the end is indicative of exactly nothing. So even if the backend is just having a hickup picking up a job with an obscure Azure error code, you better just throw that entire VM away, because you can't know if that runner will ever recover or has already done things to break the next run.
- rochacon 2y agoThe current version of GHA is "Azure DevOps v3". IIRC, it came after Microsoft purchased GitHub and I think it was part of a plan to discontinue/kill Azure DevOps altogether in favor of GitHub. I don't think they have feature parity yet, specially on the issues and permissioning parts. Although, I never saw a public announcement of this discontinuation, ADO is kind of abandoned AFAICT and even their landing page hints to use GitHub Enterprise instead [1]. [1] https://azure.microsoft.com/en-us/products/devops https://azure.microsoft.com/en-us/products/devops
- rsanheim 2y agotldr but: don't use GitHub Actions. Its a mess, the availability is often atrocious, and the UI around it is _still_ as clunky as when they first rolled it out many years ago. There are better solutions out there.
- _cenw 2y agoGitHub Actions is like Microsoft Teams. Nobody who knows better wants to use it, but it's slightly better than what most did before (email/jenkins/nothing) and came with the thing you're already using. At least your boss thinks it's better. And it's such a good deal!
- ibejoeb 2y agoDoes anyone think GHA is better than Jenkins? I was doing things more than 20 years ago in Hudson that GHA can't do now.
- everfrustrated 2y ago>Does anyone think GHA is better than Jenkins? A 1000% yes, because it means the default experice most devs have of CI is using ephemeral runners which is a massive win for security and build rot. Every company I've worked at with stateful runners was a security incident begging to happen, not to mention builds that would do different things depending on what runner host you got placed on (devs manually installing different versions of things on hosts, etc)
- williamDafoe 2y agoDoes anyone use GHA - full stop? Their stuff seems "me too" in most cases, they are definitely a follower, like microsoft Bing search ... google AI ...
- ashishb 2y ago> There are better solutions out there. And what are those?
- goosejuice 2y agoAfter using Gitlab CI for years and setting up some pretty complex scenarios, when I switched over to GitHub I found the UX to be pretty rough. Seems very opaque and I find the documentation to be at best hard to navigate. Maybe it was just the pain of switching but that was my initial impression.
- sepositus 2y agoAlso an Earthly casualty here. Now having to look at Dagger.
- netvarun 2y agoDagger (https://dagger.io https://dagger.io) recently seems to have reinvented/rebranded itself as some llm agent platform.
- 12_throw_away 2y agoThere is at a tiny glimmer of life on the earthly front - yesterday, they merged their first changes in 6 months: [1] https://github.com/earthly/earthly/commit/6d7f6786ad9fa4392f7cedb9d6aec11b726b9136 https://github.com/earthly/earthly/commit/6d7f6786ad9fa4392f... [2] https://github.com/earthly/earthly/commit/89d31fc014a8980a50d80e19d2caebbef5442d53 https://github.com/earthly/earthly/commit/89d31fc014a8980a50... I am really really hoping that someone (not me, I've already tried and failed) could slim it down into a single-purpose, self-contained, community maintainable tool ...
- peterldowns 2y agoThese are all real pains, author definitely has done a lot of work in Github Actions; respect. I'm sure these notes will save a lot of people a lot of frustration in the future, since Github Actions isn't going away --- it's too damn convenient. I wonder why they chose to move back to Github Actions rather than evaluate something like Buildkite? At least they didn't choose Cloud Build.
- ZeWaka 2y agoYep, I've run into every one of these issues in my time working with the CI. It's still leagues beyond the old Azure DevOps pipelines or god forbid, Jenkins. I think incremental progress in the CI front is chugging along nicely, and I really haven't seen any breathtaking improvements from other solutions I've tried, like CircleCI.
- fourteenminutes 2y agoUsed to use GH actions quite a bit. At my current company we set up RWX Mint (rwx.com/mint) and haven't looked back. (disclaimer: used to work at rwx but no longer affiliated)
- silisili 2y agoI worked at companies using Gitlab for a decade, and got familiar with runners. Recently switched to a company using Github, and assumed I'd be blown away by their offering because of their size. Well, I was, but not in the way I'd hoped. They're absolutely awful in comparison, and I'm beyond confused how it got to that state. If I were running a company and had to choose between the two, I'd pick Gitlab every time just because of Github actions.
- yoyohello13 2y agoGlad I’m not the only one. GitLab runners just make sense to me. A container you run scripts in. I have some GitHub actions for some side projects and it just seems so much more confusing to setup for some reason.
- usr1106 2y agoSo Github was really the perfect acquisation for the Microsoft portfolio. Applications with a big market share that are technically inferior to the competition. // Luckily still a gitlab user, but recently forced to Microsoft Teams and office.
- rhubarbtree 2y agoTechnical superiority is so irrelevant compared to distribution. Welcome to capitalism, where the market rewards marketing.
- out-of-ideas 2y ago> recently forced to Microsoft Teams my condolences to you and your team for that switch; it's my 2nd used-and-disliked thing (right next to atlassian) - oh well but one cool feature i found with ms teams that zoom did not have (some years ago - no clue now) is turning off incoming video so you dont have to be constantly distracted in meetings edit: oh yeah, re github actions and the user that said: > Glad I’m not the only one me too, me too; gh actions seem frustrating (from a user hardly using gh actions, and more gitlab things - even though gitlab seems pretty wonky at times, too)
- hn_throwaway_99 2y ago> A few days ago, someone compromised a popular GitHub Action. The response? "Just pin your dependencies to a hash." Except as comments also pointed out, almost no one does. I used GitHub actions when building a fin services app, so I absolutely used the hash to specify Action dependencies. I agree that this should be the default, or even the required, way to pull in Action dependencies, but saying "almost no one does" is a pretty lame excuse when talking about your own risk. What other people do has no bearing on your options here. Pin to hashes when pulling in Actions - it's much, much safer
- smpretzer 2y agoI have been using renovate, which automatically pins, and updates, hashes. So I can stay lazy, and only review the new hash when a renovate PR gets opened: https://docs.renovatebot.com/modules/manager/github-actions/#digest-pinning-and-updating https://docs.renovatebot.com/modules/manager/github-actions/...
- dijit 2y agoI think the HN community at large had a bit of a learning experience a couple of days ago. "Defaults matter" is a common phrase, but equally true is: "the pattern everyone recommends including example documentation matters". It is fair to criticise the usage of GH Actions, just like it's fair to criticise common usage patterns of MySQL that eat your data - even if smarter individuals (who learn from deep understanding, or from being burned) can effectively make correct decisions, since the population of users are so affected and have to learn the hard way or be educated.
- deleted 2y ago[deleted]
- hn_throwaway_99 2y agoI wholeheartedly agree, and perhaps it was just how I was interpreting the author's statement in the article. If it's saying that the "default" way of using GitHub Actions is dangerous and leads to subtle security footguns, I completely agree. But if you know the proper way to use and secure Actions, saying "everyone else does it a bad way" is irrelevant to your security posture.
- larusso 2y agoInteresting. I‘m also moving our CI to GitHub actions after years of using Jenkins with custom pipelines written in groovy etc. I checked out GitHub actions every now and then to feel if a move finally makes sense. I started with simple builds then tested adding our Jenkins macOS agents as self hosted runners. Just yesterday I wrote two actions to build and test a new .net project. I was able to run the whole thing with „act“ locally before running it on GitHub proper. I also played around and created a custom action in typescript (kicked off from the available predefined templates) to see how much work maintaining that means. All in all I‘m super happy and see no bigger issues. But here are some things that might be a reason: I split CI in build system logic which should and need to run locally and just stuff that GitHub needs to execute. At best that means describing what runs in parallel, and making specific connections. Any complicated logic needs to be abstracted away behind a a setup that is itself testable. I handle it the same for our build system components. We use gradle a lot and of a few custom plugins which encapsulate specific build / automations. It’s like dividing your problem into many smaller pieces which are tested and developed in isolation. Next to json I also used travisCI and appveyor for projects. And they all had the same (commit and pray) setup that ai hate. I wish if „act“ was a tool directly maintained by the GitHub folks though. https://github.com/nektos/act https://github.com/nektos/act
- GauntletWizard 2y agoWhenever I get mad at GitHub Actions, I refer to it by it's true name: VisualSourceSafe Actions. Because that's what it is, and it shows. If you check out their Action Runner's source code[1], you'll find the VSS prefix all over, showing it's lineage. [1] https://github.com/actions/runner/blob/6654f6b3ded8463331fb0c96a73b0435775c7c4e/src/Sdk/Common/Common/VssException.cs#L14 https://github.com/actions/runner/blob/6654f6b3ded8463331fb0...
- rurban 2y agoOh, C#. Nice!
- hinkley 2y agoI know they've fixed VSS ages ago, but for many years it was buggy af and would catastrophically lose data on automerges that it confidently made and was wrong. I had a coworker who called it Visual Sorta-Safe which is just about the best parody name I've ever heard in my entire career.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- jamesu 2y agoOne thing I found useful was writing a runner for giteas actions CI which is similar to GHA. When you dig down and ask "what is ACTUALLY happening to run this job" then a lot of things such as the docker entrypoint not being modifiable make perfect sense.
- yoyohello13 2y agoMy team uses GitLab and most other teams are on Azure dev ops. They keep trying to get us to switch telling us how amazing pipelines are. Glad to know we are not missing anything.
- Xiol32 2y agoHaving recently been involved in a Gitlab to ADO migration, keep fighting the fight. It is such a step backwards.
- LilBytes 2y agoADO is painful, GitHub has its warts but it's got much more community support from services like depot.dev and others.
- chanux 2y agoJust adding more literature here, https://news.ycombinator.com/item?id=18983586 https://news.ycombinator.com/item?id=18983586 https://rewiring.bearblog.dev/blog/?q=azure https://rewiring.bearblog.dev/blog/?q=azure PS: I am not the author of any of these posts.
- robertlagrant 2y agoAzure DevOps seems extremely basic (and flaky!) compared to GitLab. My impression is from a couple of years ago though; perhaps it's amazing now.
- itissid 2y agoI can relate to this pain. Isn't gitlab CI better at this especially the documentation and simplicity of it?
- lemagedurage 2y agoI wonder if the complexity of fixing trivial code mistakes in CI is worth it compared to catching them in a pre-commit hook.
- yeswecatan 2y agoUnfortunately people will use --no-verify to bypass hooks.
- normie3000 2y agoI don't understand commit hooks - they're like binding a macro to the MS Word save button to make it conditional.
- chuckadams 2y ago> like binding a macro to the MS Word save button to make it conditional You have no idea how much I'd love that feature. Inasmuch as "save" is still a thing anyway. I don't miss explicit saves in IDEA, I see commit as the "real" save operation now, and I don't mind being able to hook that in an IDE-independent way. I think the UX of git hooks has been sub-par for sure, but tools like the confusingly named pre-commit are helping there.
- sgarland 2y agoBecause if you haven’t auto-formatted, lined, etc. then it’s a very easy way to do that so you don’t waste time watching CI fail for something stupid like trailing comma placement. I don’t want to think about formatting, I just want everything to be consistent. A pre commit hook can run those tools for me, and if any changes occurred, it can add them to the commit.
- PhilipRoman 2y agoYou can put hooks on the server side of git. It can do pretty much anything that CI/CD can.
- 2y ago
- mcqueenjordan 2y agoUsually if you’re using it, it’s because you’re forced to. In my experience, the best strategy is to minimize your use of it — call out to binaries or shell scripts and minimize your dependence on any of the GHA world. Makes it easier to test locally too.
- sepositus 2y agoThis is what I do. I've written 90% of the logic into a Go binary and GitHub Actions just calls out to it at certain steps. It basically just leaves GHA doing the only thing it's decent at...providing a local UI for pipelines. The best part is you get unit tests, can dogfood the tool in its own pipeline, and can run stuff locally (by just having the CLI nearby).
- noisy_boy 2y agoMakes migrations easier too; better to let gitHub or gitlab etc to just be the platform to host source code and trigger events which you decide how to deal with. Your CI itself should be another source controlled repo that provides the features for the application code's thin CI layer to invoke and use. That allows you to be able to run your CI locally in a pretty realistic manner too. I have done something similar with Jenkins and groovy CI library used by Jenkins pipeline. But it wasn't super simple since a lot of it assumed Jenkins. I wonder if there is a more cleaner open source option that doesn't assume any underlying platform.
- raffraffraff 2y ago> Usually if you’re using it, it’s because you’re forced to. Like teams.
- kelseydh 2y agoWe recently had a developer -- while trying to debug container builds for a version upgrade for a PR on their local branch -- accidentally trigger a deployment of their local branch's docker container to production (!) while messing around with Github action workflow files in their pull request (not main). Outside of locking down edit access to the .github workflow yml files I'm not sure how vulnerabilities like this can be prevented.
- LilBytes 2y agoYeah it's a difficult problem, templates help, then put the templates into another repo that is managed by a specific person and imported into others. Not sure how that work in a monorepo, I expect those controls wouldn't. The problem is it's still possible to work around those controls unless you create some YAML monstrosity that stops people from making the mistake in the first place.
- carderne 2y agoYour prod deployment should require access to some secrets that are only available to workflows running against main.
- kelseydh 2y agoI'm interested in learning more about this. How would we go about adding a secret only available to runners on the main branch? Is there a configuration option on Github to create a secret only available to runners on main? Presumably anything configured via a .github workflow wouldn't assure safety, as those files can be edited to trigger unexpected actions like deploys on working branches. Our Github Action workflow yml file had a check to only deploy for changes to the main branch. The deploy got triggered because that check got removed from the workflow file in a commit on a working branch.
- everfrustrated 2y agoI haven't used it but the GitHub Environments feature allows setting Secrets by Environment. Costs extra $ tho. But for actually good security CI and CD should be different tools.
- silverwind 2y agoGHA is full of such obure behaviours. One I recently discovered is that one action can not trigger another: If one action pushes a tag to the repo, `on:tag` does not trigger. The workaround apparently is to make the first action push the tag using a custom SSH key, which magically has the ability to trigger `on:tag`.
- OptionOfT 2y agohttps://docs.github.com/en/actions/security-for-github-actions/security-guides/automatic-token-authentication#using-the-github_token-in-a-workflow https://docs.github.com/en/actions/security-for-github-actio... > When you use the repository's GITHUB_TOKEN to perform tasks, events triggered by the GITHUB_TOKEN, with the exception of `workflow_dispatch` and `repository_dispatch`, will not create a new workflow run. It has bitten me in the rear before too. I use this pattern a lot when I publish a new version, which tags a piece of code and then marks assets as part of that version (for provenance reasons I cannot rebuild code).
- geewee 2y agoWe're also struggling with this, as we'd love to e.g. just run a formatter and commit the changed code in CI rather than just fail the code.
- mook 2y agoThat actually seemed reasonable when I hit it, because you can easily accidentally have an action triggered on commit that makes a new commit, ending up in an infinite loop. The workaround is to use a token tied to you instead of GitHub Actions, so you get charged (or run out of quota).
- joshstrange 2y ago> The workaround is to use a token tied to you instead of GitHub Actions, so you get charged (or run out of quota). You get charged no matter what, a personal access token doesn’t change anything. If they are concerned about infinite loops then put a limit on how many workflows can be triggered but another workflow. Each time a workflow chains off another pass along some meta data of “runsDeep” and stop when that hits X, which can be configured. No, requiring a PAT to kick off a workflow from a workflow is gross and makes zero sense. I don’t want every tag associated with my user, I want it to be generic, the repo itself should be attributed. The only way to solve this is to create (and pay for) another GH user that you create PAT tokens under. A bunch of overhead, cost, and complexity for no good reason.
- jicea 2y agoGenuine question: what's the GitLab equivalent of GitHub Actions? I'm using GitHub Actions to easily reuse some predefined job setup (like installing a certain Python version on Linux, macOS, Windows runners). For these tyoe of tasks, I find GitHub actions very useful and convenient. If you want to reuse predefined jobs, written by someone else, with GitLab CI/CD, what can I use?
- imp0cat 2y agoCI/CD Components: https://docs.gitlab.com/ci/components/ https://docs.gitlab.com/ci/components/ (an evolution of CI/CD templates).
- wwarek 2y agoFor reusing pieces of existing pipelines I think `include` would be appropriate, especially `remote` variant: https://docs.gitlab.com/ci/yaml/#includeremote https://docs.gitlab.com/ci/yaml/#includeremote
- mqus 2y agoinclude:component is usually what you want now, you can version your components (Semver), add a nice readme and it is somewhat integrated in the gitlab UI. Not sure about the other include: ones, but you can also define inputs for component and use them at arbitrary places like template variables. Since the integration is done statically, it means gitlab can provide you a view of the pipeline script _after_ all components were included, but without actually running it. We are using this and it is so nice to set up. I have a lot of gripes with other gitlab features (e.g. environments, esp. protected ones and their package registry) but this is one they nailed so far.
- never_inline 2y agoDoesn't include:component still require all your shell script to be written inside YAML? or is there a way to move the logic to a, for instance, .sh file and call it from YAML?
- ThomasRooney 2y ago> A few days ago, someone compromised a popular GitHub Action. The response? "Just pin your dependencies to a hash." Except as comments also pointed out, almost no one does. I'm surprised nobody has mentioned dependabot yet. It automates this, keeping action dependencies pinned by hash automatically whilst also bringing in stable upgrades.
- presentation 2y agoWasn’t part of the problem though that renovate was automatically upgrading people to the compromised hash? Or is that just the fault of people configuring it to be too aggressive with upgrades?
- Arbortheus 2y agoNo, someone just impersonated renovate bot and the repo author got tricked
- huijzer 2y agoWell but that’s the problem. You cannot fully automate this. You have to manually check the diff of each dependency and only accept the dependabot PR if the changes are safe. The only automation that I know of is cargo vet. Although it doesn’t work for GitHub Actions, the idea sounds useful. Basically, vet allows people who trust each other to vet updates. So one person verifies the diff and then approves the changes. Next, everyone who trusts this person can update the dependency automatically since it has been “vetted”. [1]: https://github.com/mozilla/cargo-vet https://github.com/mozilla/cargo-vet
- deleted 2y ago[deleted]
- hinkley 2y agoDependabot is only approximately as good as your tests. If you have holes in your testing that you can drive a bus through, you're gonna have a bad time. We also, to your point, need more labels than @latest. Most of the time I want to wait a few days before taking latest, and if there have been more updates since that version, I probably don't want to touch anything for a little bit. Common reason for 2 releases in 2 days: version 1 has a terrible bug in it that version 2 tries to fix. But we won't be certain about that one either until it's been a few more days with no patch for the patch for the patch.
- deleted 2y ago[deleted]
- wordofx 2y agoGHA feels like a discontinued product that people use so they can’t switch it off.
- webworker 2y agoYeah, between the two, I strongly prefer BitBucket Pipelines. Feels much cleaner.
- kfarr 2y agoI was updating an old action last night to update gh pages and it’s from peaceiris. And it’s not bad, it did the job. But it feels kinda weird.
- jalaziz 2y agoGitHub Actions started off great as they were quickly iterating, but it very much seems that GitHub has taken its eye of the ball and the improvements have all but halted. It's really upsetting how little attention Actions is getting these days (<https://github.com/orgs/community/discussions/categories/actions?discussions_q=is%3Aopen+category%3AActions+sort%3Atop https://github.com/orgs/community/discussions/categories/act...> tells the story -- the most popular issues have gone completely unanswered). Sad to see Earthly halting development and Dagger jumping on the AI train :(. Hopefully we'll get a proper alternative. On a related note, if you're considering https://www.blacksmith.sh/ https://www.blacksmith.sh/, you really should consider https://depot.dev/ https://depot.dev/. We evaluated both but went with Depot because the team is insanely smart and they've solved some pretty neat challenges. One of the cooler features is that their caching works with the default actions/cache action. There's absolutely no need to switch out popular third party actions in favor of patched ones.
- pinkgolem 2y agoI might have missed the news, but I did not find anything in regards to earthly stopping development What happened there?
- jalaziz 2y agoI missed it too, but then found this: https://github.com/earthly/earthly/issues/4313 https://github.com/earthly/earthly/issues/4313
- 12_throw_away 2y agoSigh, this is awful. Earthly is/was not perfect, but is basically the most capable build tool I've ever used. Fingers crossed there's enough enthusiasm in the community to fork it (I'd be organizing it myself if I had any experience with Go at all)
- pimeys 2y agoWe switched to Depot last week. Our Rust builds went down from 20+ minutes to 4-8 minutes. The easy setup and their docker builds with fast caching are really good.
- voidr 2y agoI don't get the obsession with YAML and making things declarative that really should not be declarative. I'm so much happier on projects where I can use the non-declarative Jenkins pipelines instead of GH Actions or BB pipelines. These YAML pipelines are bad enough on their own, but throw in a department that is gatekeeping them and use runners as powerful as my Raspberry Pi and you have a situation where a lot of developers just give up and run things locally instead of the CI.
- hinkley 2y agoI haven't tried to step through Scons, so that may be a system that looks like how I want it to look but fails entirely to deliver on its promises for all I know. I think there's a place for making a builder that looks imperative, but can work out a tree of actions and run them. Gulp is a little bit this way, but again I haven't tried to breakpoint through it either. If the next evolution in DevEx is not caring about what your code looks like in a stepping debugger, then the one after it will be. Making libraries that present a tight demo app for the Readme.md file and then are impossible to do anything tricky with or god forbid debug just needs to fucking stop. Yesterday. And declarative systems are almost always the worst.
- neuroelectron 2y agoYes I have to agree Jenkins is the best solution but it's not the hot new thing, AI powered etc. It just works and that's not how you grow your org.
- kylegalbraith 2y agoThis was an interesting read and highlighted some of the author's top-of-mind pain points and rough edges. However, in my experience, this is definitely not an exhaustive list, and there are actually many, many, many more. Things like 10 GB cache limits in GitHub, concurrency limits based on runner type, the expensive price tag for larger GitHub runners, and that's before you even get to the security ones. Having been building Depot[0] for the past 2.5 years, I can say there are so many foot guns in GitHub Actions that you don't realize until you start seeing how folks are bending YAML workflows to their will. We've been quite surprised by the `container` job. Namely, folks want to try to use it to create a reproducible CI sandbox for their build to happen in. But it's surprisingly difficult to work with. Permissions are wonky, Docker layer caching is slow and limited, and paths don't quite work as you thought they did. With Depot, we've been focusing on making GitHub Actions exponentially faster and removing as many of these rough edges as possible. We started by making Docker image builds exponentially faster, but we have now brought that architecture and performance to our own GHA runners [1]. Building up and optimizing the compute and processes around the runner to make jobs extremely fast, like making caching 2-10x faster without having to replace or use any special cache actions of ours. Our Docker image builders are right next door on dedicated compute with fast caching, making the `container` job a lot better because we can build the image quickly, and then you can use that image right from our registry in your build job. All in all, GHA is wildly popular. But, the sentiment around even it's biggest fans is that it could be a lot better. [0] https://depot.dev/ https://depot.dev/ [1] https://depot.dev/products/github-actions https://depot.dev/products/github-actions
- SkiFire13 2y agoBy what measure is this "exponentially faster"? Surely GH doesn't take an exponential time in the number of steps of the workflow...
- Aeolun 2y agoDepot is fantastic. Can heavily recommend it. It’s like magic when your builds suddenly take 1m instead of 5+ just by switching the runner.
- tasuki 2y ago
- stephencoxza 2y agoNot sure if I'm the odd one out here. I thoroughly enjoy making the best of whatever the company wants to use. The flavour of CI/CD can be a debate similar to programming languages
- darkwater 2y agoI think it's even worse. Just like ticketing systems, people love to sunk on CI/CD because it's out of the scope of their primary focus (writing software) so having to deal with it is a PITA. The only CI/CD system most people like are the ones that are almost invisible.
- suryao 2y agoThere definitely are a ton of issues with GitHub actions. To add to the OP's list: - Self-hosting on your aws/gcp/azure account can get a little tricky. `actions-runner-controller` is nice but runs your workflows within a docker container in k8s, which leads to complex handling for isolation, cost controls because of NAT etc. - Multi-arch container builds require emulation and can be extremely slow by default. - The cache limits are absurd. - The macos runners are slow and overpriced (arguably, most of their runners are). Over the last year, we spent a good amount of time solving many of these issues with WarpBuild[1]. Having unlimited cache sizes, remote multi-arch docker builders with automatic caching, and ability to self-host runners in your aws/gcp/azure account are valuable to minimize cost and optimize performance. [1] https://warpbuild.com https://warpbuild.com
- lars512 2y agoAt Our World In Data we ended up using Buildkite to run custom CI jobs, integrated with GitHub, but on cheap, massive Hetzner machines. I can really recommend the experience!
- tobinfekkes 2y agoThis is the joy of HN, for me, at least. I'm genuinely fascinated to read that both GitHub Actions and DevOps are (apparently) so universally hated. I've been using both for many years, with barely a hiccup, and I actually really enjoy and value what they do. It would never have dawned on me, outside this thread, to think that so many people dislike it. Nice to see a different perspective! Are the Actions a little cumbersome to set up and test? Sure. Is it a little annoying to have to make somewhat-useless commits just to re-trigger an Action to see if it works? Absolutely. But once it works, I just set it and forget it. I've barely touched my workflows in ~4 years, outside of the Node version updates. Otherwise, I'm very pleased with both. My needs must just be simple enough to not run into these more complicated issues, I guess?
- IshKebab 2y agoSounds like you have the same pain points as everyone else; you're just more willing to ignore them. I am with the author - we can do better than the status quo!
- raffraffraff 2y agoIt probably depends on your org size and how specialised you are. Right now I dislike GitHub Actions and think that Gitlab CI is way better, but I also don't give it to much thought because it's a once in a blue moon task for me to mess with them. But I would absolutely hate to be a "100% DevOps guy" for a huge organisation that wants me to specialise in this stuff all the time. I think that by the end of week 1 I'd go mad.
- Marsymars 2y agoI don't mind it per se; to me the problem is then that some devs don't bother with basic debugging steps of CI failures - if anything works locally and fails in CI, their first step is to message me - so instead of being "100% DevOps" I spend a pile of time debugging other devs' local environments.
- 2y ago
- msy 2y agoOn a related note - how are people tracking the absolute thicket of permissions quirks that is Github's various secrets, tokens & repo permissions?
- deng 2y agoAlready see people saying GitLab is better: yes it is, but it also sucks in different ways. After years of dealing with this (first Jenkins, then GitLab, then GitHub), my takeaway is: * Write as much CI logic as possible in your own code. Does not really matter what you use (shell scripts, make, just, doit, mage, whatever) as long as it is proper, maintainable code. * Invest time that your pipelines can run locally on a developer machine as well (as much as possible at least), otherwise testing/debugging pipelines becomes a nightmare. * Avoid YAML as much as possible, period. * Don't bind yourself to some fancy new VC-financed thing that will solve CI once and for all but needs to get monetized eventually (see: earthly, dagger, etc.) * Always use your own runners, on-premise if possible
- Aeolun 2y agoThis is where I was going to say something about dagger, but it seems it turned into AI crud. Let me at least recommend depot.dev for having absurdly fast runners.
- oulipo 2y agocan you give more feedback about dagger? what is good/not good about it? I was going to start looking into it
- toastal 2y agoFor starts looking at their website, it looks like all collaboration is locked behind proprietary platforms… Discord, Twitter, LinkedIn, Microsoft GitHub.
- Aeolun 2y agoI liked their setup before, though I never got around to actually using it, but the tagline on the website has changed to “AI powered workflow orchestration”, which is quite different from the original “Write pipeline once, run everywhere”
- oulipo 2y ago
- grav 2y agoAt [previous company], we initially required branches to be up to date with main. Since it was a relatively big mono repo, it slowed down productivity quite a bit. Eventually we tried dropping that requirement and instead relied on testing main before deploying to production. It sped us up again, and main never broke because of bad merges while I was there.
- youdont 2y agoWhen GitHub actions are stopped GitHub just goes straight for the nuclear SIGKILL. No, asking nice first with a SIGTERM... This means that for anything that needs to gracefully cancel, like for example terraform, it's screwed. Want to cancel a run? Maybe you've got a plan being generated for every commit on a branch, but you push an update. Should be ok for GitHub to stop the previous run and run the action for the updated code, right? WRONG! That's a quick way to a broken state.
- ruuda 2y agoTo make sure that you can test CI locally, the best way I've found so far is to make sure the checks can run with Nix, and then keep the CI config itself as simple as possible and just call Nix. As for reducing boilerplate in the CI configs, GitHub Actions is a programming language with support for functions! It's just that function calls can only appear in very limited places in the program (only inside `steps`), and to define a function, you have to create a Git repository. The function call syntax is also a bit unusual, it's written with the `uses` keyword. So there is a lot of boilerplate that you can't remove this way, though there are several other yaml eDSLs hidden in GitHub Actions that address some points of it. E.g. you can create loops with `matrix`, but again, not general-purpose loops, they can only appear in a very specific syntactic location. To really duplicate stuff, rather than copy-pasting blocks of yaml, without using a mix of these special yaml eDSLs, in the past I've used Nix and Python to generate json. Now I'm using RCL for this (https://rcl-lang.org https://rcl-lang.org). All of them are general-purpose yaml deduplicators, where you can put loops or function calls anywhere you want.
- duijf 2y ago> It's just that function calls can only appear in very limited places in the program (only inside `steps`), and to define a function, you have to create a Git repository. FYI there is also `on: workflow_call` which you can use to define reusable jobs. You don't have to create a new repository for these https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#onworkflow_call https://docs.github.com/en/actions/writing-workflows/workflo...
- xlii 2y agoThere is one thing that I haven’t seen mentioned: worst possible feedback loop. I’ve noticed this phenomenon few times already, and I think there’s nothing worse than having a 30-60s feedback loop. The one that keeps you glued to the screen but otherwise is completely nonproductive. I tried for many moons to replicate GHA environment on local and it’s impossible in my context. So every change is like „push, wait for GH to pickup, act on some stupid typo or inconsistency, rinse, repeat”. It’s like a slot machine „just one more time and it will run”, eating away focus and time. It took me 25 minutes to get 5s build process. Naive build with GHA? 3 minutes, because dependencies et al. Ok, let’s add caching. 10 hours fly by. The cost of failure and focus drop is enormous.
- figmert 2y agoHighly recommend nektos/act, and if it's something complex enough, you can Ssh into the server to investigate. There are many action that facilitate this.
- kelseydh 2y agoFeel this pain so much. If you are debugging Github Action container builds, and each takes over ~40 minutes to build.. you can burn through a whole work day only testing six or seven changes. There has to be a better way. How has nobody figured this out?
- elAhmo 2y agoThere is act, that allows you to run actions locally. Although not exactly the same as the real thing, it can save time. https://github.com/nektos/act https://github.com/nektos/act
- mab122 2y agoIn organization setting this is almost useless if you are (or forced to) use some pre-made actions and/or actions that are for your organization only (they cannot be downloaded) also useless if you are forced to use self hosted runner with image that you don't have access to. Not to mention env/secrets and networking...
- quantadev 2y agoGitHub Actions is just a way to build a remote single point of failure into your pipeline. I don't get why people do this. If GitHub goes down, or has problems for whatever reasons it can interfere with your deliverables to customers. I learned early in my career not to trust 3rd parties as any part of any mission critical process. 3rd parties will always fail you, it's just a matter of time.
- kelseydh 2y agoWith widespread dependencies like AWS or Github, if they go down.. you benefit from everybody else also going down. Downtime of that kind means a lot of media coverage and an easier/more understanding conversation with your affected customers. The worst kind of downtime is when you go down but nobody else has.
- quantadev 2y ago[flagged]
- neuroelectron 2y agoIs your argument, literally, "everybody else is doing it?"
- quantadev 2y agoI think it was all about "having excuses" for one's bad decisions, to avoid taking responsibility. lol.
- usrme 2y agoTo get around the horror that is YAML, I wholeheartedly recommend writing GitHub workflows in CUE and generating the required YAML out of them. I'm hopefully never going back to writing YAML myself!
- 1a527dd5 2y agoI just wish the default wasn't bash. GHA with pwsh is a much better experience.
- oulipo 2y agoI wanted to try dagger to solve some of these issues (https://github.com/dagger/dagger https://github.com/dagger/dagger) anyone has feedback on it?
- rgilton 2y agoYep, after spending a few years with gitlab pipelines, my company started migrating over to dagger roughly mid-2024. We moved to dagger to get replicable local pipeline runs, escape the gitlab DSL, and get the enormous benefits of caching. We have explicitly chosen to avoid using the "daggerverse", and with that the cross-language stuff. Reason being that it makes modifying our pipeline slower and harder -- the opposite of the reason we moved to dagger. So we use the Dagger python API to define and run our CI builds. It's great! Like the other comments on this page about dagger, the move to "integrate AI" is highly concerning. I am hopeful that they won't continue down this path, but clearly the AI hype bubble is strong and at least some of the dagger team are inside it. I'm speculating that if the dagger team doesn't drop the AI stuff, then the dagger project will end. A fork will pop-up and we'll move to using that. Not an expert (yet!) in the buildkit API, but it seems like the stuff we're benefiting from with dagger is really just a thin wrapper around buildkit. So potentially not too challenging to create a drop-in replacement if necessary later.
- jonenst 2y agoI'm surprised the author doesn't mention environment secrets, which I think currently are the only way to avoid that anyone with push access to any repo also gets full access to all secrets (by pushing a new workflow file and triggering it). This makes org and repo secrets practically useless for any team where only admins or maintainers should have access to secrets.
- glandium 2y agoI'll take on the occasion to ask the HN crowd: have you noticed that caches, on top of being limited to 10GB, don't seem to expire as advertized, in a LRU manner?
- dilawar 2y agoIf you are annoyed by gitlab-runner deprecating run command that I used to run pipelines locally, there is https://github.com/firecow/gitlab-ci-local https://github.com/firecow/gitlab-ci-local . But it also opened my eyes to benefita of having runner invariant pipelines -- pipelines written in solution agnostic way. Use bash, make, just, doit or whatever. Nothing beats having a single script to bootstrap and run the whole pipeline e.g. `make ci`.
- toastal 2y agoI can’t believe Forgejo ever thought it was a good idea to try & copy this nonsense. Rather than trying to be some FOSS MS GitHub clone, why not pitch that they can do things better—such as not having YAML spaghetti for CI. I hope Actions stays bad tho. We need more folks to get off proprietary code forges for their open source projects—& a better CI + a better review model (PRs are awful) are 2 very low-hanging fruit that would entice folks off of the platform not for the philosophical reasons such as not supporting US corporations or endangering contributor privacy by making them agree to Microsoft’s ToS, but for technical superiority on the platform itself.
- bob1029 2y agoKeeping the tech stack simple helps a lot with the CI/CD space. I still prefer to use custom tools that are part of the project source so I don't get locked in. This is like EC2 vs FaaS for me. I'll take the vanilla abstraction please. Most of the time, I'm just running dotnet build to a zip file and s3 bucket. Then, some code or script picks it up on the other side. Things get much trickier when you're using multiple services, languages, runtimes, database technologies, etc.
- demaga 2y agoWe recently discover that if the last person to change cron __schedule__ of the workflow is removed from the organization, workflow fails with cryptic errors. It turns out, the last person to change cron __schedule__ (not the workflow file in general) is an 'actor' associated with this workflow. Very, very confusing implementation. Error messages are even more confusing - workflow runs are renamed as "{Unknown event}" and the message is "Email is unverified". Link to docs: https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows https://docs.github.com/en/actions/writing-workflows/choosin...
- joshstrange 2y agoThings like this are why I hate having to use PATs in workflows. What if I leave the company? I’ll leave a wake of broken actions in my wake. I do not like that at all, a huge point of CI/CD is automation, reproducibility, and NOT being dependent on specific developers/machines.
- quesera 2y agoI believe this is a good use for a GitHub machine account. IIRC, GitHub recommends this practice in their docs, with a username of "YOUR_USERNAME-machine". The machine user is just an ordinary GitHub user, added as a member of the organization, with all the necessary repo permissions, and a generated access token added to the GH repo Secrets. The organization owner then manages this GH machine account as well as the org, and their own personal (or work) login account.
- anttiharju 2y agoUsing machine/service accounts across an org can (relatively) easily hit rate limits, better way is to use a github app instead for generating the tokens: https://github.com/peter-evans/create-pull-request/blob/main/docs/concepts-guidelines.md#authenticating-with-github-app-generated-tokens https://github.com/peter-evans/create-pull-request/blob/main...
- 2y ago
- rejschaap 2y agoWe've all been there: $ git l * cbe9658 8 weeks ago rejschaap (HEAD -> add-ci-cd) Update deploy.yml * 0d78a6e 8 weeks ago rejschaap Update deploy.yml * e223056 8 weeks ago rejschaap Update deploy.yml * 8e1e5ea 8 weeks ago rejschaap Update deploy.yml * 459b8ea 8 weeks ago rejschaap Update deploy.yml * a104e80 8 weeks ago rejschaap Update deploy.yml * 0e11d40 8 weeks ago rejschaap Update deploy.yml * 727c1d3 8 weeks ago rejschaap Create deploy.yml
- rejschaap 2y agoBetter formatting $ git l * cbe9658 8 weeks ago rejschaap (HEAD -> add-ci-cd) Update deploy.yml * 0d78a6e 8 weeks ago rejschaap Update deploy.yml * e223056 8 weeks ago rejschaap Update deploy.yml * 8e1e5ea 8 weeks ago rejschaap Update deploy.yml * 459b8ea 8 weeks ago rejschaap Update deploy.yml * a104e80 8 weeks ago rejschaap Update deploy.yml * 0e11d40 8 weeks ago rejschaap Update deploy.yml * 727c1d3 8 weeks ago rejschaap Create deploy.yml
- maccard 2y agoThere’s a lot of confident people in this thread saying CI is easy if you “just” make it dumb and keep all the logic in scripts that you farm out to. My experience is this works for simple scripts but immediately falls apart when you start to do things like “don’t run the entire battery of integration tests against a readme change”, or “run two builds in parallel”, or “separate the test step from the build and parallelise it even if the build is serial”. It’s easy to wrap make build and go about your life, but that’s no easier than just using the GitHub action to call go build or mvn build. T he complexity comes in “pull that dependency from this place that is in a private repository on GitHub/AWS because it’s 100x faster than doing it from its source”, and managing the credentials etc for all of that stuff. This is also where the “it differs from running locally” comes into it too, funnily enough.
- ManBeardPc 2y agoCI environments like Gitlab or Github are my nemesis. Another technology that everyone swears is absolute necessary but somehow makes everything more complicated. The provided environments in companies so far are hell 100% the time and managed by inexperienced personnel with zero or little programming experience. * Barely reproducible because things like the settings of the server (environment variables are just one example) are not version controlled. * Security is a joke. * Programming in YAML or any other config format is almost always a mistake. * Separate jobs often run in their own container, losing state like build caches and downloaded dependencies. Need to be brought back by adding remote caches again. * Massive waste of resources because too many jobs install dependencies again and again or run even if not necessary. Getting the running conditions for each step right is a pain. * The above points make everything slow as hell. Spawning jobs takes forever sometimes. * Bonus points if everything is locked down and requires creating tickets. * Costs for infra often keep expanding towards infinity. We already have perfectly fine runners: the machines of the devs. Make your project testable and buildable by everyone locally. Keep it simple and avoid (brittle) dependencies. A build.sh/test.sh/release.sh (or in another programming language once it gets more complicated, see Bun.build, build.zig) and a simple docker-compose.yml that runs your DB, Pub-Sub or whatever. Works pretty well in languages like Go, Rust or TS (Bun). Having results in seconds even if you are offline or the company network/servers have issues is a blessing for development. There are still things like the mentioned heavy integration tests, merges to main and the release cycle where it makes sense to run it in such environments. I'm just not happy how this CI/CD environments work and are used currently.
- mab122 2y agobut then, but then your corporate provided laptop with 8GB RAM and 128GB with locked down Windows Entprise™® may need 1000th of security policy exceptions and won't even fit all dependencies on its disk. Not to mention that it would be building for like 10h. Think of the shareholders! The corporation would have to buy actually usable hardware for it's workers! Think of the cost! /j For real tho, not every project can be build by everyone locally, but at least parts of it should be locally runnable for devs to be able work (at all IMO). What I am noticing is more and more coding is being done on some server somewhere Github Codespaces anyone? Google Colab? etc. What I am also noticing is that this tools like GH-A there is not really a way to test the CI code other than.. commit, push, wait, commit, push, wait... That's just absurd to me. Obviously all CIs have some quirks that sometimes you have _just run it_ and see if it works but this... it's like that for everything! Abusrd I say!
- ahub 2y agoI don't see sourcehut [0] mentionned here. I tested github and gitlab CI, sourcehut is MILES ahead. I'll drop two key features here : - any CI run successful or not, gives you back a ssh URI so you can log into the machine to inspect/tweak/tinker - CI files are NOT in the project's repository. no need to wrangle with your git branches when working on CI anymore [0] : https://man.sr.ht/builds.sr.ht/ https://man.sr.ht/builds.sr.ht/
- terminalbraid 2y ago> CI files are NOT in the project's repository I don't use sourcehut, but interpreting what you wrote I'd argue this is an antifeature and would be a dealbreaker for me. CI typically evolves with the underlying code and decoupling that from the code makes it difficult to go backwards. It loses cohesion.
- enriquto 2y agoyou can put them in the same repository, if that is your thing. If you put the build files in a .builds/ folder at the root of your repository, they will be run upon each commit. Just like in github or gitlab. You are just not forced into this way of life. If you prefer, you can store the build files separately, and run them independently of your commits. Moreover, the build files don't need to be associated to any repository, inside or outside sourcehut.
- terminalbraid 2y agoI see, that is nice. Thank you for the patient explanation.
- tigerlily 2y agoI've found Actions and Codespaces to be different from one another. Actions once recently came with a borked gcc compiler, which failed to build some code that was fine before, and there was no convenient way to debug this. As in no way for me to spin up exactly the same GH Actions environment in Codespaces. Why not align these tools? Then there might be less pain. What a good idea.
- joshstrange 2y ago> Why do I need a custom token? Because without it, the release completes, but doesn't trigger our post-release workflow. This is so frustrating. Having to inject a PAT into the workflow just so it will kick off another workflow is not only annoying but it just feels wrong. Also not lots of operations are tied to my user which I don't like. > It doesn't help that you can't really try any of this locally (I know of [act](https://github.com/nektos/act https://github.com/nektos/act) but it only supports a small subset of the things you're trying to do in CI). This is the biggest issue with GH Actions (and most CIs), testing your flows locally is hard if not impossible All that said I think I prefer GH Actions over everything else I've used (Jenkins and GitLab), it just still has major shortcomings. I highly recommend you use custom runners. The speed increase and cost savings are significant. I use WarpBuild [0] and have been very happy with them. I always look at alternatives when they are mentioned but I don't think I've found another service that provides macOS runners. [0] https://www.warpbuild.com https://www.warpbuild.com
- kylegalbraith 2y agoJust flagging that Depot now has macOS and Windows runners [0] as well if you're looking for even faster builds. I also recognize that constantly reevaluating runners isn't on everyone's priority list. [0] https://depot.dev/docs/github-actions/runner-types https://depot.dev/docs/github-actions/runner-types
- knazarov 2y agoWe use a combination of AWS autoscaling and Nix to make our CI pipeline bearable. For autoscaling we use terraform-aws-github-runner which will bring up ephemeral AWS machines if there are CI jobs queued on GitHub. Machines are then destroyed after 15 minutes of inactivity so they are always fresh and clean. For defining build pipelines we use Nix. It is used both for building various components (C++, Go, JS, etc) as well as for running tests. This helps to make sure that any developer on the team can do exactly the same thing that the CI is doing. It also utilizes caching on an S3 bucket so components that don't change between PRs don't get rebuilt and re-tested. It was a bit of a pain to set up (and occasionally a pain to maintain), but overall it's worth it.
- jFriedensreich 2y agoI am completely switching my mental model of what a ci/cd system should be at the moment: i use docker compose for absolutely everything possible. unit tests? runs as part of the container build. linear build dependent steps? multi stage docker biuld. DAG of build steps? dependencies in docker compose. This way every developer has the same system that ci/cd uses locally. debugging the dev setup is the same as debugging the ci/cd. The purpose of the actual ci/cd is reduced to handling/configuring triggers, handling env vars/secrets and triggering the docker compose command with the proper selected docker context This also reduces the lock in by orders of magnitude.
- punkbit 2y agoSounds like a good option. I’ll try something similar next time.
- klysm 2y agoWrite a script that works anywhere and execute it via your ci tooling as a thin wrapper
- DanielHB 2y agoIf you have complex multi-step actions I recommend tools like nx (for frontend projects) or Bazel. It massively simplifies caching parts of your CI workflows and works locally too. We have a very complicated build process in my current project, but our CI pipelines are actually just a couple of hundred of lines of GHA yaml. Most of which are boilerplate or doing stuff like posting PR comments. The actual logic is in NX configuration.
- deleted 2y ago[deleted]
- cantagi 2y agoI have a problem with the Github Actions documentation. There is a lot of it, but it feels as though it was written from a "product" perspective, to explain how to use the product. None of it usefully explains how GHA works from the ground up, in a way that would help me solve problems I encounter.
- adminm 2y agoI try to use as little of GHA specific things as possible. Use it as a runner but don't lock yourself into the platform. I want to be able to develop and run the CI outside GHA thank you very much.
- packetlost 2y agoMost "CI" platforms suck in some way. I attribute it to a mix of misaligned incentives (less efficient pipelines, more premium-rate CPU cycles to resell, lockin, etc.) and the fact that it's actually just a hard problem. See: https://packetlost.dev/Why%20Does%20CI%20Suck https://packetlost.dev/Why%20Does%20CI%20Suck
- TheRealPomax 2y agoIs there a decent setup that one can run on their own server(s) instead? Because I'd much rather have a dedicated server sitting in a closet connected to fiber whose only job it is to be a CI runner.
- neuroelectron 2y agoYes
- acedTrex 2y agoGithub is too busy dumping all their engineering power into more useless copilot features than actually doing anything to improve their platform.
- rodolphoarruda 2y agoSidenote -- What a cool looking website. What kind of tool do you need to create those animations?
- orliesaurus 2y agoI am building Toolhouse.ai and I've had headaches with Github actions but luckily I ask the AI to help out when I am trying to do something. My biggest annoyance is that it's oddly hard to debug things without running it. Even the Github Actions syntax helper on VSCode isn't super helpful. I have recently discovered `act`[1] and I will be investing time to using it because it really * hopefully * makes a difference. [1]https://github.com/nektos/act https://github.com/nektos/act
- gjohnhazel 2y agoThis was actually an extremely valuable article for me. I was unaware of act, the tool to test GH workflows, and your personal flow of using a separate branch to troubleshot the yaml code makes perfect sense.
- solatic 2y ago> Trivial mistakes (formatting, unused deps, lint issues) should be fixed automatically, not cause failures. Do people really consider this best practice? I disagree. I absolutely don't want CI touching my code. I don't want to have to remember to rebase on top of whatever CI may or may not have done to my code. Not all linters are auto-fixable so anyway some of the time I would need to fix it from my laptop. If it's a trivial check it should run as a pre-commit hook anyway. What's next, CI should run an LLM to auto-fix failing test cases? Do people actually prefer CI auto-fixing anything?
- ben_pfaff 2y agoI'm new to CI auto-fixes. My early experience with it is mixed. I find it annoying that it touches my code at all, but it does sometimes allow a PR to get further through the CI system to produce more useful feedback later on. And then a lot of the time I end up force-pushing a branch that is revised in other ways, in which case I fold in whatever the CI auto-fix did, either by squashing it in or by applying it in some other way. (Most of the time, the auto-fix is just running "cargo fmt".)
- stared 2y agoI do such things with pre-commit. Doing it in CI sounds like making things more complicated by resetting to remote branches after pushing commits. And, in the worst case, something that actually brakes code that works locally.
- Marsymars 2y agoI have team members who complain that installing and running pre-commit is too much overhead, so instead I see them pushing commit after broken commit that tie up CI resources to fail on the pre-commit workflow. :(
- michpoch 2y ago> I have team members who complain that installing and running pre-commit is too much overhead Why do they have a say in this? This is up to tech leadership to set standards that need to be followed.
- 0xbadcafebee 2y agoI have used Travis, CircleCI, GitHub Actions, GitLab Pipelines, AWS CodeBuild/CodeDeploy, Bazel, Drone, GoCD, and Jenkins. And I have used GitLab, GitHub, and Bitbucket for hosting VCS files. (I'm the guy who manages this crap for a living, so I have used it all extensively, from startups to enterprises) GitHub Actions is the worst possible CI platform - except for all the others. Every single CI platform has weird limitations, missing features, gotchas, footguns, pain points. Every single one requires workarounds, leaves you tearing your hair out, banging the table trying to figure out how to do something that should be simple. Of all of them I've tried, Drone is the platonic ideal of the best, simplest, most generally useful system. It is limited. But that limitation is usually easy to work around and doesn't impose artificial constrictions. However, you won't find nearly as many canned solutions or plugins as GitHub Marketplace, and the enterprise features are few. GHA is great because of things like Dependabot, and the million canned Marketplace actions, and it's all tightly integrated with GH's features, so you don't have to work hard to get anything advanced or specific to work. Tight integration can save you weeks to months of development time on a CI solution. I've literally seen teams throw out versioning of dependencies entirely because they weren't updating their dependencies, because there's no Dependabot orb for CircleCI. If they had just been on GHA using Dependabot it would have saved them literal years of headaches. Jenkins is, ironically, both the most full-featured, and the absolute worst to configure/maintain. Worst design, worst security, worst everything... except it does have a plugin for everything, and a UI for everything. I hate it with the fire of a million suns. But people won't stop using it, partially because it's so goddamn configurable, and they learned it years ago and won't stop using it. If anyone wants to write a replacement, I'm happy to help (I even wrote a design doc!).
- tech_tuna 2y agoIt's funny, I've used them all too. . . I like GHA overall but it sure has its quirks. Anyone who claims that GHA is garbage and any of the others are amazing is either doing something very basic or is crazy, or lying. At the end of the day, you run shell scripts and commands using a YAML based config language (except for Jenkins). Amazingly, it's hard to build something that does that with the right abstractions and compromises between flexibility and good hygiene.
- 2y ago
- 999900000999 2y agoWhatever happened to picking the right tool for the job ? It looks like they have a very specific and unique build process which they really should handle with something more customizable like Jenkins. Instead they're using something that's really intended for quick and light deployments for intense dev ops setup. I really like GitHub actions, but I'm only doing very simple things. Don't call a fork bad because it's not great when you're eating soup
- ohgr 2y agoThe only people who picked the right tool for the job are the people you don't hear about.
- 999900000999 2y agoThat's a good point, at least once a week someone decides that instead of reading the documentation and understanding the limitations of the technologies or frameworks they want to use... They either just write a long blog post about how they can't screw in nails with a hammer. Or they leave their security rules wide open and about half the comments are like, we need tools which stop us from doing stupid things. No other industry works like this.
- lolinder 2y ago> something more customizable like Jenkins If they had, we'd be reading a different article about how terribly complex and unintuitive Jenkins is. CI is just a very very hard problem and no provider makes it easy.
- timewizard 2y agoWe use GitHub actions. We have a single job step. It has an "if: false" property on it. When triggered the action immediately completes and no runners are engaged. What it really does is fire off a WebHook. Repository custom properties and the name of the action are properties that are included in the workflow_job webook. With this you can do anything you want and you're not at all constrained by YAML or runners.
- dboreham 2y agoThere's a meta-problem here: GitHub Actions is one of those things that when you first encounter it, is presented as: "we've got it all figured out and all goin on here, and anyone who is scratching their head must be dumb". This pattern, in my experience, shows up frequently in the software realm. Then, typically there follows some period where you try to do whatever you need to do by reading docs and copying what you see others do. Frustration and head scratching grows finally culminating in a process of "Ok WTF are the core concepts of this thing, what were they thinking when they designed it, what is it really going???". The article is what you end up finding after that stage has been gone through. The conclusion of course is that whoever invented this stuff really wasn't thinking clearly and certainly didn't have the time to write decent documentation to explain what they were thinking. And now the whole world has to try to deal with their mess. My theory as to how this ends up happening is that the people creating the thing began with some precursor thing as their model. They made the new thing as "old thing, with a few issues fixed". Except they didn't fully understand the concepts in that thing, and we never got to see that thing. You'll see many projects that have this form: bun is "yarn fixed". Yarn is "npm fixed". And so on. None of these projects ever has to fully articulate their concepts.
- danfritz 2y agoNot my experience, I have done fairly complex things like building / releasing a whitelabel ios and android app which is branded in ci per customer. Like othes have suggested, keep the actions simple by having lots of scripts which you can iterate on locally and making the actions dump to just run the scripts
- actinium226 2y agoAs another commenter said, it's good to write as much CI logic outside the yaml file as possible. I take this a step further and approach CI with the mentality that I should be able to run all of my CI jobs locally with a decent interface (i.e. not by running 10 steps in a row), and then I use CI to automate my workflow (or scale it, as the case may be). But it always starts with being able to run a given task locally and then building CI on top of it, not building it in CI in the first place.
- K3UL 2y agoAs someone who's been using it at very large scale, I still miss gitlab but I think they are not that bad. But two major pains I did not see : the atrocious UI and the pricing. Their pricing model goes against any good practice, as it counts a minute for any job even if it runs for 2 seconds. Let's say you run 100jobs in parallel and they all take 30sec. You will pay 100 minutes instead of 50. Now translate this to an enterprise operating at a big scale and I assure you have seen crazy differences between actual time and billable time.
- neycoda 2y ago1st mistake: rebasing in CI. Stop rebasing. This should only happen if absolutely necessary to fix major merge mistakes. Rebasing changes history and I've seen more problems prevented from removing it as a CI strategy. Every CI strategy I've seen relying on rebasing had a better alternative in SDLC. You just need to level up your project management, period.
- AtNightWeCode 2y agoGithub Actions are simple things for tasks. I do btw also not like them. But, there are so many red flags in this post. Clearly this corp does not know how to build, test and release professional software.
- eYrKEC2 2y agoHas anyone used argo on kubernetes to automate their CI pipeline?
- andrea81 2y agoGit is a pain
- justinko 2y agoOnce again, Rails is ahead of the curve on this: https://github.com/rails/rails/pull/54693 https://github.com/rails/rails/pull/54693
- ris 2y agoWorst part of GitHub Actions? Its encouragement of the total misuse of containers as what have essentially become installer scripts - complete with all the flakiness you'd imagine. Go look at your workflows and see how much of the runtime is spent running installers upon installers for various languages, package managers and so on. Containers were not supposed to be like this.
- zzo38computer 2y agoI only use GitHub Actions for auto assigning issues (and I never merge pull-requests directly; I will always handle pull-requests manually). Here is the entire file: on: issues: types: - opened pull_request: types: - opened permissions: contents: read issues: write pull-requests: write jobs: default: runs-on: ubuntu-latest steps: - run: gh issue edit ${{ github.event.issue.number }} --add-assignee ${{ github.repository_owner }} env: GH_TOKEN: ${{ github.token }} GH_REPO: ${{ github.repository }} I set the permissions to only allow writing to issues and pull-requests (so that if gh is modified to do malicious things (or has a security flaw that allows it to do malicious things even if not intended), it cannot affect anything other than issues and pull-requests). As far as I can tell from the documentation, this is correct (although can do things other than add assignees, and it does not seem that it can be set more finely), but if I am wrong then you can tell me that I am wrong. Documentation for GitHub Actions says, "If you specify the access for any of these permissions, all of those that are not specified are set to none." The article says "I do think a better "default" would be to start with no privileges and require the user to add whatever is needed", and it would seem that this is already the case if you explicitly add a "permissions" command into your GitHub Actions file. So, it would seem that the "default permissions" are only used if you do not add the "permissions" command, although maybe that is not what it means and the documentation is confusing; if so, then it should be corrected. Anyways, you can also change the default permission setting to restrictive or permissive (and probably ought to be restrictive by default). Allowing to set finer permissions probably would also help.
- a1o 2y agoI really urge people to try Cirrus CI, it's almost unknown but it has pretty amazing features!
- youssefabdelm 2y agoOff-topic but DALL-E has turned the web into slop-city. What a mess. Everything looks the same, cheap and ugly and stupid.
- sontek 2y agoI'm surprised that they were already using earthly and decided not to continue using it inside github actions. That is my favorite pattern so that what github actions is doing is the same as what I'd do locally. No `act` necessary: - name: Install Earthly if: steps.check_changes.outputs.relevant_changes == 'true' uses: earthly/actions-setup@v1 with: version: v${{ env.EARTHLY }} - name: Run tests and generate coverage summary if: steps.check_changes.outputs.relevant_changes == 'true' run: cd src/webapp && earthly --build-arg GO_VERSION=${{ env.GOLANG }} +coverage-summary
- r3tr0 2y agowe are working on a platform https://yeet.cx https://yeet.cx that lets you audit github actions pretty easily. you just activate some probes and write SQL queries to sift through the information.
- ramesh31 2y agoHalf these problems and complexity could be solved with precommit hooks.
- stasge 2y agoWhile I agree that GitHub Actions could be more "secure and safe" by default most gaps are easy enough to fill. With https://github.com/marketplace/actions/check-actions https://github.com/marketplace/actions/check-actions you can ensure timeouts, permissions and version pinning. With https://registry.terraform.io/modules/giner/repo/github https://registry.terraform.io/modules/giner/repo/github you can manage all repos together with workflows.
- Sleaker 2y agoI just don't understand why people have gone down this ci in the cloud train. I've been using some form of build scripts locally and then just extended that to run on jenkins as a standard flow for the longest time and reproducibility is key. It always will be. If you just have a generic script or tool that builds your project and people can just pick it up and read it then it's going to be way easier to adopt and debug. Stop trying to use platforms with a dsl, with breaking down pieces in their proprietary way. It makes no sense.
- punkbit 2y agoGitHub actions provide me the best experience I ever encountered when dealing with ops. I pair it with bash scripts where I find important to run outside GitHub facilitating test and maintenance. Although, I still find need to run multiple times or iterate over the actual GitHub action run time, which is a bit slow to iterate but find it best to catch any issues. If the dev feedback loop fixed it save me a lot of precious time. I know there’s a third party to run locally but it’s not the same… Thus, bash scripting is great due to portability.
- ratmeadow 2y agoThis is a fascinating read as someone who is currently working on migrating CI platforms. We deal with hundreds of internal repos where many CI platforms just don't scale well. We are probably going to choose Github Actions because it looks like the simplest, least-BS option. The problems the author mentions would be the least of our concerns. Totally agree with other comments saying to keep as much logic out of CI config as possible. Many CI features are too convoluted for their own good. Keep it really simple. You can use any CI platform and have a shit time if you don't take the right approach.
- neuroelectron 2y agoBuilding in the cloud seems like the worst possible thing you could do.
- jalapenos 2y agoIt seems that whenever something becomes good enough to become popular, you start the clock on people starting to write about how it's the worst thing ever.
- sylware 2y agoOn the pain side: is it "official" that github issues cannot be posted anymore with noscript/basic (x)html browsers?
- stuff4ben 2y agoI still love my Jenkins Pipelines, Groovy shared libraries, and Bash scripts. It may be old, the UI may be a little crusty, but it's well understood, it just works, and I don't think much about the tool anymore.
- ronef 2y agoI've run into similar pain points with GitHub Actions in past roles but I still very much use them/get value. One approach that's helped us at Flox is indeed using Nix and we've now seen customers start leveraging that. Significant drop in “works on my machine” issues and more reliable CI pipelines, etc... It’s not a silver bullet, but it offers a solid technical foundation for tackling these challenges. On the Flox side we are very much on the integrate and improve rather than full replace for scenarios like this one <3 Happy to answer any Nix items on this!
- sam_bristow 2y agoModern CI tends to grow features until it's reproduced the all the features of a build system. You then end up with the actual logic of your build smeared across mutiple layers. Or contorting your CI setup to allow the build system to do caching etc properly.
- TheHackerDev 2y agoIf you want an easy solution for GitHub Actions security, check out Garnet.ai (formerly listen.dev). They were built for GitHub first. And it’s free for single projects - https://dashboard.listen.dev/ https://dashboard.listen.dev/.
- belikebakar 2y agoDoes it allow to integrate directly into the action runner?
- fkjadoon94 2y agoYes, its a one step integration into your workflow file, typically before the steps you want to monitor eg. build, test if you don't want to see everything happening in your runner host. It has worked pretty well with ubuntu-latest and stock Linux runners from GH out of the box.
- fkjadoon94 2y agoThe integration basically wraps *jibril - a single binary linux edr which allows for detection and enforcement in the runner https://jibril.sh https://jibril.sh
- solosito 2y agoDid you consider Pkl instead of Yaml to write your GitHub actions? https://github.com/StefMa/pkl-gha https://github.com/StefMa/pkl-gha It could save you already some time.