10 ms·

> no way of running actions locally My policy is to never let pipeline DSLs contain any actual logic outside orchestration for the task, relying solely on one-

> no way of running actions locally My policy is to never let pipeline DSLs contain any actual logic outside orchestration for the task, relying solely on one-liner build or test commands. If the task is more complicated than a one-liner, make a script for it in the repo to make it a one-liner. Doesn't matter if it's GitHub Actions, Jenkins, Azure DevOps (which has super cursed yaml), etc. This in turn means that you can do what the pipeline does with a one-liner too, whether manually, from a vscode launch command, a git hook, etc. This same approach can fix the mess of path-specific validation too - write a regular script (shell, python, JS, whatever you fancy) that checks what has changed and calls the appropriate validation script. The GitHub action is only used to run the script on PR and to prepare the CI container for whatever the script needs, and the same pipeline will always run.

Tainnor 2y ago
The reason why many CI configs devolve into such a mess isn't typically that they don't extract complicated logic into scripts, it's about all the interactions with the CI system itself. This includes caching, sharing of artifacts, generating reports, configuring permissions, ordering of jobs, deciding when which jobs will run, deciding what to do when jobs fail, etc. All of this can get quite messy in a large enough project.
- pydry 2y ago
  It never becomes unbearably messy this way though. The reason it gets unbearably messy is because most people google "how to do x in github actions" (e.g. send a slack message) and there is a way and it's almost always worse than scripting it yourself.
  - SOLAR_FIELDS 2y ago
    The reason it gets unbearably messy is that GitHub has constructed an ecosystem that encourages developers to write Turing complete imperative behavior into YAML without providing the same language constructs/tooling that a proper adult language provides to encourage code reuse and debugging. Without tooling like this any sufficiently complex system is guaranteed to evolve into a spaghetti mess, because no sane way exists to maintain such a system at scale without proper tooling, which one would need to hand roll themselves against a giant, ever changing mostly undocumented black box proprietary system (GitHub Actions). Someone tried to do this, the project is called “act”. The results are described by the author in the article as “subpar”. The only sane way to use GitHub Actions at scale is to take the subset of its features that you leverage to perform the execution (event triggers, runs-on, etc) and only use those features, and farm out all the rest of the work in something that is actually maintainable eg Buildkit, Bazel, Gradle etc
    - WorldMaker 2y ago
      Which I feel is a recurring lesson in CI in general. CI systems have to be scriptable to get their job done because they can't anticipate every build system. With even terrible scriptability comes Turing completeness, because it is hard to avoid, a natural law. Eventually someone figures out how to make it do wild things. Eventually those wild things becomes someone's requirements in their CI pipeline. Eventually someone is blog posting about how terrible that entire CI system is because of how baroque their pipeline has become and how many crazy scripts it has that are hard to test and harder to fix. It's like the circle of life for CI systems.
    - Tainnor 2y ago
      It's not just Github, Gitlab uses the same mess of YAML-programming with custom extensions. So do, I believe, many other systems. Ironically, Jenkins does this one correctly, they just give you a regular programming language. It's a shame there are so many other pitfalls, though.
      - SOLAR_FIELDS 2y ago
        Jenkins has almost the opposite problem: they give you the freedom to do everything, but point the gun right at your foot in the process of doing so
- bluGill 2y ago
  You should generate your report with regular scripts. You need ci config to deploy them but that is the only part that should be different.
  - evilduck 2y ago
    This doesn’t really work when you start sharding tests.
    - bluGill 2y ago
      It works just fine - you have ci scripts for tests-group-1, test-group-2, and so on. You data collection will need to aggregate data from them all, but that is something most data collection systems have (and at least the ones I know of also allow individuals to upload their local test data thus meaning you can test that upload locally). If you break those test groups up right most developers will know which they should run as a result of their changes (if your tests are not so long that developers wouldn't run them all before pushing then you shouldn't shard anyway, though it may be reasonable to say your CI shards are still longer than what a developer would run locally) I have had many tests which manipulate the global environment (integration tests should do this, though I'm not convinced the distinction between integration and unit tests is valuable) so the ability run the same tests as CI is very helpful in finding and verifying you fixed these.
- DanielHB 2y ago
  Caching and sharing artifacts is usually the main culprit. My company has been using https://nx.dev/ https://nx.dev/ for that. It works locally as well and CI and it just works. Our NX is pointed to store artifacts in GHA, but our GHA scripts don't do any caching directly, it is all handled by NX. It works so well I would even consider pulling a nodejs environment to run it in non-nodejs projects (although I haven't tried, probably would run into some problems). It is somewhat heavy on configuration, but it just moves the complexity from CI configuration to NX configuration (which is nicer IMO). Our CI pipelines are super fast if you don't hit one of one of our slow compilling parts of the codebase. With NX your local dev environment can pull cached items that were built from previous CI ran-jobs or other devs. We have some native C++ dependencies that are kind of a pain to build locally, our dev machines can pull the built binaries built by other devs (since all devs and CI also share the same cache-artifacts storage). So it makes developing locally a lot easier as well, I don't even remember last time I had to build the native C++ stuff myself since I don't work on it.
  - anbotero 2y ago
    Do you know the criteria used to pick the nx.dev? That is, do you pay for their Cloud, or do you do some plumbing yourselves to make it work on GitHub and other things? Looks interesting. We’ve picked tools based on time saved without too much extra knowledge or overhead required, so this may prove promising.
    - DanielHB 2y ago
      To be honest, I wasn't the one who added it and have only occasionally done some small changes to the NX configuration. I don't think we pay for their Cloud, I think all our stored artifacts are stored in GHA caching system and we pull them using our github SSH keys. Although I don't know exactly how that was set up. The fact that someone set it up and I just started using it and it just works is a testament of how good it works. NX is good because it does the caching part of CI in a way that works both locally and on CI. But of course it doesn't really help at all with the other points raised by the OP. One interesting thing about NX as well is that it helps with you managing your own local build chain, like in the example I mentioned above, when I run a project that requires the C++ native dependency, that project gets built automatically (or rather my computer pulls the built binaries from the remote cache). But for all of this to work you need to set up these dependency chains explicitly in your NX configuration, but that is formalizing an actual requirement instead of leaving it implicit (or in Makefiles or in scripts that only run in CI). I do have to say that our NX configuration is quite long though, but I feel that once you start using NX it is just too tempting to split your project up in individual cacheable steps even if said steps are very fast to run and produce no artifacts. Although you don't have to. For example we have separate steps for linting, typescript type-checking, code formatting, unit testing for each unique project in our mono-repo. In practice they could be all the same step because they all get invalidated at the same time (basically on any file change).
izietto 2y ago
This. I'd go more relaxed on the one-liner requirement, a few lines are fine, but the approach is correct IMHO.
jicea 2y ago
I don't understand why this is not the evident approach for everyone writing GitHub Actions/GitLab CI/CD yaml etc.... I've struggled in some teams to explained why it's better to extract your command in scripts (ShellCheck on it, scripts are simple to run locally etc...) instead of writing a Frankenstein of YAML and shell commands. I hope someday to find an authoritative guidelines on writing pipeline that promote this approach so at least I can point to this link instead of defending myself being a dinosaur!
- arghwhat 2y ago
  My "favorite" is when I see people go all in, writing thousands of lines of Jenkins-flavor Groovy that parses JSON build specifications of arbitrary complexity to sort out how to build that particular project. "But then we can reuse the same pipeline for all our projects!"
  - lowercased 2y ago
    I can rarely reuse the same pipeline for the same project 6 months down the road, much less reuse for anything else. The few bits that end up getting reused are the externalized bash scripts.
  - ozim 2y ago
    I think that is pitfall of software devs. For me it was an epiphany as software dev - not to write reusable extensible scripts - I am so much more productive after that.
  - baby_souffle 2y ago
    > "But then we can reuse the same pipeline for all our projects!" oh god just reading that gave me PTSD flash backs. At $priorGig there was the "omni-chart". It was a helm chart that was so complex it needed to be wrapped in terraform and used composable terraform modules w/ user var overrides as needed. Debugging anything about it meant clearing your calendar for the day and probably the following day, too.
- mrweasel 2y ago
alkonaut 2y ago
That’s usually very hard or impossible for many things. The AzDo yaml consists of a lot of steps that are specific to the CI environment (fetching secrets, running tests on multiple nodes, storing artifacts of various kinds). Even if the ”meat” of the script is a single build.ps oneliner, I quickly end up with 200 line yaml scripts which have no chance of working locally.
- arghwhat 2y ago
  Azure DevOps specifically has a very broken approach to YAML pipelines, because they effectively took their old graphical pipeline builder and just made a YAML representation of it. The trick to working with this is that you don't need any of their custom Azure DevOps task types, and can use the shell type (which has a convenient shorthand) just as well as in any other CI environment. Even the installer tasks are redundant - in other CI systems, you either use a container image with what you need, or install stuff at the start, and Azure DevOps works with both of these strategies. So no, it's neither hard nor impossible, but Microsoft's half-assed approach to maintaining Azure DevOps and overall overcomplicated legacy design makes it a bit hard to realize that doing what their documentation suggests is a bad idea, and that you can use it in a modern way just fine. At least their docs do not recommend that you use the dedicated NPM-type task for `npm install` anymore... (I could rant for ages about Azure DevOps and how broken and unloved it is from Microsoft's side. From what I can tell, they're just putting in the minimum effort to keep old Enterprise customers that have been there through every rename since Team Foundation Server from jumping ship - maybe just until Github's enterprise side has matured enough? Azure DevOps doesn't even integrate well with Azure, despite its name!)
  - mplanchard 2y ago
    This doesn’t seem to address the parent comment’s point at all, which was about required non-shell configuration such as for secrets, build parallelism, etc.
  - noen 2y ago
    It has been on life support for a long time AFAIK. I designed Visual Studio Online (the first launch of AzDO) - and every engineer, PM, and executive I worked with is either in leadership at GitHub or retired.
potamic 2y ago
This is the right way to use CI/CD systems, as dumb orchestrators without inherent knowledge of your software stack. But the problem is, everything from their documentation, templates, marketplace encourage you to do exactly the opposite and couple your build tightly with their system. It's poor product design imo, clearly optimising for vendor lock-in over usability.
riperoni 2y ago
I agree with wrapping things like build scripts to test locally. Still, some actions or CI steps are also not meant to be run locally. Like when it publishes to a repo or needs any credentials that are used by more than one person. Btw, Github actions and corresponding YAML are derived from Azure DevOps and are just as cursed. The whole concept of Github CI is just pure misuse of containers when you need huge VM images - container is technically correct, but a far fetched word for this - that have all kinds of preinstalled garbage to run typescript-wrapped code to call shell scripts.
nunez 2y ago
While youre correct, environmental considerations are another advantage that testing locally SHOULD be able to provide (i.e. you can test your scripts or Make targets or whatever in the same runner that runs in the actual build system.) This is not possible with GHA.
- arghwhat 2y ago
  Of course you can, just specify a container image of your choice and run the same container for testing locally. However, replicating environmental details is only relevant where the details are known to matter. A lot of effort has been wasted and workflows crippled by the idea that everything must be 100% identical irrespective of actual dependencies and real effects.
hinkley 2y ago
I’ll go so far as to say the massive add on/plugin list and featuritis of CI/CD tools is actively harmful to the sanity of your team. The only functionality a CI tool should be providing is: - starting and running an environment to build shit in - accurately tracking success or failure - accurate association of builds with artifacts - telemetry (either their own or integration) and audit trails - correlation with project planning software - scheduled builds - build chaining That’s a lot, but it’s a lot less than any CI tool made in the last 15 years does, and that’s enough. There’s a big difference for instance between having a tool that understands Maven information enough to present a build summary, and one with a Maven fetch/push task. The latter is a black box you can’t test locally, and your lead devs can’t either, so when it breaks, it triggers helplessness. If the only answer to a build failure is to stare at config and wait for enlightenment, you fucked up.
no_wizard 2y ago
Folks pick the wrong tool for the job at hand. I suspect the author of the article could greatly simplify matters if they used a task running tool to orchestrate running tasks, for example. Pick whatever manner of decoupling you want really, most of the time this is the path to simplified CI actions. CI is best when its thought of as a way to stand up fresh copies of an environment to run things inside of. I have never had the struggles that so many have had with CI as a result. Frankly, I'm consistently surprised at how overly complex people make their CI configurations. There's better tools for orchestration and dependency dependent builds, which is not its purpose to begin with.
- ledauphin 2y ago
  I generally agree with you, but I'd be interested to hear your take on what the purpose of CI _actually is_. It seems to me that a big part of the problem here (which I have also seen/experienced) is that there's no one specific thing that something like GitHub Actions is uniquely suited for. Instead, people want "a bunch of stuff to happen" when somebody pushes a commit, and they imagine that the best way to trigger all of that is to have an incredibly complex - and also bespoke - system on the other end that does all of it. It's like we learned the importance of modularity in the the realm of software design, but never applied what we learned to the tools that we work with.
  - no_wizard 2y ago
    Standing up fresh images for validation and post validation tasks. CI shines for running tests against a clean environment for example. Really any task that benefits from a clean image being stood before running a task. The key though is to decouple the tasks from the CI. Complexity like pre-packaging artifacts is not a great fit for CI configuration, that is best pushed to a tool that doesn’t require waterfall logic to make it work. There is a reason task runners are very popular still
ukoki 2y ago
> My policy is to never let pipeline DSLs contain any actual logic outside orchestration for the task, I call this “isomorphic CI” — ie: as long as you set the correct env vars, it should run identically on GitHub actions, Jenkins, your local machine, a VM etc
- reactordev 2y ago
  This is the only DevOps way. Abstract the build into a single step.
  - vvillena 2y ago
    And yet, you would be surprised at the amount of people who react like that's an ignorant statement ("not feasible in real world conditions"), an utopic goal ("too much time to implement"), an impossible feat ("automation difficults human oversight"), or, my favorite, the "this is beneath us" excuse ("see, we are special and this wouldn't work here"). Automation renders knowledge into a set of executable steps, which is much better than rendering knowledge into documentation, or leaving it to rot in people's minds. Compiling all rendered knowledge into a single step is the easiest way to ensure all elements around the build and deployment lifecycle work in unison and are guarded around failures.
    - necovek 2y ago
      Building a Github-specific CI pipeline similarly transfer it into a set of executable steps. The only difference is that you are now tied to a vendor for executing that logic, and the issue is really that this tooling is proprietary software (otherwise, you could just take their theoretical open source runner system and run it locally). To me, this is mostly a question of using non-open-source development tools or not.
- jamesfinlayson 2y ago
  Yep. I remember at a previous company multiple teams had manually created steps in TeamCity (and it wasn't even being backed up in .xml files). I just did my own thing and wrapped everything deploy.sh and test.sh and when the shift to another system came... well it was still kind of annoying, but at least I wasn't recreating the whole thing.
InvertedRhodium 2y ago
I use gitlab-ci-local to run Gitlab pipelines locally - does such a thing not exist for GitHub actions?
- mdaniel 2y ago
  Oh, yeah, I remember looking at that a while back. I don't recall how much it had implemented at the time but it seems that firecow took a vastly different approach than nektos/act did, going so far as to spend what must have been an enormous amount of time/energy to cook up https://github.com/firecow/gitlab-ci-local/blob/4.56.2/src/schema/schema.ts https://github.com/firecow/gitlab-ci-local/blob/4.56.2/src/s... (and, of course, choosing a dynamically typed language versus golang)
0xbadcafebee 2y ago
100%. The ci/cd job should be nothing more than a wrapper around the actual logic which is code in your repo. I write a script called `deploy.sh` which is my wrapper for my ci/cd jobs. It takes options and uses those options to find the piece of code to run. The ci/cd job can be parameterized or matrixed. The eventually-run individual jobs have arguments, and those are passed to deploy.sh. Secrets/environment variables are set from the ci/cd system, also parameterized/matrixed (or alternately, a self-hosted runner can provide deploy.sh access to a vault). End result: from my laptop I can run `deploy.sh deploy --env test --modules webserver` to deploy the webserver to test, and the CI/CD job also runs the same job the same way. The only thing I maintain that's CI/CD-specific is the GitHub Action-specific logic of how to get ready to run `deploy.sh`, which I write once and never change. Thus I could use 20 different CI/CD systems, but never have to refactor my actual deployment code, which also always works on my laptop. Vendor lock-in is impossible, thanks to a little abstraction. (If you have ever worked with a team with 1,000 Jenkins jobs and the team has basically decided they can never move off of Jenkins because it would take too much work to rewrite all the jobs, you'll understand why I do it this way)
- hardwaresofton 2y ago
  Hey if you’ve never heard of it consider using just[0], it’s a better makefile and supports shell scripting explicitly (so at least equivalent in power, though so is Make) [0]: https://github.com/casey/just https://github.com/casey/just
  - 0xbadcafebee 2y ago
    Thank you, I have seen it, but I prefer Make.
    - eru 2y ago
      I bet all your targets are .PHONY?
      - derkades 2y ago
Norfair 2y ago
https://nix-ci.com https://nix-ci.com does this by construction.
rtpg 2y ago
The actual subtle issue here is that sometimes you actually need CI features around caching and the like, so you are forced to engage with the format a bit. You can, of course, chew it down to a bare minimum. But I really wish more CI systems would just show up with "you configure us with scripts" instead of the "declarative" nonsense.
- Guvante 2y ago
  CI that isn't running on your servers wants very deep understanding of how your process works so they can minimize their costs (this is true whether or not you pay for using CI)
  - rtpg 2y ago
    Totally! It's a legitimate thing! I just wish that I had more tools for dynamically providing this information to CI so that it could work better but I could also write relatively general tooling with a general purpose language. The ideal for me is (this is very silly and glib and a total category error) LSP but for CI. Tooling that is relatively normalized, letting me (for example) have a pytest plugin that "does sharding" cleanly across multiple CI operators. There's some stuff and conventions already of course, but in particular caching and spinning up jobs dynamically are still not there.
thatxliner 2y ago
How would you set up tool installations? Inside the CI or inside the script?
- atkbrah 2y ago
  In our environment (our product is a windows desktop application) we use packer to build a custom windows server 2022 image with all the required tools installed. Build agents run on a azure vm scale set that uses the said image for the instance os.
cnunciato 2y ago
>Lack of local development. It's a known thing that there is no way of running GitHub Actions locally. This is one thing I really love about Buildkite[0] -- being able to run the agent locally. (Full disclosure: I also work for Buildkite.) The Buildkite agent runs as a normal process too (rather than as a Docker container), which makes the process of workflow development way simpler, IMO. I also keep a handful of agents running locally on my laptop for personal projects, which is nice. (Why run those processes on someone else's infra if I don't have to?) >Reusability and YAML This is another one that I believe is unique to Buildkite and that I also find super useful. You can write your workflows in YAML of course -- but you can also write them in your language of choice, and then serialize to YAML or JSON when you're ready to start the run (or even add onto the run as it's running if you need to). This lets you encapsulate and reuse (and test, etc.) workflow logic as you need. We have many large customers that do some amazing things with this capability. [0]: https://buildkite.com/docs/agent/v3 https://buildkite.com/docs/agent/v3
- whstl 2y ago
  Are you two talking about the same thing? I believe the grandparent is talking about running it locally on development machines, often for testing purposes. Asking because Github Action also supports Self-Hosted runners [1]. [1] https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/using-self-hosted-runners-in-a-workflow https://docs.github.com/en/actions/hosting-your-own-runners/...
  - cnunciato 2y ago
    Same thing, yeah, IIUC (i.e., running the agent/worker locally for testing). It's conceptually similar to self-hosted runners, yes, but also different in a few practical ways that may matter to you, depending on how you plan to run in production. For one, with GitHub Actions, hosted and self-hosted runners are fundamentally different applications; hosted runners are fully configured container images, (with base OS, tools, etc., on board), whereas self-hosted runners are essentially thin, unconfigured shell scripts. This means that unless you're planning on using self-hosted runners in production (which some do of course, but most don't), it wouldn't make sense to dev/test with them locally, given how different they are. With Buildkite, there's only one "way" -- `buildkite-agent`, the single binary I linked to above. The connection models are also different. While both GHA self-hosted runners and the Buildkite agent connect to a remote service to claim and run jobs, GHA runners must first be registered with a GitHub org or repository before you can use them, and then workflows must also be configured to use them (e.g., with `runs-on` params). With Buildkite, any `buildkite-agent` with a proper token can connect to a queue to run a job. There are others, but hopefully that gives you an idea.
mhh__ 2y ago
Indeed. Anything else is just asking for trouble. CI must only be an application of tools available to developers locally. This, other than for practical reasons, is a tax on complexity.
sirlone 2y ago
So for example, my action builds on 4 different platforms (win-64, linux-amd64, mac-intel, mac-arm), it does this in parallel then gets the artifacts for all for and bundles them into a single package. How would you suggest I do this following your advice?
ironfootnz 2y ago
That’s my policy too. I see way too many Jenkins/Actions scripts with big logic blocks jammed into YAML. If the entire build and test process is just a single script call, we can run it locally, in a GitHub workflow, or anywhere else. Makes it less painful to switch CI systems, and devs can debug easily without pushing blind commits. It’s surprising how many teams don’t realize local testing alone saves huge amounts of time.
saghm 2y ago
This is my preferred way of doing things as well. Not being able to run the exact same thing that's running in CI easily locally is a bit of a red flag in my opinion. I think the only exception I've ever encountered to this is when working on client software for HSMs, which had some end-to-end tests that couldn't be run without actually connecting to the specific hardware that took some setup to be able to access when running tests locally.
lloeki 2y ago
I'm increasingly designing CI stuff around rake tasks. Then I run rake in the workflow. But that caters only for each individual command... as you mention the orchestration is still coded in, and duplicated from what rake knows and would do. So I'm currently trying stuff that has a pluggable output: one output (the default) is that it runs stuff, but with just a rake var, instead of generating then running commands it generates workflow content that ultimately gets merged in an ERB workflow template. The model I like the most though is Nix-style distributed builds: it doesn't matter if you do `nix build foo#bar` (local) or `nix build -j0 foo#bar` (zero local jobs => use a remote builder†), the `foo#bar` "task" and its dependents gets "built" (a.k.a run). † builders get picked matching target platform and label-like "features" constraints. Ever since there has been gitlab-runner, I've wondered why the hell can't I just submit some job to a (list of) runner(s) - some of which could be local - without the whole push-to-repo+CI orchestrator? I mean I don't think it would be out of this world to write a CLI command that locally parses whatever-ci.yml, creates jobs out of it, and submit them to a local runner.
Sparkyte 2y ago
When I automate my github actions I keep everything task orientated and if anything is pushing code or builds it has user step to verify the work by the automation. You approve and then merge it kicks off promotional pipelines not necessarily for deployment but to promote a build as stable through tagging.
jcarrano 2y ago
100% agree and that has been my experience too. It also makes testing the logic locally much easier, just run the script in the appropriate container. The pipeline DSLs, because they are not full programming languages have to include lots of specific features and options and if you want something slightly outside of what they are designed for, you are out of luck. In a way it feels like how graphics were in the age of fixed-pipeline, when there had to be a complex API to cover all use cases yet it was not flexible enough.
azriel91 2y ago
there's: https://github.com/Pernosco/gha-runner https://github.com/Pernosco/gha-runner but I'm not sure how complete it is, and probably doesn't satisfy the author's use cases
necovek 2y ago
I think it's mostly obvious how you could implement CI with "local-first" programs (scripts?), but systems like Github provide value on top by making some of the artifacts or steps first-class objects. Not sure how much of that does Github do, but it could parse test output (I know we used that feature in Gitlab back when I was using that on a project), track flaky tests for you or give you a link to code for a failing test directly. Or it could highlight releases on their "releases" page, with release notes prominently featured. And they allow you to group pipelines by type and kind, filter by target environment and such. On top of that, they provide a library of re-usable workflows like AWS login, or code checkout or similar. With all that, the equation is not as clear cut: you need to figure out how to best leverage some of those (or switch to external tools providing them), and suddenly, with time pressure, just going with the flow is a more obvious choice.
olafmol 2y ago
Although there are definitely merits in moving the complex logic outside of the CI/CD JSON/YAML DSL, especially when using monorepo setups that can become rather complex in their logic (that they made Google create Bazel, I can think of some interesting Borg/K8s analogies btw), I also believe that modern CI/CD platforms have made several sensible steps in the right direction to handle these more complicated use cases. (Disclaimer: I work at CircleCI) At CircleCI for example, we have added valuable features like a VSCode extension[0] to validate and "dry-run" config from within your IDE, we have local runners[1] that you can use to test and run pipelines on your local machine and your own infra, we have dynamic config[2], a Javascript/Typescript SDK[3], a CLI that can validate and run workflows locally[4], and QoL additions like a no-op job type[5] and flexible requires, along with flexible when statements and expression based job filters[6]. And finally, it's of course also possible to combine different approaches into a "best of both worlds" approach, f.e. combining Dagger with CircleCI[7]. [0]https://circleci.com/docs/vs-code-extension-overview/ https://circleci.com/docs/vs-code-extension-overview/ [1]https://circleci.com/blog/using-runner-for-local-testing/ https://circleci.com/blog/using-runner-for-local-testing/ [2]https://circleci.com/docs/dynamic-config/ https://circleci.com/docs/dynamic-config/ [3]https://circleci.com/docs/circleci-config-sdk/ https://circleci.com/docs/circleci-config-sdk/ [4]https://circleci.com/docs/how-to-use-the-circleci-local-cli/ https://circleci.com/docs/how-to-use-the-circleci-local-cli/ [5]https://circleci.com/changelog/new-job-type-no-op-job-can-make-complex-workflows-simple/ https://circleci.com/changelog/new-job-type-no-op-job-can-ma... [6]https://circleci.com/changelog/more-flexible-job-required-capabilities/ https://circleci.com/changelog/more-flexible-job-required-ca... [7]https://docs.dagger.io/integrations/circleci/ https://docs.dagger.io/integrations/circleci/

Tainnor 2y ago

The reason why many CI configs devolve into such a mess isn't typically that they don't extract complicated logic into scripts, it's about all the interactions with the CI system itself. This includes caching, sharing of artifacts, generating reports, configuring permissions, ordering of jobs, deciding when which jobs will run, deciding what to do when jobs fail, etc. All of this can get quite messy in a large enough project.

pydry 2y ago
It never becomes unbearably messy this way though. The reason it gets unbearably messy is because most people google "how to do x in github actions" (e.g. send a slack message) and there is a way and it's almost always worse than scripting it yourself.
- SOLAR_FIELDS 2y ago
  The reason it gets unbearably messy is that GitHub has constructed an ecosystem that encourages developers to write Turing complete imperative behavior into YAML without providing the same language constructs/tooling that a proper adult language provides to encourage code reuse and debugging. Without tooling like this any sufficiently complex system is guaranteed to evolve into a spaghetti mess, because no sane way exists to maintain such a system at scale without proper tooling, which one would need to hand roll themselves against a giant, ever changing mostly undocumented black box proprietary system (GitHub Actions). Someone tried to do this, the project is called “act”. The results are described by the author in the article as “subpar”. The only sane way to use GitHub Actions at scale is to take the subset of its features that you leverage to perform the execution (event triggers, runs-on, etc) and only use those features, and farm out all the rest of the work in something that is actually maintainable eg Buildkit, Bazel, Gradle etc
  - WorldMaker 2y ago
    Which I feel is a recurring lesson in CI in general. CI systems have to be scriptable to get their job done because they can't anticipate every build system. With even terrible scriptability comes Turing completeness, because it is hard to avoid, a natural law. Eventually someone figures out how to make it do wild things. Eventually those wild things becomes someone's requirements in their CI pipeline. Eventually someone is blog posting about how terrible that entire CI system is because of how baroque their pipeline has become and how many crazy scripts it has that are hard to test and harder to fix. It's like the circle of life for CI systems.
  - Tainnor 2y ago
    It's not just Github, Gitlab uses the same mess of YAML-programming with custom extensions. So do, I believe, many other systems. Ironically, Jenkins does this one correctly, they just give you a regular programming language. It's a shame there are so many other pitfalls, though.
    - SOLAR_FIELDS 2y ago
      Jenkins has almost the opposite problem: they give you the freedom to do everything, but point the gun right at your foot in the process of doing so
bluGill 2y ago
You should generate your report with regular scripts. You need ci config to deploy them but that is the only part that should be different.
- evilduck 2y ago
  This doesn’t really work when you start sharding tests.
  - bluGill 2y ago
    It works just fine - you have ci scripts for tests-group-1, test-group-2, and so on. You data collection will need to aggregate data from them all, but that is something most data collection systems have (and at least the ones I know of also allow individuals to upload their local test data thus meaning you can test that upload locally). If you break those test groups up right most developers will know which they should run as a result of their changes (if your tests are not so long that developers wouldn't run them all before pushing then you shouldn't shard anyway, though it may be reasonable to say your CI shards are still longer than what a developer would run locally) I have had many tests which manipulate the global environment (integration tests should do this, though I'm not convinced the distinction between integration and unit tests is valuable) so the ability run the same tests as CI is very helpful in finding and verifying you fixed these.
DanielHB 2y ago
Caching and sharing artifacts is usually the main culprit. My company has been using https://nx.dev/ https://nx.dev/ for that. It works locally as well and CI and it just works. Our NX is pointed to store artifacts in GHA, but our GHA scripts don't do any caching directly, it is all handled by NX. It works so well I would even consider pulling a nodejs environment to run it in non-nodejs projects (although I haven't tried, probably would run into some problems). It is somewhat heavy on configuration, but it just moves the complexity from CI configuration to NX configuration (which is nicer IMO). Our CI pipelines are super fast if you don't hit one of one of our slow compilling parts of the codebase. With NX your local dev environment can pull cached items that were built from previous CI ran-jobs or other devs. We have some native C++ dependencies that are kind of a pain to build locally, our dev machines can pull the built binaries built by other devs (since all devs and CI also share the same cache-artifacts storage). So it makes developing locally a lot easier as well, I don't even remember last time I had to build the native C++ stuff myself since I don't work on it.
- anbotero 2y ago
  Do you know the criteria used to pick the nx.dev? That is, do you pay for their Cloud, or do you do some plumbing yourselves to make it work on GitHub and other things? Looks interesting. We’ve picked tools based on time saved without too much extra knowledge or overhead required, so this may prove promising.
  - DanielHB 2y ago
    To be honest, I wasn't the one who added it and have only occasionally done some small changes to the NX configuration. I don't think we pay for their Cloud, I think all our stored artifacts are stored in GHA caching system and we pull them using our github SSH keys. Although I don't know exactly how that was set up. The fact that someone set it up and I just started using it and it just works is a testament of how good it works. NX is good because it does the caching part of CI in a way that works both locally and on CI. But of course it doesn't really help at all with the other points raised by the OP. One interesting thing about NX as well is that it helps with you managing your own local build chain, like in the example I mentioned above, when I run a project that requires the C++ native dependency, that project gets built automatically (or rather my computer pulls the built binaries from the remote cache). But for all of this to work you need to set up these dependency chains explicitly in your NX configuration, but that is formalizing an actual requirement instead of leaving it implicit (or in Makefiles or in scripts that only run in CI). I do have to say that our NX configuration is quite long though, but I feel that once you start using NX it is just too tempting to split your project up in individual cacheable steps even if said steps are very fast to run and produce no artifacts. Although you don't have to. For example we have separate steps for linting, typescript type-checking, code formatting, unit testing for each unique project in our mono-repo. In practice they could be all the same step because they all get invalidated at the same time (basically on any file change).

izietto 2y ago

This. I'd go more relaxed on the one-liner requirement, a few lines are fine, but the approach is correct IMHO.

jicea 2y ago

I don't understand why this is not the evident approach for everyone writing GitHub Actions/GitLab CI/CD yaml etc.... I've struggled in some teams to explained why it's better to extract your command in scripts (ShellCheck on it, scripts are simple to run locally etc...) instead of writing a Frankenstein of YAML and shell commands. I hope someday to find an authoritative guidelines on writing pipeline that promote this approach so at least I can point to this link instead of defending myself being a dinosaur!

arghwhat 2y ago
My "favorite" is when I see people go all in, writing thousands of lines of Jenkins-flavor Groovy that parses JSON build specifications of arbitrary complexity to sort out how to build that particular project. "But then we can reuse the same pipeline for all our projects!"
- lowercased 2y ago
  I can rarely reuse the same pipeline for the same project 6 months down the road, much less reuse for anything else. The few bits that end up getting reused are the externalized bash scripts.
- ozim 2y ago
  I think that is pitfall of software devs. For me it was an epiphany as software dev - not to write reusable extensible scripts - I am so much more productive after that.
- baby_souffle 2y ago
  > "But then we can reuse the same pipeline for all our projects!" oh god just reading that gave me PTSD flash backs. At $priorGig there was the "omni-chart". It was a helm chart that was so complex it needed to be wrapped in terraform and used composable terraform modules w/ user var overrides as needed. Debugging anything about it meant clearing your calendar for the day and probably the following day, too.
mrweasel 2y ago

alkonaut 2y ago

That’s usually very hard or impossible for many things. The AzDo yaml consists of a lot of steps that are specific to the CI environment (fetching secrets, running tests on multiple nodes, storing artifacts of various kinds). Even if the ”meat” of the script is a single build.ps oneliner, I quickly end up with 200 line yaml scripts which have no chance of working locally.

arghwhat 2y ago
Azure DevOps specifically has a very broken approach to YAML pipelines, because they effectively took their old graphical pipeline builder and just made a YAML representation of it. The trick to working with this is that you don't need any of their custom Azure DevOps task types, and can use the shell type (which has a convenient shorthand) just as well as in any other CI environment. Even the installer tasks are redundant - in other CI systems, you either use a container image with what you need, or install stuff at the start, and Azure DevOps works with both of these strategies. So no, it's neither hard nor impossible, but Microsoft's half-assed approach to maintaining Azure DevOps and overall overcomplicated legacy design makes it a bit hard to realize that doing what their documentation suggests is a bad idea, and that you can use it in a modern way just fine. At least their docs do not recommend that you use the dedicated NPM-type task for `npm install` anymore... (I could rant for ages about Azure DevOps and how broken and unloved it is from Microsoft's side. From what I can tell, they're just putting in the minimum effort to keep old Enterprise customers that have been there through every rename since Team Foundation Server from jumping ship - maybe just until Github's enterprise side has matured enough? Azure DevOps doesn't even integrate well with Azure, despite its name!)
- mplanchard 2y ago
  This doesn’t seem to address the parent comment’s point at all, which was about required non-shell configuration such as for secrets, build parallelism, etc.
- noen 2y ago
  It has been on life support for a long time AFAIK. I designed Visual Studio Online (the first launch of AzDO) - and every engineer, PM, and executive I worked with is either in leadership at GitHub or retired.

potamic 2y ago

This is the right way to use CI/CD systems, as dumb orchestrators without inherent knowledge of your software stack. But the problem is, everything from their documentation, templates, marketplace encourage you to do exactly the opposite and couple your build tightly with their system. It's poor product design imo, clearly optimising for vendor lock-in over usability.

riperoni 2y ago

I agree with wrapping things like build scripts to test locally. Still, some actions or CI steps are also not meant to be run locally. Like when it publishes to a repo or needs any credentials that are used by more than one person. Btw, Github actions and corresponding YAML are derived from Azure DevOps and are just as cursed. The whole concept of Github CI is just pure misuse of containers when you need huge VM images - container is technically correct, but a far fetched word for this - that have all kinds of preinstalled garbage to run typescript-wrapped code to call shell scripts.

nunez 2y ago

While youre correct, environmental considerations are another advantage that testing locally SHOULD be able to provide (i.e. you can test your scripts or Make targets or whatever in the same runner that runs in the actual build system.) This is not possible with GHA.

arghwhat 2y ago
Of course you can, just specify a container image of your choice and run the same container for testing locally. However, replicating environmental details is only relevant where the details are known to matter. A lot of effort has been wasted and workflows crippled by the idea that everything must be 100% identical irrespective of actual dependencies and real effects.

hinkley 2y ago

I’ll go so far as to say the massive add on/plugin list and featuritis of CI/CD tools is actively harmful to the sanity of your team. The only functionality a CI tool should be providing is: - starting and running an environment to build shit in - accurately tracking success or failure - accurate association of builds with artifacts - telemetry (either their own or integration) and audit trails - correlation with project planning software - scheduled builds - build chaining That’s a lot, but it’s a lot less than any CI tool made in the last 15 years does, and that’s enough. There’s a big difference for instance between having a tool that understands Maven information enough to present a build summary, and one with a Maven fetch/push task. The latter is a black box you can’t test locally, and your lead devs can’t either, so when it breaks, it triggers helplessness. If the only answer to a build failure is to stare at config and wait for enlightenment, you fucked up.

no_wizard 2y ago

Folks pick the wrong tool for the job at hand. I suspect the author of the article could greatly simplify matters if they used a task running tool to orchestrate running tasks, for example. Pick whatever manner of decoupling you want really, most of the time this is the path to simplified CI actions. CI is best when its thought of as a way to stand up fresh copies of an environment to run things inside of. I have never had the struggles that so many have had with CI as a result. Frankly, I'm consistently surprised at how overly complex people make their CI configurations. There's better tools for orchestration and dependency dependent builds, which is not its purpose to begin with.

ledauphin 2y ago
I generally agree with you, but I'd be interested to hear your take on what the purpose of CI _actually is_. It seems to me that a big part of the problem here (which I have also seen/experienced) is that there's no one specific thing that something like GitHub Actions is uniquely suited for. Instead, people want "a bunch of stuff to happen" when somebody pushes a commit, and they imagine that the best way to trigger all of that is to have an incredibly complex - and also bespoke - system on the other end that does all of it. It's like we learned the importance of modularity in the the realm of software design, but never applied what we learned to the tools that we work with.
- no_wizard 2y ago
  Standing up fresh images for validation and post validation tasks. CI shines for running tests against a clean environment for example. Really any task that benefits from a clean image being stood before running a task. The key though is to decouple the tasks from the CI. Complexity like pre-packaging artifacts is not a great fit for CI configuration, that is best pushed to a tool that doesn’t require waterfall logic to make it work. There is a reason task runners are very popular still

ukoki 2y ago

> My policy is to never let pipeline DSLs contain any actual logic outside orchestration for the task, I call this “isomorphic CI” — ie: as long as you set the correct env vars, it should run identically on GitHub actions, Jenkins, your local machine, a VM etc

reactordev 2y ago
This is the only DevOps way. Abstract the build into a single step.
- vvillena 2y ago
  And yet, you would be surprised at the amount of people who react like that's an ignorant statement ("not feasible in real world conditions"), an utopic goal ("too much time to implement"), an impossible feat ("automation difficults human oversight"), or, my favorite, the "this is beneath us" excuse ("see, we are special and this wouldn't work here"). Automation renders knowledge into a set of executable steps, which is much better than rendering knowledge into documentation, or leaving it to rot in people's minds. Compiling all rendered knowledge into a single step is the easiest way to ensure all elements around the build and deployment lifecycle work in unison and are guarded around failures.
  - necovek 2y ago
    Building a Github-specific CI pipeline similarly transfer it into a set of executable steps. The only difference is that you are now tied to a vendor for executing that logic, and the issue is really that this tooling is proprietary software (otherwise, you could just take their theoretical open source runner system and run it locally). To me, this is mostly a question of using non-open-source development tools or not.
jamesfinlayson 2y ago
Yep. I remember at a previous company multiple teams had manually created steps in TeamCity (and it wasn't even being backed up in .xml files). I just did my own thing and wrapped everything deploy.sh and test.sh and when the shift to another system came... well it was still kind of annoying, but at least I wasn't recreating the whole thing.

InvertedRhodium 2y ago

I use gitlab-ci-local to run Gitlab pipelines locally - does such a thing not exist for GitHub actions?

mdaniel 2y ago
Oh, yeah, I remember looking at that a while back. I don't recall how much it had implemented at the time but it seems that firecow took a vastly different approach than nektos/act did, going so far as to spend what must have been an enormous amount of time/energy to cook up https://github.com/firecow/gitlab-ci-local/blob/4.56.2/src/schema/schema.ts https://github.com/firecow/gitlab-ci-local/blob/4.56.2/src/s... (and, of course, choosing a dynamically typed language versus golang)

0xbadcafebee 2y ago

100%. The ci/cd job should be nothing more than a wrapper around the actual logic which is code in your repo. I write a script called `deploy.sh` which is my wrapper for my ci/cd jobs. It takes options and uses those options to find the piece of code to run. The ci/cd job can be parameterized or matrixed. The eventually-run individual jobs have arguments, and those are passed to deploy.sh. Secrets/environment variables are set from the ci/cd system, also parameterized/matrixed (or alternately, a self-hosted runner can provide deploy.sh access to a vault). End result: from my laptop I can run `deploy.sh deploy --env test --modules webserver` to deploy the webserver to test, and the CI/CD job also runs the same job the same way. The only thing I maintain that's CI/CD-specific is the GitHub Action-specific logic of how to get ready to run `deploy.sh`, which I write once and never change. Thus I could use 20 different CI/CD systems, but never have to refactor my actual deployment code, which also always works on my laptop. Vendor lock-in is impossible, thanks to a little abstraction. (If you have ever worked with a team with 1,000 Jenkins jobs and the team has basically decided they can never move off of Jenkins because it would take too much work to rewrite all the jobs, you'll understand why I do it this way)

hardwaresofton 2y ago
Hey if you’ve never heard of it consider using just[0], it’s a better makefile and supports shell scripting explicitly (so at least equivalent in power, though so is Make) [0]: https://github.com/casey/just https://github.com/casey/just
- 0xbadcafebee 2y ago
  Thank you, I have seen it, but I prefer Make.
  - eru 2y ago
    I bet all your targets are .PHONY?
    - derkades 2y ago

Norfair 2y ago

https://nix-ci.com https://nix-ci.com does this by construction.

rtpg 2y ago

The actual subtle issue here is that sometimes you actually need CI features around caching and the like, so you are forced to engage with the format a bit. You can, of course, chew it down to a bare minimum. But I really wish more CI systems would just show up with "you configure us with scripts" instead of the "declarative" nonsense.

Guvante 2y ago
CI that isn't running on your servers wants very deep understanding of how your process works so they can minimize their costs (this is true whether or not you pay for using CI)
- rtpg 2y ago
  Totally! It's a legitimate thing! I just wish that I had more tools for dynamically providing this information to CI so that it could work better but I could also write relatively general tooling with a general purpose language. The ideal for me is (this is very silly and glib and a total category error) LSP but for CI. Tooling that is relatively normalized, letting me (for example) have a pytest plugin that "does sharding" cleanly across multiple CI operators. There's some stuff and conventions already of course, but in particular caching and spinning up jobs dynamically are still not there.

thatxliner 2y ago

How would you set up tool installations? Inside the CI or inside the script?

atkbrah 2y ago
In our environment (our product is a windows desktop application) we use packer to build a custom windows server 2022 image with all the required tools installed. Build agents run on a azure vm scale set that uses the said image for the instance os.

cnunciato 2y ago

>Lack of local development. It's a known thing that there is no way of running GitHub Actions locally. This is one thing I really love about Buildkite[0] -- being able to run the agent locally. (Full disclosure: I also work for Buildkite.) The Buildkite agent runs as a normal process too (rather than as a Docker container), which makes the process of workflow development way simpler, IMO. I also keep a handful of agents running locally on my laptop for personal projects, which is nice. (Why run those processes on someone else's infra if I don't have to?) >Reusability and YAML This is another one that I believe is unique to Buildkite and that I also find super useful. You can write your workflows in YAML of course -- but you can also write them in your language of choice, and then serialize to YAML or JSON when you're ready to start the run (or even add onto the run as it's running if you need to). This lets you encapsulate and reuse (and test, etc.) workflow logic as you need. We have many large customers that do some amazing things with this capability. [0]: https://buildkite.com/docs/agent/v3 https://buildkite.com/docs/agent/v3

whstl 2y ago
Are you two talking about the same thing? I believe the grandparent is talking about running it locally on development machines, often for testing purposes. Asking because Github Action also supports Self-Hosted runners [1]. [1] https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/using-self-hosted-runners-in-a-workflow https://docs.github.com/en/actions/hosting-your-own-runners/...
- cnunciato 2y ago
  Same thing, yeah, IIUC (i.e., running the agent/worker locally for testing). It's conceptually similar to self-hosted runners, yes, but also different in a few practical ways that may matter to you, depending on how you plan to run in production. For one, with GitHub Actions, hosted and self-hosted runners are fundamentally different applications; hosted runners are fully configured container images, (with base OS, tools, etc., on board), whereas self-hosted runners are essentially thin, unconfigured shell scripts. This means that unless you're planning on using self-hosted runners in production (which some do of course, but most don't), it wouldn't make sense to dev/test with them locally, given how different they are. With Buildkite, there's only one "way" -- `buildkite-agent`, the single binary I linked to above. The connection models are also different. While both GHA self-hosted runners and the Buildkite agent connect to a remote service to claim and run jobs, GHA runners must first be registered with a GitHub org or repository before you can use them, and then workflows must also be configured to use them (e.g., with `runs-on` params). With Buildkite, any `buildkite-agent` with a proper token can connect to a queue to run a job. There are others, but hopefully that gives you an idea.

mhh__ 2y ago

Indeed. Anything else is just asking for trouble. CI must only be an application of tools available to developers locally. This, other than for practical reasons, is a tax on complexity.

sirlone 2y ago

So for example, my action builds on 4 different platforms (win-64, linux-amd64, mac-intel, mac-arm), it does this in parallel then gets the artifacts for all for and bundles them into a single package. How would you suggest I do this following your advice?

ironfootnz 2y ago

That’s my policy too. I see way too many Jenkins/Actions scripts with big logic blocks jammed into YAML. If the entire build and test process is just a single script call, we can run it locally, in a GitHub workflow, or anywhere else. Makes it less painful to switch CI systems, and devs can debug easily without pushing blind commits. It’s surprising how many teams don’t realize local testing alone saves huge amounts of time.

saghm 2y ago

This is my preferred way of doing things as well. Not being able to run the exact same thing that's running in CI easily locally is a bit of a red flag in my opinion. I think the only exception I've ever encountered to this is when working on client software for HSMs, which had some end-to-end tests that couldn't be run without actually connecting to the specific hardware that took some setup to be able to access when running tests locally.

lloeki 2y ago

I'm increasingly designing CI stuff around rake tasks. Then I run rake in the workflow. But that caters only for each individual command... as you mention the orchestration is still coded in, and duplicated from what rake knows and would do. So I'm currently trying stuff that has a pluggable output: one output (the default) is that it runs stuff, but with just a rake var, instead of generating then running commands it generates workflow content that ultimately gets merged in an ERB workflow template. The model I like the most though is Nix-style distributed builds: it doesn't matter if you do `nix build foo#bar` (local) or `nix build -j0 foo#bar` (zero local jobs => use a remote builder†), the `foo#bar` "task" and its dependents gets "built" (a.k.a run). † builders get picked matching target platform and label-like "features" constraints. Ever since there has been gitlab-runner, I've wondered why the hell can't I just submit some job to a (list of) runner(s) - some of which could be local - without the whole push-to-repo+CI orchestrator? I mean I don't think it would be out of this world to write a CLI command that locally parses whatever-ci.yml, creates jobs out of it, and submit them to a local runner.

Sparkyte 2y ago

When I automate my github actions I keep everything task orientated and if anything is pushing code or builds it has user step to verify the work by the automation. You approve and then merge it kicks off promotional pipelines not necessarily for deployment but to promote a build as stable through tagging.

jcarrano 2y ago

100% agree and that has been my experience too. It also makes testing the logic locally much easier, just run the script in the appropriate container. The pipeline DSLs, because they are not full programming languages have to include lots of specific features and options and if you want something slightly outside of what they are designed for, you are out of luck. In a way it feels like how graphics were in the age of fixed-pipeline, when there had to be a complex API to cover all use cases yet it was not flexible enough.

azriel91 2y ago

there's: https://github.com/Pernosco/gha-runner https://github.com/Pernosco/gha-runner but I'm not sure how complete it is, and probably doesn't satisfy the author's use cases

necovek 2y ago

I think it's mostly obvious how you could implement CI with "local-first" programs (scripts?), but systems like Github provide value on top by making some of the artifacts or steps first-class objects. Not sure how much of that does Github do, but it could parse test output (I know we used that feature in Gitlab back when I was using that on a project), track flaky tests for you or give you a link to code for a failing test directly. Or it could highlight releases on their "releases" page, with release notes prominently featured. And they allow you to group pipelines by type and kind, filter by target environment and such. On top of that, they provide a library of re-usable workflows like AWS login, or code checkout or similar. With all that, the equation is not as clear cut: you need to figure out how to best leverage some of those (or switch to external tools providing them), and suddenly, with time pressure, just going with the flow is a more obvious choice.

olafmol 2y ago

Although there are definitely merits in moving the complex logic outside of the CI/CD JSON/YAML DSL, especially when using monorepo setups that can become rather complex in their logic (that they made Google create Bazel, I can think of some interesting Borg/K8s analogies btw), I also believe that modern CI/CD platforms have made several sensible steps in the right direction to handle these more complicated use cases. (Disclaimer: I work at CircleCI) At CircleCI for example, we have added valuable features like a VSCode extension[0] to validate and "dry-run" config from within your IDE, we have local runners[1] that you can use to test and run pipelines on your local machine and your own infra, we have dynamic config[2], a Javascript/Typescript SDK[3], a CLI that can validate and run workflows locally[4], and QoL additions like a no-op job type[5] and flexible requires, along with flexible when statements and expression based job filters[6]. And finally, it's of course also possible to combine different approaches into a "best of both worlds" approach, f.e. combining Dagger with CircleCI[7]. [0]https://circleci.com/docs/vs-code-extension-overview/ https://circleci.com/docs/vs-code-extension-overview/ [1]https://circleci.com/blog/using-runner-for-local-testing/ https://circleci.com/blog/using-runner-for-local-testing/ [2]https://circleci.com/docs/dynamic-config/ https://circleci.com/docs/dynamic-config/ [3]https://circleci.com/docs/circleci-config-sdk/ https://circleci.com/docs/circleci-config-sdk/ [4]https://circleci.com/docs/how-to-use-the-circleci-local-cli/ https://circleci.com/docs/how-to-use-the-circleci-local-cli/ [5]https://circleci.com/changelog/new-job-type-no-op-job-can-make-complex-workflows-simple/ https://circleci.com/changelog/new-job-type-no-op-job-can-ma... [6]https://circleci.com/changelog/more-flexible-job-required-capabilities/ https://circleci.com/changelog/more-flexible-job-required-ca... [7]https://docs.dagger.io/integrations/circleci/ https://docs.dagger.io/integrations/circleci/