3 ms·
> I want my pipelines to be declarative anyway. Is there any CI system that allows one to write declarative pipelines? What would that even mean? GitHub Action
by matrss 3y ago
> I want my pipelines to be declarative anyway.
Is there any CI system that allows one to write declarative pipelines? What would that even mean? GitHub Actions, GitLab Pipelines, etc. are all effectively just shell scripts disguised as more or less verbose yaml, with some trigger conditions added.
I guess nix's hydra is the closest to a declarative CI system that exists, because nix does the hard work of abstracting the imperative build steps into a declarative interface. Even then, if you want to do anything outside of nix derivations you would be writing something imperative.
- pydry 3y agoit's github actions but you keep it short and put everything you can in a script that gets shelled out to.
- matrss 3y agoSo, only use the CI system for it's dispatch/scheduling part and put the imperative steps into a dedicated script? Reasonable, and what I have done with some GitLab Pipelines as well, but I don't see how that would be considered more "declarative".
- summarity 3y agoBuildkite was once that.
- tonyhb 3y agoCheck out https://dagger.io/ https://dagger.io/. Write declarative pipelines in code, reproducibly run anywhere.
- danpalmer 3y agoI'm not looking for literally everything to be declarative, but ideally each snippet of shell script would be entirely independent and run effectively reproducibly. GitHub/GitLab do this well enough, although not perfectly. Concourse does this really well. Bazel is also great at this.
- matrss 3y ago> [...], but ideally each snippet of shell script would be entirely independent and run effectively reproducibly. Sure, if you design your scripts carefully (e.g. make them idempotent, reproducible, make all inputs explicit, etc.) you can build an abstraction to what you want to achieve which can then be used declaratively in the yaml flavor of choice. I think that is a very good way to do it. The script is still imperative though. I guess what I want to say is that a generic CI or even just job scheduling system cannot be declarative without limiting what it can do. You can build your own scripts which can then provide a declarative interface to the imperative world to your CI pipeline, but you will be limited to the scope of your script. You can call out to other such abstractions (nix for your packaging needs, terraform for infrastructure, etc.) as well, but again you will be limited to the scopes of those. As long as the CI system retains the ability to run jobs in some order instead of just being able to declare what you want and let the CI system figure everything else out (e.g. build first, then test, then possibly release; even if I declared what I want in a different order) it is still an imperative interface. To make this clear: I don't think declarative CI/CD pipelines are desirable. They should be an (ideally as short as possible) list of steps executed in order to achieve some goal, because that is what fits the problem domain best (interact with the imperative world in arbitrary ways). Parts of those steps can be pushed out to declarative abstractions though where that makes sense, e.g. building and testing in nix so you get caching, sandboxing and reproducibility of builds basically for free, or deploying infrastructure changes through terraform so you get idempotence on those changes mostly for free. A generic CI system can then be reduced to it's bare minimum: a way to run a short script when some event happens.