4 ms·

This looks interesting, thanks for sharing. I've been looking for a solution to some of the problems that this targets; in general our build steps are container

This looks interesting, thanks for sharing. I've been looking for a solution to some of the problems that this targets; in general our build steps are containerized, but it's annoying to have to layer a `make <step>` command on top of a dependency tree of docker/other commands inside a given CI step, and it's worse to have multi-step commands in the CI/CD config file, making the jobs harder to reproduce. One more general idea I'd be interested in others' thoughts on -- there seems to be an artificial split between build systems (e.g. Bazel) and CI/CD systems (e.g. Gitlab). As these tools become more generic, they tend to overlap almost-completely; ultimately the hard part is DAG management and latency-optimized distributed computation. When you look at "how to use Bazel in Gitlab"[1] or "how to use Earthly in Github Actions"[2], it's basically having a single CI step that calls `buildtool <run my build DAG>`. But there could be other CI/CD steps like "run the e2es", and in general I want multiple steps in my Gitlab CI/CD pipeline so I can break open the individual steps, retry flaky runs, see which step failed, and otherwise get all the nice tight integration that Gitlab affords. Unfortunately I have a DAG in the CI server, and then another DAG in the build job; it really seems like the CI/CD system should somehow unwrap the build-tool's DAG into the CI/CD DAG, then I could rerun individual failed steps etc. (though I think you might perhaps want some folding, because Bazel DAGs are probably in general too fine-grained for the way CI/CD servers tend to think about the visualization problem). Maybe this is just a missing feature in GitLab? It seems like the Github workflow is less well integrated with the CI/CD pipeline (always a click or two to get to the pipeline result details) so maybe users there don't see any missing features. CI/CD systems like Concourse have a really nice model of "artifact revisions as inputs/outputs" which would play really well with this model; different parts of the pipeline accept a hash and will trigger downstream jobs only when those change. I wonder if the Earthly team has any thoughts on integrating with the CI/CD DAG in this way? [1]: https://about.gitlab.com/blog/2020/09/01/using-bazel-to-speed-up-gitlab-ci-builds/ https://about.gitlab.com/blog/2020/09/01/using-bazel-to-spee... [2]: https://docs.earthly.dev/examples/ci-examples/gh-actions-integration https://docs.earthly.dev/examples/ci-examples/gh-actions-int...

agbell 5y ago
I agree that build systems and CI/CD systems should converge. One way this could happen is considering one side, say the Bazel build or the earthfile canonical, and generating the config for the other side from it, including various sub-steps and so on. Or Bazel or an Earthfile could be the basis for a CI system, but I think things can go further than that. Building locally and building in CI should converge, and building on a single machine vs. building across a cluster of machines should also converge. I'd love to hear other's thoughts or ideas on this. We are thinking about this as well. There is a lot of innovation to come in this space.
- theptip 5y ago
  Good point. Looking at this, it seems that maybe you could build Earthly=>Gitlab with the current GL API. With "parent/child pipelines" [1] you can have step 1 generate the GL YAML for the rest of the graph, and then step 2 run that YAML. One thing that might be needed on the Earthly side is some way to run only the named job, and fail if the deps aren't built ("pipeline mode"?) so that the downstream steps don't re-run the upstream ones. But feels like the model you're considering is actually close. Anyway, glad to get validation that others are thinking this way, and I'll follow your project to see where you guys get to! [1]: https://docs.gitlab.com/ee/ci/pipelines/parent_child_pipelines.html https://docs.gitlab.com/ee/ci/pipelines/parent_child_pipelin...
wikibob 5y ago
99.9% of what is called "CI/CD" systems are NOT doing Continuous Integration. Great article here on "CBOI: Continuous Build, Occasional Integration": https://www.aspect.dev/blog/cboi-continuous-build-occasional-integration https://www.aspect.dev/blog/cboi-continuous-build-occasional...
- adonovan 5y ago
  Another great article on this theme: https://gregoryszorc.com/blog/2021/04/07/modern-ci-is-too-complex-and-misdirected/ https://gregoryszorc.com/blog/2021/04/07/modern-ci-is-too-co... It is frustrating that we seem to be evolving towards the nesting of one build system inside another, such as "go build" inside "docker build". (One of the projects I work on uses janky, docker compose, docker build, Go build, Make, Rake, and Bundle.) This approach is opaque, coarse-grained, inefficient, unnecessarily sequential, and doesn't materialize "the" build graph as a single entity from which we can identify our software artifacts and their relationships and derive and automate all our workflows. Bazel addresses these problems with a single, canonical, first-class graph, and many projects that use Bazel achieve close-to-ideal build efficiency, scalability, and reproducibility, as well as a good foundation for other workflows. But switching to Bazel is a large undertaking for a project that has already become entangled in the kind of situation described above. Maintaining hand-written or generated BUILD files for dependencies can be a burden. (I led the design of Blaze in 2006, at which time Google was already using BUILD files to declare software artifacts and dependencies in its more primitive predecessor system; we simply could not have achieved Blaze even at Google's then scale otherwise.) I often dream of a future world in which it is expected that compiler writers will provide a formal definition of the toolchain's workflow in a form that allows rule-sets for Bazel and its successors to be derived. So there is a place for tools like Earthly that try to retrofit some of the goodness of pure functional builds onto the mess of Docker files, whose natural abstraction is not "consume inputs, produce outputs", but "mutate the file system", which doesn't compose. By strange coincidence I had just started playing last week---a mandatory vacation at my employer---with a proof-of-concept of a tool just like Earthly, so the timing of this announcement couldn't be better.

agbell 5y ago

I agree that build systems and CI/CD systems should converge. One way this could happen is considering one side, say the Bazel build or the earthfile canonical, and generating the config for the other side from it, including various sub-steps and so on. Or Bazel or an Earthfile could be the basis for a CI system, but I think things can go further than that. Building locally and building in CI should converge, and building on a single machine vs. building across a cluster of machines should also converge. I'd love to hear other's thoughts or ideas on this. We are thinking about this as well. There is a lot of innovation to come in this space.

theptip 5y ago
Good point. Looking at this, it seems that maybe you could build Earthly=>Gitlab with the current GL API. With "parent/child pipelines" [1] you can have step 1 generate the GL YAML for the rest of the graph, and then step 2 run that YAML. One thing that might be needed on the Earthly side is some way to run only the named job, and fail if the deps aren't built ("pipeline mode"?) so that the downstream steps don't re-run the upstream ones. But feels like the model you're considering is actually close. Anyway, glad to get validation that others are thinking this way, and I'll follow your project to see where you guys get to! [1]: https://docs.gitlab.com/ee/ci/pipelines/parent_child_pipelines.html https://docs.gitlab.com/ee/ci/pipelines/parent_child_pipelin...

wikibob 5y ago

99.9% of what is called "CI/CD" systems are NOT doing Continuous Integration. Great article here on "CBOI: Continuous Build, Occasional Integration": https://www.aspect.dev/blog/cboi-continuous-build-occasional-integration https://www.aspect.dev/blog/cboi-continuous-build-occasional...

adonovan 5y ago
Another great article on this theme: https://gregoryszorc.com/blog/2021/04/07/modern-ci-is-too-complex-and-misdirected/ https://gregoryszorc.com/blog/2021/04/07/modern-ci-is-too-co... It is frustrating that we seem to be evolving towards the nesting of one build system inside another, such as "go build" inside "docker build". (One of the projects I work on uses janky, docker compose, docker build, Go build, Make, Rake, and Bundle.) This approach is opaque, coarse-grained, inefficient, unnecessarily sequential, and doesn't materialize "the" build graph as a single entity from which we can identify our software artifacts and their relationships and derive and automate all our workflows. Bazel addresses these problems with a single, canonical, first-class graph, and many projects that use Bazel achieve close-to-ideal build efficiency, scalability, and reproducibility, as well as a good foundation for other workflows. But switching to Bazel is a large undertaking for a project that has already become entangled in the kind of situation described above. Maintaining hand-written or generated BUILD files for dependencies can be a burden. (I led the design of Blaze in 2006, at which time Google was already using BUILD files to declare software artifacts and dependencies in its more primitive predecessor system; we simply could not have achieved Blaze even at Google's then scale otherwise.) I often dream of a future world in which it is expected that compiler writers will provide a formal definition of the toolchain's workflow in a form that allows rule-sets for Bazel and its successors to be derived. So there is a place for tools like Earthly that try to retrofit some of the goodness of pure functional builds onto the mess of Docker files, whose natural abstraction is not "consume inputs, produce outputs", but "mutate the file system", which doesn't compose. By strange coincidence I had just started playing last week---a mandatory vacation at my employer---with a proof-of-concept of a tool just like Earthly, so the timing of this announcement couldn't be better.