5 ms·

> It is much harder to set up caching/parallelization with GitHub Actions, Jenkins, Docker, and Make than it is with just Earthly I never used Earthly but I ha

by rewmie 3y ago

> It is much harder to set up caching/parallelization with GitHub Actions, Jenkins, Docker, and Make than it is with just Earthly I never used Earthly but I have to call nonsense on this take. Caching and parallelization is a solved problem in GitLab, GitHub, CircleCI, etc. It's so trivial that it's pipelines 101. Caching and parallelization might be easier in Earthly but any improvement over "trivial" is something not worth wasting time over.

shepherdjerred 3y ago
How do you usually do caching for complex, multi-language projects in this case? How do you guarantee _only_ changed targets are rebuilt when dependencies, like a language version or external library, change? With Earthly you just... write your Earthfile. You get everything for free. Caching and parallelization _also_ works locally, so I see that speedup in development.
- howinteresting 3y ago
  I think there is certainly some value to that (the other posts don't appear to be fully grasping the complexity of the problem), but Earthly feels closer to an open source consultancy project than venture capital funded startup.
  - shepherdjerred 3y ago
    Yeah, I'm a little lost on how the value proposition of Earthly isn't being understood by others. I'm guessing that many of the posters don't understand that Earthly works locally.
- Too 3y ago
  Dependency graph is a feature of the build system with better understanding of the language import rules than the CI system can have (or whatever terminology you want to use). Good ones can hermetically, concurrently and deterministic build only the files that were changed. Earthly is just a thin wrapper on top of docker multi stage builds[1]. Such docker layers are one-dimensional, meaning if one of the steps are calling “make”, that step will have to rebuild that entire layer, not just the one c++ file that changed. There are solutions to this with --mount,type=cache but they take away the reproducibility and and are equally possible with plain dockerfiles. [1] tbh I don’t know if this is the case, but that’s how others explain it. This is core problem with Earthly. They don’t say how it works, just that you will magically get 20x faster. Without understanding how, I don’t know how to take that seriously and I can only assume it’s snake oil. I mean, there is a need for a better Dockerfile syntax. But then market it as such! Not as magic 20x faster builds.
- rewmie 3y ago
  > How do you usually do caching for complex, multi-language projects in this case? As a software engineer, I do my best to keep my systems as simple as possible. Each build stage either caches artifacts or delivers deployable packages, and they consume dependencies that are either built in each pipeline run or already delivered through other means. > How do you guarantee _only_ changed targets are rebuilt when dependencies, Why do you want that? Other than saving a few pennies here and there in pipeline runs, you do not have anything to gain. Your builds should consume dependencies already packaged and delivered, either from a OS image, your own container image, or a package deployed by you or a third party, and your build must be reproducible and tied to each and every pipeline run. > like a language version or external library, change? If you're mindlessly bumping up language versions and external libraries, you have more pressing problems with your setup than what's supported by the CICD system you're using.
code_biologist 3y ago
This unanswered Github Actions community request for parallel steps support drives me up the wall: https://github.com/orgs/community/discussions/14484?sort=top https://github.com/orgs/community/discussions/14484?sort=top It can be papered over, but intelligibility by more junior engineers is one of the reasons I'd opt for GHA over something like Jenkins.

by rewmie 3y ago

shepherdjerred 3y ago

How do you usually do caching for complex, multi-language projects in this case? How do you guarantee _only_ changed targets are rebuilt when dependencies, like a language version or external library, change? With Earthly you just... write your Earthfile. You get everything for free. Caching and parallelization _also_ works locally, so I see that speedup in development.

howinteresting 3y ago
I think there is certainly some value to that (the other posts don't appear to be fully grasping the complexity of the problem), but Earthly feels closer to an open source consultancy project than venture capital funded startup.
- shepherdjerred 3y ago
  Yeah, I'm a little lost on how the value proposition of Earthly isn't being understood by others. I'm guessing that many of the posters don't understand that Earthly works locally.
Too 3y ago
Dependency graph is a feature of the build system with better understanding of the language import rules than the CI system can have (or whatever terminology you want to use). Good ones can hermetically, concurrently and deterministic build only the files that were changed. Earthly is just a thin wrapper on top of docker multi stage builds[1]. Such docker layers are one-dimensional, meaning if one of the steps are calling “make”, that step will have to rebuild that entire layer, not just the one c++ file that changed. There are solutions to this with --mount,type=cache but they take away the reproducibility and and are equally possible with plain dockerfiles. [1] tbh I don’t know if this is the case, but that’s how others explain it. This is core problem with Earthly. They don’t say how it works, just that you will magically get 20x faster. Without understanding how, I don’t know how to take that seriously and I can only assume it’s snake oil. I mean, there is a need for a better Dockerfile syntax. But then market it as such! Not as magic 20x faster builds.
rewmie 3y ago
> How do you usually do caching for complex, multi-language projects in this case? As a software engineer, I do my best to keep my systems as simple as possible. Each build stage either caches artifacts or delivers deployable packages, and they consume dependencies that are either built in each pipeline run or already delivered through other means. > How do you guarantee _only_ changed targets are rebuilt when dependencies, Why do you want that? Other than saving a few pennies here and there in pipeline runs, you do not have anything to gain. Your builds should consume dependencies already packaged and delivered, either from a OS image, your own container image, or a package deployed by you or a third party, and your build must be reproducible and tied to each and every pipeline run. > like a language version or external library, change? If you're mindlessly bumping up language versions and external libraries, you have more pressing problems with your setup than what's supported by the CICD system you're using.

code_biologist 3y ago

This unanswered Github Actions community request for parallel steps support drives me up the wall: https://github.com/orgs/community/discussions/14484?sort=top https://github.com/orgs/community/discussions/14484?sort=top It can be papered over, but intelligibility by more junior engineers is one of the reasons I'd opt for GHA over something like Jenkins.