4 ms·

Perusing this article quickly, it means a CI that is automatically handling things like caching build artifacts so that you don't need to recompile your entire

Perusing this article quickly, it means a CI that is automatically handling things like caching build artifacts so that you don't need to recompile your entire repository every single commit. It's not about a faster program to call exec; it's about a program that knows it need not even call exec.

aidos 3y ago
I looked at Earthly just a couple of weeks back. It’s a tool that makes perfect sense - especially when I was deep in a makefile wishing the caching was as automated as docker. In the end I decided that I couldn’t face the complexity of dealing with someone else’s hosted tooling. Luckily in this case.
alexvitkov 3y ago
Incremental builds are something even enterprise build systems fail at. I've had Visual Studio and MSBuild fail on me multiple times by deciding not to recompile a certain .cpp file, because they got the dependency graph wrong. This results in a running but inconsistent executable that's a bitch to debug. If you're selling me another tool that duplicates this work, but that has even less context on my project than the build system, I'll pass.
- no_wizard 3y ago
  One thing these tools should do automatically is at the end log to the console (or whatever system) what was compiled and what wasn't. This made debugging so much easier for these issues. They're bound to happen, because we're all human and bugs are a thing. Automatically being able to audit the build as step 1 makes this easy to spot
  - jcparkyn 3y ago
    Visual studio does give a fairly detailed explanation of what was compiled each time, in the output>build view.
- pharmakom 3y ago
  Bazel does this. I have to be honest - MS Build does not scale well. Companies use it because it’s the default.
  - no_wizard 3y ago
    Seems they came up with a way to scale up build toolchains with BuildXL[0] Doesn't seem fully baked yet though [0]: https://github.com/microsoft/BuildXL https://github.com/microsoft/BuildXL
  - CobsterLock 3y ago
    I don't have a ton of experience in Bazel, but from what I have experienced with Bazel I think it would be more accurate to say that you configure the dependency graph yourself in Bazel. Then Bazel can determine what to build from scratch and what to take from cache
  - howinteresting 3y ago
    Yes, but Bazel requires an all-in commitment and also tons of work to fit the rest of the world into it.
theamk 3y ago
except not really.. their tutorial [0] includes no-compile languages like python and JS, with only non-trivial compilation being and Java... and how do they do it? RUN gradle build At least in this case, Earthly has no insight into how repository is organized, and it _will_ recompile the entire repository with every single commit. So what's the real value of it? this basically seems like a better "docker build" alternative, giving nicer inter-layer caching. Maybe will save some time on dependencies if they take some time to install and devs were too lazy to set up own workers and used ephemeral runners instead. [0] https://docs.earthly.dev/basics/part-1-a-simple-earthfile https://docs.earthly.dev/basics/part-1-a-simple-earthfile
- danudey 3y ago
  That's step one, 'how to write a basic Earthfile that just does the thing', so they don't delve into the details of caching and how it works, but look at the whole block and not just that one line; it operates like a Dockerfile: build: COPY build.gradle ./ COPY src src RUN gradle build RUN gradle install SAVE ARTIFACT build/install/java-example/bin /bin SAVE ARTIFACT build/install/java-example/lib /lib The COPY phases copy files into the build context. If the files haven't changed, then the commands don't need to be run. In other words, if `build.gradle` and `src` are unchanged, it won't run `gradle build` or `gradle install`, and it'll just give you the artifacts from the previous build. They have a golang example in part three[0], which they redesign to use caching effectively: build: # Download deps before copying code. COPY go.mod go.sum . RUN go mod download # Copy and build code. COPY main.go . RUN go build -o output/example main.go SAVE ARTIFACT output/example AS LOCAL local-output/go-example They copy in go.mod and go.sum and then run `go mod download`. If the mod and sum files haven't changed, then `go mod download` doesn't need to be run. Then they copy `main.go` in, and run `go build`. If the `main.go` file hasn't changed, `go build` doesn't need to be run. [0] https://docs.earthly.dev/basics/part-3-adding-dependencies-with-caching https://docs.earthly.dev/basics/part-3-adding-dependencies-w...
  - brabel 3y ago
    Gradle can do the caching of tasks itself in a level of granularity no CI can. If you keep your build cache "live" in the CI, Gradle knows to only recompile what has changed, and to only run tests which changed (or tests something which changed). I am not sure if many other build systems do that, but this is something I would absolutely expect from the build system, not CI. Do you not expect caching when building locally?? The whole idea with CI doing this is wrong IMO. All I want CI to do is call a couple of commands after setting up the environment (which would include the Gradle cache or whatever to speed up things - no need for your revolutionary CI build system).
adamgordonbell 3y ago
Yes, exactly. Earthly is explicit about inputs ( using COPY) and outputs of each build step. This let's it be 100% certain about whether it can cache parts of the build. If you are familiar with docker layer-based caching and Bazel, you can imagine how Earthly works to eliminate rework in builds. Here is a small monorepo example: https://github.com/earthly/earthly-solutions https://github.com/earthly/earthly-solutions
dgb23 3y ago
Isn't this a complex solution for a problem that shouldn't exist?

aidos 3y ago

I looked at Earthly just a couple of weeks back. It’s a tool that makes perfect sense - especially when I was deep in a makefile wishing the caching was as automated as docker. In the end I decided that I couldn’t face the complexity of dealing with someone else’s hosted tooling. Luckily in this case.

alexvitkov 3y ago

Incremental builds are something even enterprise build systems fail at. I've had Visual Studio and MSBuild fail on me multiple times by deciding not to recompile a certain .cpp file, because they got the dependency graph wrong. This results in a running but inconsistent executable that's a bitch to debug. If you're selling me another tool that duplicates this work, but that has even less context on my project than the build system, I'll pass.

no_wizard 3y ago
One thing these tools should do automatically is at the end log to the console (or whatever system) what was compiled and what wasn't. This made debugging so much easier for these issues. They're bound to happen, because we're all human and bugs are a thing. Automatically being able to audit the build as step 1 makes this easy to spot
- jcparkyn 3y ago
  Visual studio does give a fairly detailed explanation of what was compiled each time, in the output>build view.
pharmakom 3y ago
Bazel does this. I have to be honest - MS Build does not scale well. Companies use it because it’s the default.
- no_wizard 3y ago
  Seems they came up with a way to scale up build toolchains with BuildXL[0] Doesn't seem fully baked yet though [0]: https://github.com/microsoft/BuildXL https://github.com/microsoft/BuildXL
- CobsterLock 3y ago
  I don't have a ton of experience in Bazel, but from what I have experienced with Bazel I think it would be more accurate to say that you configure the dependency graph yourself in Bazel. Then Bazel can determine what to build from scratch and what to take from cache
- howinteresting 3y ago
  Yes, but Bazel requires an all-in commitment and also tons of work to fit the rest of the world into it.

theamk 3y ago

except not really.. their tutorial [0] includes no-compile languages like python and JS, with only non-trivial compilation being and Java... and how do they do it? RUN gradle build At least in this case, Earthly has no insight into how repository is organized, and it _will_ recompile the entire repository with every single commit. So what's the real value of it? this basically seems like a better "docker build" alternative, giving nicer inter-layer caching. Maybe will save some time on dependencies if they take some time to install and devs were too lazy to set up own workers and used ephemeral runners instead. [0] https://docs.earthly.dev/basics/part-1-a-simple-earthfile https://docs.earthly.dev/basics/part-1-a-simple-earthfile

danudey 3y ago
That's step one, 'how to write a basic Earthfile that just does the thing', so they don't delve into the details of caching and how it works, but look at the whole block and not just that one line; it operates like a Dockerfile: build: COPY build.gradle ./ COPY src src RUN gradle build RUN gradle install SAVE ARTIFACT build/install/java-example/bin /bin SAVE ARTIFACT build/install/java-example/lib /lib The COPY phases copy files into the build context. If the files haven't changed, then the commands don't need to be run. In other words, if `build.gradle` and `src` are unchanged, it won't run `gradle build` or `gradle install`, and it'll just give you the artifacts from the previous build. They have a golang example in part three[0], which they redesign to use caching effectively: build: # Download deps before copying code. COPY go.mod go.sum . RUN go mod download # Copy and build code. COPY main.go . RUN go build -o output/example main.go SAVE ARTIFACT output/example AS LOCAL local-output/go-example They copy in go.mod and go.sum and then run `go mod download`. If the mod and sum files haven't changed, then `go mod download` doesn't need to be run. Then they copy `main.go` in, and run `go build`. If the `main.go` file hasn't changed, `go build` doesn't need to be run. [0] https://docs.earthly.dev/basics/part-3-adding-dependencies-with-caching https://docs.earthly.dev/basics/part-3-adding-dependencies-w...
- brabel 3y ago
  Gradle can do the caching of tasks itself in a level of granularity no CI can. If you keep your build cache "live" in the CI, Gradle knows to only recompile what has changed, and to only run tests which changed (or tests something which changed). I am not sure if many other build systems do that, but this is something I would absolutely expect from the build system, not CI. Do you not expect caching when building locally?? The whole idea with CI doing this is wrong IMO. All I want CI to do is call a couple of commands after setting up the environment (which would include the Gradle cache or whatever to speed up things - no need for your revolutionary CI build system).

adamgordonbell 3y ago

Yes, exactly. Earthly is explicit about inputs ( using COPY) and outputs of each build step. This let's it be 100% certain about whether it can cache parts of the build. If you are familiar with docker layer-based caching and Bazel, you can imagine how Earthly works to eliminate rework in builds. Here is a small monorepo example: https://github.com/earthly/earthly-solutions https://github.com/earthly/earthly-solutions

dgb23 3y ago

Isn't this a complex solution for a problem that shouldn't exist?