10 ms·

Show HN: Earthly – Better Builds

223 points by adamgordonbell 5y ago

agbell 5y ago
Hello HN. Earthly is a tool for build automation. I like to think of our solution as a modern version of a makefile. Many people are familiar with makefiles, and back before every CI server was configured using YAML and groovy, this was a common way to build things. We are trying to improve the simplicity of make by adding on the repeatability of docker containers. By mixing those two concepts together, a lot of cool properties fall out. Because each target is containerized, there is no works-on-my-machine issues, or tests failing because something is not installed on one Jenkins runner or so on. Second, that isolation means that dependencies between steps need to be explicitly declared, and with this knowledge, earthly can infer parallelism. Third, containerization gives caching abilities. We can tell if nothing’s changed since a previous run, and earthly can cache steps across build runs or even across machines. You get to use a docker registry as a build cache. We shared this on here about a year ago[1], and since that, lots has changed. First of all, a year ago it was just Vlad working on this, and now there is a number of us, including me. But for end-users, we have shipped many great features such as multi-arch builds, docker-compose support (which is handy for running integration tests ), inline and explicit caching, and an interactive debug mode. Interactive debug mode is neat but straightforward: a failing build step pops you into a shell at the point of failure. Let me know what you think of what we have built. [1] https://news.ycombinator.com/item?id=22890612 https://news.ycombinator.com/item?id=22890612
thdxr 5y ago
Earthly is great! Finally makes it so CI runs the same everywhere. I try to use it wherever I can
- agbell 5y ago
  Thanks for using it! Yeah, I was sick of having to write build scripts in vendor specific formats and then not being able to run those builds locally without jumping through a lot of hoops. If earthly didn't exist I think I would use a combination of make inside docker files to define my build and then use the bare minimum of CI specific configs to just call into those. That way when I need to switch from Travis to Github or Circle or whatever, I don't need to rewrite the steps.
emptysongglass 5y ago
I don't get it: why wouldn't I just use a Dockerfile? The Earthfiles look the same. Dockerfiles run the same way every time and Docker supports caching. I just see more complexity on top without value add. If I want to move beyond local builds then I'd use something like Garden or Skaffold. Skaffold builds the same way every time and supports remote builders.
- agbell 5y ago
  You can take a look at how the phoenix framework uses earthly to simplify their builds. They are testing across OTP versions and across different databases (mysql, SQL Server, postgres) all in an earthfile[1]. Ecto_sql does something similar[2]. [1] https://phoenixframework.org/blog/improving-testing-and-continuous-integration-in-phoenix https://phoenixframework.org/blog/improving-testing-and-cont... [2] https://github.com/elixir-ecto/ecto_sql/blob/master/Earthfile https://github.com/elixir-ecto/ecto_sql/blob/master/Earthfil...
- ithkuil 5y ago
  The Lansing page says: earthfile = makefile + dockerfile There is also an easy to understand example with several targets with dependencies between them. You can more or less achieve the same with multiple stages in a docker file, with named targets etc, but it get complicated fast.
  - ossusermivami 5y ago
    indeed looks a lot better than pure dockerfile
- deleted 5y ago
  [deleted]
petethepig 5y ago
I'm working on our CI/CD right now and this looks like it would made my life easier. Killer feature for me here would be if you could run the same Earthfile on a mac or a windows machine.
- agbell 5y ago
  Thanks for checking it out. You can totally do that. We are using moby/buildkit and docker to provide this platform independence.
  - corpdronejuly 5y ago
    Even better would be support for Windows instead of WSL. I've really come to be frustrated with the a pattern where developers treat "it builds on Ubuntu LTS WSL2" as the same thing as "it works on windows" Could I do something like run one build on a windows CI host and also test that things behave the same way on linux and mac.
    - mamcx 5y ago
      Yeah, I use azure so I can get build support for linux, windows, osx. I wish I could simplify all that, but with windows I need call native C++ toolchain (using rust).
    - vladaionescu 5y ago
      Earthly team member here. Yes, we're working on native windows support as we speak. This has been highly requested.
      - corpdronejuly 5y ago
        Glad to hear it!
    - deleted 5y ago
      [deleted]
AlessandroA 5y ago
This looks very interesting. I couldn't see TeamCity mentioned in the CI integration section. Any plan to support it?
- agbell 5y ago
  Thanks for checking it out. I've not tried this and I'm not very familiar with TeamCity but it should just work. If you are running the teamcity builds inside of containers, you may need to switch that off, as Earthly brings its own containerization.
escot 5y ago
This seems like it could be better than Docker when you need to manage dependent images. Like if I want to build a simple base image for my backend that is then used to build images for both a prod deployment, a testing image, etc.
- verdverm 5y ago
  It's built on Docker's BuildKit which you can also enable with some config twerking (lol, not fixing this spell check because it's probably more accurate). That's how you get caching and DAG solving. BuildKit doesn't have a great wrapper or interface a.t.m. Earthly is one of several companies building on BuildKit and providing a better DX
geekdave 5y ago
Does this work with GitHub Actions?
- agbell 5y ago
  It does. You just put all your actually build logic in an earthfile and call it from your github actions. A benefit of this is its easy to move CIs and you can test your build locally. Here is an example githubaction script[1]. https://github.com/phoenixframework/phoenix/blob/master/.github/workflows/ci.yml https://github.com/phoenixframework/phoenix/blob/master/.git...
- quickthrower2 5y ago
  Same question for Travis CI, is there an example setup for this?
  - agbell 5y ago
    So you just have to pull in earthly and then call it. Here is an example. Note that the travis file has almost nothing, because you can move all that to an Earthfile so that you can easily port it and run it locally. https://github.com/zedtux/database_cleaner-nobrainer/blob/master/.travis.yml https://github.com/zedtux/database_cleaner-nobrainer/blob/ma... https://github.com/zedtux/database_cleaner-nobrainer/blob/master/Earthfile https://github.com/zedtux/database_cleaner-nobrainer/blob/ma...
n3t 5y ago
How would you compare Earthly to creating Docker images using Nix[0]? I love Nix declarative nature with strong emphasis on reproducibility. [0]: https://nixos.org/guides/building-and-running-docker-images.html https://nixos.org/guides/building-and-running-docker-images....
- agbell 5y ago
  I like nix. If you dig into nixpkgs far enough back you can find commits from me bumping version hashes on postgres. The caching and our future plans for distributed builds have some similarities with Nix Hydra and Bazel but creating images in earthly is closer to docker multistage builds than it is to nix. We are not just for building docker images, though. In the real world a 'build' is more than just building an image. It will have linting, and tests and integration tests and maybe some services need to be stood up and tore down in between to make sure an api contract wasn't broken and you might need different binaries generated for different architectures and so on. I could be wrong but I think nix is focused more on just the producing the artifact part. I do love how they approach every build as a pure function though.
  - n3t 5y ago
    It makes sense! Thank you for your response.
  - mikepurvis 5y ago
    Nix does have some limited support for a "check" phase, but I think it's meant to be for an ultra-quick Homebrew style smoke test, basically just confirming that `thing-bin --version` writes to stdout and returns 0, rather than something usable for larger scope integration.
  - gravypod 5y ago
    Do you plan on having some support for hermetic and reproducability? That's the main thing Bazel brings to the table and I'd like to see some solutions that can scale to hundreds of services like that
gingerlime 5y ago
Great to see earthly alive and kicking. Was in touch with Vlad nearly a year ago, and my main blocker to using it was longevity of the project in quite a saturated space. Perhaps it’s time to take a closer look :) It looks great!
quickthrower2 5y ago
I love the idea. Yesterday I gave up building something locally and just got CI to do it because to run locally you have to kind of make a poor mans .travis.yml file out of bash locally then play spot the difference (I never figured out what was different that was breaking it locally, resulting in a k8s deployment with a missing mount/volume). This might sound weird but when you inherit some legacy code you’ll know what I mean. Something where you can run the CI locally is such a boon. Also CI providers do have downtime. If you can run locally, you can’t deploy your app!
picardo 5y ago
I've been wishing for something like this. Can I generate AWS CodeBuild buildspec.yml files from an Earthfile?
- agbell 5y ago
  Not right now, but we are happy to take a PR :) You can probably just manually write a buildspec that has a single earthly call: command: - earthly +build I've not used CodeBuild though, so I'm not totally sure. Long term, the idea of generating the config for N possible build systems from Earthly is something we want to explore but we aren't there yet.
  - picardo 5y ago
    After a cursory look, it seems like there is no support for parallel and dependent builds in the syntax right now. Is this correct? Are there any plans to support that? That's the advantage of CodeBuild spec files –– they let me specify build dependency and order, but I have to write a file for each job. I'd rather write a single file with a simple syntax like the Earthfile.
    - agbell 5y ago
      Hi, just on my phone but Dependencies and parrellism are supported. It's inferred from the structure of targets.
      - picardo 5y ago
        Ah, I see. That's pretty cool. But all builds are run on the same machine, which is not a deal breaker; it'd be nice to distribute them on multiple machines, as well.
        agbell 5y ago
        Thanks for checking it out. This is something we have plans for.
mdaniel 5y ago
I would value hearing more about the business risk that going BSL was designed to address: https://github.com/earthly/earthly/blob/main/LICENSE https://github.com/earthly/earthly/blob/main/LICENSE the associated PR just says "doing it" without saying why https://github.com/earthly/earthly/pull/730 https://github.com/earthly/earthly/pull/730 I have a very hard time imagining "AWS Infinibuild(R)" that eats the "makefile + dockerfile that builds locally" use case, which is why I was curious to learn more --- Separately, I would value GitHub's license detector gizmo using some of that sweet machine-learning sauce to identify the BSL, since it seems to be the path forward for these situations
- deleted 5y ago
  [deleted]
- captn3m0 5y ago
  GitHub licensee (the detection gizmo) uses ChooseALicense as the source. They recommend filing a PR there, but BSL doesn’t fit the requirements at https://github.com/github/choosealicense.com/blob/gh-pages/CONTRIBUTING.md#adding-a-license https://github.com/github/choosealicense.com/blob/gh-pages/C... https://github.com/licensee/licensee/blob/master/docs/CONTRIBUTING.md https://github.com/licensee/licensee/blob/master/docs/CONTRI...
- agbell 5y ago
  Hi, We have a blog post discussing the license change here: https://earthly.dev/blog/every-open-core-company-should-be-a-source-available-company/ https://earthly.dev/blog/every-open-core-company-should-be-a...
  - mdaniel 5y ago
    Thank you for that; I would suggest putting that link in the PR description (or a comment) since not everyone who is similarly curious will be on a Show HN thread with the authoritative parties I'm not trying to start trouble or get into a massive licensing thread on HN, but I want to point out that 2/5ths of those exemplar links of BSL success are in fact Apache 2.0 licensed, and not just source-available. I believe you that there are (evidently) parts of Kafka which one must pay money for, but my understanding is that the Apache Software Foundation would not tolerate hosting source-available projects on the apache.org domain nor in the apache GitHub organization. I wish you all the best with your project
no_circuit 5y ago
I wonder what the scope of what would be a competitive service now that Earthly's license changed to BSL from MPL on 2021-02-01 [0]? Seems like Earthly Technologies Inc will launch a SaaS [1]. However, I'd assume it won't matter for most developers except for those selling a CI/CD SaaS. In any case, I'm looking forward to seeing what they end up offering! [0] https://github.com/earthly/earthly/pull/730 https://github.com/earthly/earthly/pull/730 [1] https://earthly.dev/tos https://earthly.dev/tos
theptip 5y ago
This looks interesting, thanks for sharing. I've been looking for a solution to some of the problems that this targets; in general our build steps are containerized, but it's annoying to have to layer a `make <step>` command on top of a dependency tree of docker/other commands inside a given CI step, and it's worse to have multi-step commands in the CI/CD config file, making the jobs harder to reproduce. One more general idea I'd be interested in others' thoughts on -- there seems to be an artificial split between build systems (e.g. Bazel) and CI/CD systems (e.g. Gitlab). As these tools become more generic, they tend to overlap almost-completely; ultimately the hard part is DAG management and latency-optimized distributed computation. When you look at "how to use Bazel in Gitlab"[1] or "how to use Earthly in Github Actions"[2], it's basically having a single CI step that calls `buildtool <run my build DAG>`. But there could be other CI/CD steps like "run the e2es", and in general I want multiple steps in my Gitlab CI/CD pipeline so I can break open the individual steps, retry flaky runs, see which step failed, and otherwise get all the nice tight integration that Gitlab affords. Unfortunately I have a DAG in the CI server, and then another DAG in the build job; it really seems like the CI/CD system should somehow unwrap the build-tool's DAG into the CI/CD DAG, then I could rerun individual failed steps etc. (though I think you might perhaps want some folding, because Bazel DAGs are probably in general too fine-grained for the way CI/CD servers tend to think about the visualization problem). Maybe this is just a missing feature in GitLab? It seems like the Github workflow is less well integrated with the CI/CD pipeline (always a click or two to get to the pipeline result details) so maybe users there don't see any missing features. CI/CD systems like Concourse have a really nice model of "artifact revisions as inputs/outputs" which would play really well with this model; different parts of the pipeline accept a hash and will trigger downstream jobs only when those change. I wonder if the Earthly team has any thoughts on integrating with the CI/CD DAG in this way? [1]: https://about.gitlab.com/blog/2020/09/01/using-bazel-to-speed-up-gitlab-ci-builds/ https://about.gitlab.com/blog/2020/09/01/using-bazel-to-spee... [2]: https://docs.earthly.dev/examples/ci-examples/gh-actions-integration https://docs.earthly.dev/examples/ci-examples/gh-actions-int...
- agbell 5y ago
  I agree that build systems and CI/CD systems should converge. One way this could happen is considering one side, say the Bazel build or the earthfile canonical, and generating the config for the other side from it, including various sub-steps and so on. Or Bazel or an Earthfile could be the basis for a CI system, but I think things can go further than that. Building locally and building in CI should converge, and building on a single machine vs. building across a cluster of machines should also converge. I'd love to hear other's thoughts or ideas on this. We are thinking about this as well. There is a lot of innovation to come in this space.
mssundaram 5y ago
> Made with <3 on Planet Earth Does anyone else gag with when companies do this (made with love)? It feels too emotional, but maybe it's really true for them? Edit: Replace the heart character with <3 as HN doesn't seem to render it
choeger 5y ago
Maybe this is a stupid question, but what is the "make" part? I see a lot of docker syntax and a few targets without dependencies. But make is all about dependencies. What kind of make dependencies do you support? What kind of variables, macros, etc?
- agbell 5y ago
  We support dependencies in a number of ways, but are a bit different than make dependencies, because the file system of each step is separate unless you explicitly copy in or depend on other targets. This helps us do things in parallel because you are being explicit about the DAG of the build. Copying or using file from a step causes your step to depend on it. Similarly for a `from +target`. We also have variables, user defined commands and conditional logic. User defined commands are our way to composition. https://docs.earthly.dev/docs/guides/udc https://docs.earthly.dev/docs/guides/udc https://docs.earthly.dev/docs/earthfile#copy https://docs.earthly.dev/docs/earthfile#copy
wikibob 5y ago
Can anyone who's familiar with Earthly explain why somebody would want to use it instead of Bazel?
- netvarun 5y ago
  Same question here too!
- oftenwrong 5y ago
  I had the same thought, and found this: https://github.com/earthly/earthly#how-is-earthly-different-from-bazel https://github.com/earthly/earthly#how-is-earthly-different-...
- thundergolfer 5y ago
  The best summary from them looks to be: > Overall, compared to Bazel, Earthly sacrifices some correctness and reproducibility in favor of significantly better usability and composability with existing open-source technologies Exactly how much correctness is getting sacrificed is pretty important. A key insight of Bazel is that you can’t have speed at scale without correctness. Would be good to have Earthly described within the framework of “Build Systems a la Carte”.
- no_circuit 5y ago
  I don't think it is meant to replace any particular build system like Bazel or make. Rather it is a better dependency glue flavor for Docker image building and testing. Earthly uses Makefile-like targets which can be leveraged through more advanced references [0]... essentially can replace plain "copy --from=" with a new magic strings syntax which also understand git. Build systems like Bazel would be invoked within a container. I don't think the "why" using it instead of Bazel (and its plugins) will become clear until the Earthly SaaS gets announced that is hinted about in their TOS [1]. Build resource pool management, image release management, testing/dev environment definitions, custom URLs to test pull requests as seen elsewhere? Arguably one could say they'd probably be able do the same thing by writing some Skylark and use remote build execution, but that could be close to saying Dropbox wasn't special in a Show HN thread. :-) [0] https://docs.earthly.dev/docs/guides/target-ref https://docs.earthly.dev/docs/guides/target-ref [1] https://earthly.dev/tos https://earthly.dev/tos
einpoklum 5y ago
I looked at their main page and got the impression this tool is not very / not at all useful - except perhaps for some niche I'm not in. > Earthly is a syntax for defining your build. I already have syntax for defining my build. > It works with your existing build system. If I have a build system, or a build system generator, why do I need to redefine my build another way? > With Earthly, all builds are containerized Huh? What? Why? ... That's already a deal breaker. I don't want to be limited to building within containers. > With Earthly, all builds are ... repeatable Isn't that true for any decent build system? > With Earthly, all builds are ... language agnostic An ambiguous statement. Especially since a build process does depend, to some extent, on the language of the source files. > Get repeatable and understandable builds today. (shrug) more marketing talk. > It's like Makefile and Dockerfile had a baby. I don't want Dockerfile in the family, and as for Makefile - it's too low-level. Makefiles get generated these days (and have been for over 20 years for most software packages, IIANM.) > No more git commit -m "try again" . This has never been a problem for me. (Well, ok, if there's network trouble, then downloading things during a build will fail; but Earthly can't magically reconnect a broken network.) > "Earthly is a pragmatic and incremental solution to a thorny problem that all organizations face" Which thorny problem is that? The "non-repeatable builds"? I'm confused.
- fishywang 5y ago
  My thoughts exactly. I guess there's a niche for Mac/Windows developers to be able to run the build system on linux, because that's what their CI system runs (so it might fail for some issues that does not fail on their host machines), but most CI systems cannot run nested containers (because docker-in-docker requires some privileges), and as a result their CI system usually can not run this thing, which totally defeats the purpose.
- verdverm 5y ago
  Earthly is essentially a make/dockerfile syntax for a BuildKit frontend. If you understand and agree that BuildKit has value, than that's the jist of it. Agree with all your points for sure, but people need words on their landing page
- yagizdegirmenci 5y ago
duped 5y ago
How tightly coupled is this to docker? My major gripe is that I legitimately need cross platform build and test (windows, Linux, and macos). I will pay hundreds to thousands of dollars a month to someone that solves these problems (I already do, and release engineers aren't hurting!).
- detaro 5y ago
  Cross-platform support would be my question too, yes.
- agbell 5y ago
  We do have plans for podman and other container runtime support. Here is our roadmap: https://github.com/earthly/earthly/projects/1 https://github.com/earthly/earthly/projects/1
  - duped 5y ago
    A guess my question is more "how tightly coupled is this to Linux containerization technologies" because if the answer is "fundamentally so" then I don't see a legitimate use for it that creates any value. I spend a lot of time and money on build automation because nobody has solved it for cross platform builds, and I see forcing containerization of key elements into backend services as an insane solution to that problem.
    - agbell 5y ago
      It is fundamentally a containerized build technology and people use it for that reason. That might make it not have value for you, buts fundamental to the reason that those who use it do find it valuable.
kortex 5y ago
This is AWESOME! I already have a super container heavy workflow and abuse dockerfiles to do things like this. It's great to be able to generate artifacts in a container and slurp them back out (really tricky with normal dockerfile builds, you need exterior commands). How well does it work with big builds in a single stage that fail? E.g. If I have a 3h CMake build that fails, does it keep the intermediate files in an image somewhere?
- agbell 5y ago
  It's pure target transactional, so it would not keep intermediate files in that case.
icy 5y ago
Maybe don't call Earthly "free and open source" when it clearly is neither -- it is licensed under the Business Source License.
- hitekker 5y ago
  I’m trouble finding the words “free and open source” or “foss” in their marketing material. Rather, it looks like they’re calling Earthly “open core” or “source available” according to https://earthly.dev/blog/every-open-core-company-should-be-a-source-available-company/ https://earthly.dev/blog/every-open-core-company-should-be-a...
  - icy 5y ago
    Landing page, below the 'Get Earthly' buttons (they have a few): > Free and open
    - InitialBP 5y ago
      to be fair "Free and open" is different than "Free and Open Source". Open source has a much stronger connotation than open and I could certainly see calling it open if the source is available for the core of the product.
stonewareslord 5y ago
Earthly has been great to use. My one issue is with docker. I would greatly prefer to use podman but it seems there is no support for it yet. Every issue I have had with earthly is with some weird quirk with Docker I don't have with podman. For the most part I can get around this with setting DOCKER_HOST, but it is still weird to depend on dockerd when I would expect podman to work fine.
- agbell 5y ago
  Thanks for checking it out. We do get a lot of requests for podmad support and have it on our roadmap here: https://github.com/earthly/earthly/projects/1 https://github.com/earthly/earthly/projects/1
PufPufPuf 5y ago
I wonder how well will this play with [VSCode devcontainers](https://code.visualstudio.com/docs/remote/containers https://code.visualstudio.com/docs/remote/containers) -- perhaps one could create a target to build a development image, that extends the CI image by adding development tools? I'll definitely experiment with this, sounds promising.

Hello HN. Earthly is a tool for build automation. I like to think of our solution as a modern version of a makefile. Many people are familiar with makefiles, and back before every CI server was configured using YAML and groovy, this was a common way to build things. We are trying to improve the simplicity of make by adding on the repeatability of docker containers. By mixing those two concepts together, a lot of cool properties fall out. Because each target is containerized, there is no works-on-my-machine issues, or tests failing because something is not installed on one Jenkins runner or so on. Second, that isolation means that dependencies between steps need to be explicitly declared, and with this knowledge, earthly can infer parallelism. Third, containerization gives caching abilities. We can tell if nothing’s changed since a previous run, and earthly can cache steps across build runs or even across machines. You get to use a docker registry as a build cache. We shared this on here about a year ago[1], and since that, lots has changed. First of all, a year ago it was just Vlad working on this, and now there is a number of us, including me. But for end-users, we have shipped many great features such as multi-arch builds, docker-compose support (which is handy for running integration tests ), inline and explicit caching, and an interactive debug mode. Interactive debug mode is neat but straightforward: a failing build step pops you into a shell at the point of failure. Let me know what you think of what we have built. [1] https://news.ycombinator.com/item?id=22890612 https://news.ycombinator.com/item?id=22890612

thdxr 5y ago

Earthly is great! Finally makes it so CI runs the same everywhere. I try to use it wherever I can

agbell 5y ago
Thanks for using it! Yeah, I was sick of having to write build scripts in vendor specific formats and then not being able to run those builds locally without jumping through a lot of hoops. If earthly didn't exist I think I would use a combination of make inside docker files to define my build and then use the bare minimum of CI specific configs to just call into those. That way when I need to switch from Travis to Github or Circle or whatever, I don't need to rewrite the steps.

emptysongglass 5y ago

I don't get it: why wouldn't I just use a Dockerfile? The Earthfiles look the same. Dockerfiles run the same way every time and Docker supports caching. I just see more complexity on top without value add. If I want to move beyond local builds then I'd use something like Garden or Skaffold. Skaffold builds the same way every time and supports remote builders.

agbell 5y ago
You can take a look at how the phoenix framework uses earthly to simplify their builds. They are testing across OTP versions and across different databases (mysql, SQL Server, postgres) all in an earthfile[1]. Ecto_sql does something similar[2]. [1] https://phoenixframework.org/blog/improving-testing-and-continuous-integration-in-phoenix https://phoenixframework.org/blog/improving-testing-and-cont... [2] https://github.com/elixir-ecto/ecto_sql/blob/master/Earthfile https://github.com/elixir-ecto/ecto_sql/blob/master/Earthfil...
ithkuil 5y ago
The Lansing page says: earthfile = makefile + dockerfile There is also an easy to understand example with several targets with dependencies between them. You can more or less achieve the same with multiple stages in a docker file, with named targets etc, but it get complicated fast.
- ossusermivami 5y ago
  indeed looks a lot better than pure dockerfile
deleted 5y ago
[deleted]

petethepig 5y ago

I'm working on our CI/CD right now and this looks like it would made my life easier. Killer feature for me here would be if you could run the same Earthfile on a mac or a windows machine.

agbell 5y ago
Thanks for checking it out. You can totally do that. We are using moby/buildkit and docker to provide this platform independence.
- corpdronejuly 5y ago
  Even better would be support for Windows instead of WSL. I've really come to be frustrated with the a pattern where developers treat "it builds on Ubuntu LTS WSL2" as the same thing as "it works on windows" Could I do something like run one build on a windows CI host and also test that things behave the same way on linux and mac.
  - mamcx 5y ago
    Yeah, I use azure so I can get build support for linux, windows, osx. I wish I could simplify all that, but with windows I need call native C++ toolchain (using rust).
  - vladaionescu 5y ago
    Earthly team member here. Yes, we're working on native windows support as we speak. This has been highly requested.
    - corpdronejuly 5y ago
      Glad to hear it!
  - deleted 5y ago
    [deleted]

AlessandroA 5y ago

This looks very interesting. I couldn't see TeamCity mentioned in the CI integration section. Any plan to support it?

agbell 5y ago
Thanks for checking it out. I've not tried this and I'm not very familiar with TeamCity but it should just work. If you are running the teamcity builds inside of containers, you may need to switch that off, as Earthly brings its own containerization.

escot 5y ago

This seems like it could be better than Docker when you need to manage dependent images. Like if I want to build a simple base image for my backend that is then used to build images for both a prod deployment, a testing image, etc.

verdverm 5y ago
It's built on Docker's BuildKit which you can also enable with some config twerking (lol, not fixing this spell check because it's probably more accurate). That's how you get caching and DAG solving. BuildKit doesn't have a great wrapper or interface a.t.m. Earthly is one of several companies building on BuildKit and providing a better DX

geekdave 5y ago

Does this work with GitHub Actions?

agbell 5y ago
It does. You just put all your actually build logic in an earthfile and call it from your github actions. A benefit of this is its easy to move CIs and you can test your build locally. Here is an example githubaction script[1]. https://github.com/phoenixframework/phoenix/blob/master/.github/workflows/ci.yml https://github.com/phoenixframework/phoenix/blob/master/.git...
quickthrower2 5y ago
Same question for Travis CI, is there an example setup for this?
- agbell 5y ago
  So you just have to pull in earthly and then call it. Here is an example. Note that the travis file has almost nothing, because you can move all that to an Earthfile so that you can easily port it and run it locally. https://github.com/zedtux/database_cleaner-nobrainer/blob/master/.travis.yml https://github.com/zedtux/database_cleaner-nobrainer/blob/ma... https://github.com/zedtux/database_cleaner-nobrainer/blob/master/Earthfile https://github.com/zedtux/database_cleaner-nobrainer/blob/ma...

n3t 5y ago

How would you compare Earthly to creating Docker images using Nix[0]? I love Nix declarative nature with strong emphasis on reproducibility. [0]: https://nixos.org/guides/building-and-running-docker-images.html https://nixos.org/guides/building-and-running-docker-images....

agbell 5y ago
I like nix. If you dig into nixpkgs far enough back you can find commits from me bumping version hashes on postgres. The caching and our future plans for distributed builds have some similarities with Nix Hydra and Bazel but creating images in earthly is closer to docker multistage builds than it is to nix. We are not just for building docker images, though. In the real world a 'build' is more than just building an image. It will have linting, and tests and integration tests and maybe some services need to be stood up and tore down in between to make sure an api contract wasn't broken and you might need different binaries generated for different architectures and so on. I could be wrong but I think nix is focused more on just the producing the artifact part. I do love how they approach every build as a pure function though.
- n3t 5y ago
  It makes sense! Thank you for your response.
- mikepurvis 5y ago
  Nix does have some limited support for a "check" phase, but I think it's meant to be for an ultra-quick Homebrew style smoke test, basically just confirming that `thing-bin --version` writes to stdout and returns 0, rather than something usable for larger scope integration.
- gravypod 5y ago
  Do you plan on having some support for hermetic and reproducability? That's the main thing Bazel brings to the table and I'd like to see some solutions that can scale to hundreds of services like that

gingerlime 5y ago

Great to see earthly alive and kicking. Was in touch with Vlad nearly a year ago, and my main blocker to using it was longevity of the project in quite a saturated space. Perhaps it’s time to take a closer look :) It looks great!

quickthrower2 5y ago

I love the idea. Yesterday I gave up building something locally and just got CI to do it because to run locally you have to kind of make a poor mans .travis.yml file out of bash locally then play spot the difference (I never figured out what was different that was breaking it locally, resulting in a k8s deployment with a missing mount/volume). This might sound weird but when you inherit some legacy code you’ll know what I mean. Something where you can run the CI locally is such a boon. Also CI providers do have downtime. If you can run locally, you can’t deploy your app!

picardo 5y ago

I've been wishing for something like this. Can I generate AWS CodeBuild buildspec.yml files from an Earthfile?

agbell 5y ago
Not right now, but we are happy to take a PR :) You can probably just manually write a buildspec that has a single earthly call: command: - earthly +build I've not used CodeBuild though, so I'm not totally sure. Long term, the idea of generating the config for N possible build systems from Earthly is something we want to explore but we aren't there yet.
- picardo 5y ago
  After a cursory look, it seems like there is no support for parallel and dependent builds in the syntax right now. Is this correct? Are there any plans to support that? That's the advantage of CodeBuild spec files –– they let me specify build dependency and order, but I have to write a file for each job. I'd rather write a single file with a simple syntax like the Earthfile.
  - agbell 5y ago
    Hi, just on my phone but Dependencies and parrellism are supported. It's inferred from the structure of targets.
    - picardo 5y ago
      Ah, I see. That's pretty cool. But all builds are run on the same machine, which is not a deal breaker; it'd be nice to distribute them on multiple machines, as well.
      - agbell 5y ago
        Thanks for checking it out. This is something we have plans for.

mdaniel 5y ago

I would value hearing more about the business risk that going BSL was designed to address: https://github.com/earthly/earthly/blob/main/LICENSE https://github.com/earthly/earthly/blob/main/LICENSE the associated PR just says "doing it" without saying why https://github.com/earthly/earthly/pull/730 https://github.com/earthly/earthly/pull/730 I have a very hard time imagining "AWS Infinibuild(R)" that eats the "makefile + dockerfile that builds locally" use case, which is why I was curious to learn more --- Separately, I would value GitHub's license detector gizmo using some of that sweet machine-learning sauce to identify the BSL, since it seems to be the path forward for these situations

deleted 5y ago
[deleted]
captn3m0 5y ago
GitHub licensee (the detection gizmo) uses ChooseALicense as the source. They recommend filing a PR there, but BSL doesn’t fit the requirements at https://github.com/github/choosealicense.com/blob/gh-pages/CONTRIBUTING.md#adding-a-license https://github.com/github/choosealicense.com/blob/gh-pages/C... https://github.com/licensee/licensee/blob/master/docs/CONTRIBUTING.md https://github.com/licensee/licensee/blob/master/docs/CONTRI...
agbell 5y ago
Hi, We have a blog post discussing the license change here: https://earthly.dev/blog/every-open-core-company-should-be-a-source-available-company/ https://earthly.dev/blog/every-open-core-company-should-be-a...
- mdaniel 5y ago
  Thank you for that; I would suggest putting that link in the PR description (or a comment) since not everyone who is similarly curious will be on a Show HN thread with the authoritative parties I'm not trying to start trouble or get into a massive licensing thread on HN, but I want to point out that 2/5ths of those exemplar links of BSL success are in fact Apache 2.0 licensed, and not just source-available. I believe you that there are (evidently) parts of Kafka which one must pay money for, but my understanding is that the Apache Software Foundation would not tolerate hosting source-available projects on the apache.org domain nor in the apache GitHub organization. I wish you all the best with your project

no_circuit 5y ago

I wonder what the scope of what would be a competitive service now that Earthly's license changed to BSL from MPL on 2021-02-01 [0]? Seems like Earthly Technologies Inc will launch a SaaS [1]. However, I'd assume it won't matter for most developers except for those selling a CI/CD SaaS. In any case, I'm looking forward to seeing what they end up offering! [0] https://github.com/earthly/earthly/pull/730 https://github.com/earthly/earthly/pull/730 [1] https://earthly.dev/tos https://earthly.dev/tos

theptip 5y ago

This looks interesting, thanks for sharing. I've been looking for a solution to some of the problems that this targets; in general our build steps are containerized, but it's annoying to have to layer a `make <step>` command on top of a dependency tree of docker/other commands inside a given CI step, and it's worse to have multi-step commands in the CI/CD config file, making the jobs harder to reproduce. One more general idea I'd be interested in others' thoughts on -- there seems to be an artificial split between build systems (e.g. Bazel) and CI/CD systems (e.g. Gitlab). As these tools become more generic, they tend to overlap almost-completely; ultimately the hard part is DAG management and latency-optimized distributed computation. When you look at "how to use Bazel in Gitlab"[1] or "how to use Earthly in Github Actions"[2], it's basically having a single CI step that calls `buildtool <run my build DAG>`. But there could be other CI/CD steps like "run the e2es", and in general I want multiple steps in my Gitlab CI/CD pipeline so I can break open the individual steps, retry flaky runs, see which step failed, and otherwise get all the nice tight integration that Gitlab affords. Unfortunately I have a DAG in the CI server, and then another DAG in the build job; it really seems like the CI/CD system should somehow unwrap the build-tool's DAG into the CI/CD DAG, then I could rerun individual failed steps etc. (though I think you might perhaps want some folding, because Bazel DAGs are probably in general too fine-grained for the way CI/CD servers tend to think about the visualization problem). Maybe this is just a missing feature in GitLab? It seems like the Github workflow is less well integrated with the CI/CD pipeline (always a click or two to get to the pipeline result details) so maybe users there don't see any missing features. CI/CD systems like Concourse have a really nice model of "artifact revisions as inputs/outputs" which would play really well with this model; different parts of the pipeline accept a hash and will trigger downstream jobs only when those change. I wonder if the Earthly team has any thoughts on integrating with the CI/CD DAG in this way? [1]: https://about.gitlab.com/blog/2020/09/01/using-bazel-to-speed-up-gitlab-ci-builds/ https://about.gitlab.com/blog/2020/09/01/using-bazel-to-spee... [2]: https://docs.earthly.dev/examples/ci-examples/gh-actions-integration https://docs.earthly.dev/examples/ci-examples/gh-actions-int...

agbell 5y ago
I agree that build systems and CI/CD systems should converge. One way this could happen is considering one side, say the Bazel build or the earthfile canonical, and generating the config for the other side from it, including various sub-steps and so on. Or Bazel or an Earthfile could be the basis for a CI system, but I think things can go further than that. Building locally and building in CI should converge, and building on a single machine vs. building across a cluster of machines should also converge. I'd love to hear other's thoughts or ideas on this. We are thinking about this as well. There is a lot of innovation to come in this space.

mssundaram 5y ago

> Made with <3 on Planet Earth Does anyone else gag with when companies do this (made with love)? It feels too emotional, but maybe it's really true for them? Edit: Replace the heart character with <3 as HN doesn't seem to render it

choeger 5y ago

Maybe this is a stupid question, but what is the "make" part? I see a lot of docker syntax and a few targets without dependencies. But make is all about dependencies. What kind of make dependencies do you support? What kind of variables, macros, etc?

agbell 5y ago
We support dependencies in a number of ways, but are a bit different than make dependencies, because the file system of each step is separate unless you explicitly copy in or depend on other targets. This helps us do things in parallel because you are being explicit about the DAG of the build. Copying or using file from a step causes your step to depend on it. Similarly for a `from +target`. We also have variables, user defined commands and conditional logic. User defined commands are our way to composition. https://docs.earthly.dev/docs/guides/udc https://docs.earthly.dev/docs/guides/udc https://docs.earthly.dev/docs/earthfile#copy https://docs.earthly.dev/docs/earthfile#copy

wikibob 5y ago

Can anyone who's familiar with Earthly explain why somebody would want to use it instead of Bazel?

netvarun 5y ago
Same question here too!
oftenwrong 5y ago
I had the same thought, and found this: https://github.com/earthly/earthly#how-is-earthly-different-from-bazel https://github.com/earthly/earthly#how-is-earthly-different-...
thundergolfer 5y ago
The best summary from them looks to be: > Overall, compared to Bazel, Earthly sacrifices some correctness and reproducibility in favor of significantly better usability and composability with existing open-source technologies Exactly how much correctness is getting sacrificed is pretty important. A key insight of Bazel is that you can’t have speed at scale without correctness. Would be good to have Earthly described within the framework of “Build Systems a la Carte”.
no_circuit 5y ago
I don't think it is meant to replace any particular build system like Bazel or make. Rather it is a better dependency glue flavor for Docker image building and testing. Earthly uses Makefile-like targets which can be leveraged through more advanced references [0]... essentially can replace plain "copy --from=" with a new magic strings syntax which also understand git. Build systems like Bazel would be invoked within a container. I don't think the "why" using it instead of Bazel (and its plugins) will become clear until the Earthly SaaS gets announced that is hinted about in their TOS [1]. Build resource pool management, image release management, testing/dev environment definitions, custom URLs to test pull requests as seen elsewhere? Arguably one could say they'd probably be able do the same thing by writing some Skylark and use remote build execution, but that could be close to saying Dropbox wasn't special in a Show HN thread. :-) [0] https://docs.earthly.dev/docs/guides/target-ref https://docs.earthly.dev/docs/guides/target-ref [1] https://earthly.dev/tos https://earthly.dev/tos

einpoklum 5y ago

I looked at their main page and got the impression this tool is not very / not at all useful - except perhaps for some niche I'm not in. > Earthly is a syntax for defining your build. I already have syntax for defining my build. > It works with your existing build system. If I have a build system, or a build system generator, why do I need to redefine my build another way? > With Earthly, all builds are containerized Huh? What? Why? ... That's already a deal breaker. I don't want to be limited to building within containers. > With Earthly, all builds are ... repeatable Isn't that true for any decent build system? > With Earthly, all builds are ... language agnostic An ambiguous statement. Especially since a build process does depend, to some extent, on the language of the source files. > Get repeatable and understandable builds today. (shrug) more marketing talk. > It's like Makefile and Dockerfile had a baby. I don't want Dockerfile in the family, and as for Makefile - it's too low-level. Makefiles get generated these days (and have been for over 20 years for most software packages, IIANM.) > No more git commit -m "try again" . This has never been a problem for me. (Well, ok, if there's network trouble, then downloading things during a build will fail; but Earthly can't magically reconnect a broken network.) > "Earthly is a pragmatic and incremental solution to a thorny problem that all organizations face" Which thorny problem is that? The "non-repeatable builds"? I'm confused.

fishywang 5y ago
My thoughts exactly. I guess there's a niche for Mac/Windows developers to be able to run the build system on linux, because that's what their CI system runs (so it might fail for some issues that does not fail on their host machines), but most CI systems cannot run nested containers (because docker-in-docker requires some privileges), and as a result their CI system usually can not run this thing, which totally defeats the purpose.
verdverm 5y ago
Earthly is essentially a make/dockerfile syntax for a BuildKit frontend. If you understand and agree that BuildKit has value, than that's the jist of it. Agree with all your points for sure, but people need words on their landing page
yagizdegirmenci 5y ago

duped 5y ago

How tightly coupled is this to docker? My major gripe is that I legitimately need cross platform build and test (windows, Linux, and macos). I will pay hundreds to thousands of dollars a month to someone that solves these problems (I already do, and release engineers aren't hurting!).

detaro 5y ago
Cross-platform support would be my question too, yes.
agbell 5y ago
We do have plans for podman and other container runtime support. Here is our roadmap: https://github.com/earthly/earthly/projects/1 https://github.com/earthly/earthly/projects/1
- duped 5y ago
  A guess my question is more "how tightly coupled is this to Linux containerization technologies" because if the answer is "fundamentally so" then I don't see a legitimate use for it that creates any value. I spend a lot of time and money on build automation because nobody has solved it for cross platform builds, and I see forcing containerization of key elements into backend services as an insane solution to that problem.
  - agbell 5y ago
    It is fundamentally a containerized build technology and people use it for that reason. That might make it not have value for you, buts fundamental to the reason that those who use it do find it valuable.

kortex 5y ago

This is AWESOME! I already have a super container heavy workflow and abuse dockerfiles to do things like this. It's great to be able to generate artifacts in a container and slurp them back out (really tricky with normal dockerfile builds, you need exterior commands). How well does it work with big builds in a single stage that fail? E.g. If I have a 3h CMake build that fails, does it keep the intermediate files in an image somewhere?

agbell 5y ago
It's pure target transactional, so it would not keep intermediate files in that case.

icy 5y ago

Maybe don't call Earthly "free and open source" when it clearly is neither -- it is licensed under the Business Source License.

hitekker 5y ago
I’m trouble finding the words “free and open source” or “foss” in their marketing material. Rather, it looks like they’re calling Earthly “open core” or “source available” according to https://earthly.dev/blog/every-open-core-company-should-be-a-source-available-company/ https://earthly.dev/blog/every-open-core-company-should-be-a...
- icy 5y ago
  Landing page, below the 'Get Earthly' buttons (they have a few): > Free and open
  - InitialBP 5y ago
    to be fair "Free and open" is different than "Free and Open Source". Open source has a much stronger connotation than open and I could certainly see calling it open if the source is available for the core of the product.

stonewareslord 5y ago

Earthly has been great to use. My one issue is with docker. I would greatly prefer to use podman but it seems there is no support for it yet. Every issue I have had with earthly is with some weird quirk with Docker I don't have with podman. For the most part I can get around this with setting DOCKER_HOST, but it is still weird to depend on dockerd when I would expect podman to work fine.

agbell 5y ago
Thanks for checking it out. We do get a lot of requests for podmad support and have it on our roadmap here: https://github.com/earthly/earthly/projects/1 https://github.com/earthly/earthly/projects/1

PufPufPuf 5y ago

I wonder how well will this play with [VSCode devcontainers](https://code.visualstudio.com/docs/remote/containers https://code.visualstudio.com/docs/remote/containers) -- perhaps one could create a target to build a development image, that extends the CI image by adding development tools? I'll definitely experiment with this, sounds promising.