4 ms·

Being able to run your pipelines locally is a pretty neat use case. That's something that often is difficult or basically impossible. Except for maybe GitHub a

Being able to run your pipelines locally is a pretty neat use case. That's something that often is difficult or basically impossible. Except for maybe GitHub actions through Act (https://github.com/nektos/act https://github.com/nektos/act). I'd still lean to something in the yaml sphere if it eventually would be used in deployment pipelines and such. For example a solution incorporating ansible. It also seems to me that the argument you make is mostly focused on the building step? Earthly certainly seems focused on that aspect.

SOLAR_FIELDS 3y ago
Act is also quite imperfect around its emulation and has a lot of edge cases. Not by any virtue of being a bad project or lack of effort, it’s just that the way GHA works is slightly insane
shepherdjerred 3y ago
Act works pretty well in my experience, but it's tied to GHA. Once nice thing about Earthly + Dagger is that your build + pipeline code are bundled together, and you can run them on any compute. As for YAML, it is ubiquitous and more accessible to those who are less comfortable writing code. One thing that I don't like about YAML si that for larger GHA workflows become hard to understand quite quickly. It's nice to work in "real" languages like Go/TypeScript where you can create and compose functions, especially if the language you're writing your pipeline in is the same language as your app -- I think that's a very powerful use case where all devs can feel comfortable/empowered to tweak CI/build steps. > It also seems to me that the argument you make is mostly focused on the building step? Earthly certainly seems focused on that aspect. Earthly and Dagger solve both builds and CI pipelines. They're essentially identical aside from the syntax that you use. Earthly and Dagger fit into CI in the same way, too. You'll have a GHA that installs Earthly/Dagger, and then calls your pipeline/build step. For Dagger, this is invoking your pipeline code. For Earthly, you'd want to define a `ci` target, and invoke it, e.g. `earthly +ci` (this means that locally you can run the entire pipeline with the same command). In both cases, you'd want your GHA (or whatever runner you're using) to be as dumb as possible, e.g. it should literally clone the repo, install the CLI, and run one shell command. Maybe it can also upload test reports/annotate PRs/etc, but the more you add to your GHA/test runner code outside of these tools, the less these tools can help you. Here's an example using Earthly for CI: https://github.com/shepherdjerred/glitter-boys/blob/main/.github/workflows/deploy.yml https://github.com/shepherdjerred/glitter-boys/blob/main/.gi... A similar example using Dagger: https://github.com/shepherdjerred/shepherdjerred.com/blob/main/.github/workflows/dagger.yml https://github.com/shepherdjerred/shepherdjerred.com/blob/ma...
- shepherdjerred 3y ago
  I'll follow up on this and say: give Earthly a try! It's super easy to figure out if you've ever written a Dockerfile/Makefile. If you like Earthly and feel limited by syntax, then try out Dagger.
marcosnils 3y ago
Hi there, Dagger contributor here. One thing that generally is not immediately obvious is all the side benefits that using a programming language actually unlocks. Need type checking at build time? done. Need a package manager to version and publish your CI code? done. Need linting / formatting? done. Need to import functions from other places? done Need a test framework? done. Need a better collaborative IDE? done. All these things above bring massive improvements to both Dev and Devops engineers that need to interact with build & deploy pipelines daily. Generally it'll be quite difficult for any other configuration or custom language to provide these kind of tooling.
- shepherdjerred 3y ago
  I completely agree, which is why Dagger is perfect for medium/large projects. I think Earthly is better for small projects with a simple build system since it's so much simpler to write and understand. I think Dagger is a worthwhile investment once your Earthfile is >200-400 lines.
  - marcosnils 3y ago
    > I completely agree, which is why Dagger is perfect for medium/large projects. Initially that was the case, but with the upcoming work on modules (aka project Zenith), hopefully this changes. After all, IMO the ultimate aspects that will make it super easy to adopt is the ecosystem and learning how to use the tool. The later is mostly tackled by the "pick your favorite language" story; the former is where we hope modules can help users the most.
- idoubtit 3y ago
  I'm sorry, I can't make sense of what you wrote, those "side benefits that using a programming language [in CI] actually unlocks". Could you please give a concrete example? > Need type checking at build time? done. It's not unlocked by Dagger since every CI can already do this. And it's almost always about launching a (shell) command. How is that line easier in Go/any-language than in yaml? > Need to import functions from other places? done You mean "import functions into the CI code"? Why would I need that? It's usually possible through plugin systems, but I've never felt the need to put complex code into CI actions. > Need a test framework? done. Is that a test framework to test the CI code from within the CI? I hope I'll never want one! I guess I completely misunderstood... On the other side of those "benefits", I can smell problems. For instance, when Docker images and Compose systems are built from the Dagger API. If the need for some advanced features comes in, this extra layer could prove costly.
melezhik 3y ago
Yes,debugging long running /complex / multistage pipelines is a tedious process, however imho it does not make a huge difference if one runs the same pipelines locally (they won’t turn into something more lightweight after that local run), however instead we would rather have simple steps ( scripts) which we can debug individually

SOLAR_FIELDS 3y ago

Act is also quite imperfect around its emulation and has a lot of edge cases. Not by any virtue of being a bad project or lack of effort, it’s just that the way GHA works is slightly insane

shepherdjerred 3y ago

Act works pretty well in my experience, but it's tied to GHA. Once nice thing about Earthly + Dagger is that your build + pipeline code are bundled together, and you can run them on any compute. As for YAML, it is ubiquitous and more accessible to those who are less comfortable writing code. One thing that I don't like about YAML si that for larger GHA workflows become hard to understand quite quickly. It's nice to work in "real" languages like Go/TypeScript where you can create and compose functions, especially if the language you're writing your pipeline in is the same language as your app -- I think that's a very powerful use case where all devs can feel comfortable/empowered to tweak CI/build steps. > It also seems to me that the argument you make is mostly focused on the building step? Earthly certainly seems focused on that aspect. Earthly and Dagger solve both builds and CI pipelines. They're essentially identical aside from the syntax that you use. Earthly and Dagger fit into CI in the same way, too. You'll have a GHA that installs Earthly/Dagger, and then calls your pipeline/build step. For Dagger, this is invoking your pipeline code. For Earthly, you'd want to define a `ci` target, and invoke it, e.g. `earthly +ci` (this means that locally you can run the entire pipeline with the same command). In both cases, you'd want your GHA (or whatever runner you're using) to be as dumb as possible, e.g. it should literally clone the repo, install the CLI, and run one shell command. Maybe it can also upload test reports/annotate PRs/etc, but the more you add to your GHA/test runner code outside of these tools, the less these tools can help you. Here's an example using Earthly for CI: https://github.com/shepherdjerred/glitter-boys/blob/main/.github/workflows/deploy.yml https://github.com/shepherdjerred/glitter-boys/blob/main/.gi... A similar example using Dagger: https://github.com/shepherdjerred/shepherdjerred.com/blob/main/.github/workflows/dagger.yml https://github.com/shepherdjerred/shepherdjerred.com/blob/ma...

shepherdjerred 3y ago
I'll follow up on this and say: give Earthly a try! It's super easy to figure out if you've ever written a Dockerfile/Makefile. If you like Earthly and feel limited by syntax, then try out Dagger.

marcosnils 3y ago

Hi there, Dagger contributor here. One thing that generally is not immediately obvious is all the side benefits that using a programming language actually unlocks. Need type checking at build time? done. Need a package manager to version and publish your CI code? done. Need linting / formatting? done. Need to import functions from other places? done Need a test framework? done. Need a better collaborative IDE? done. All these things above bring massive improvements to both Dev and Devops engineers that need to interact with build & deploy pipelines daily. Generally it'll be quite difficult for any other configuration or custom language to provide these kind of tooling.

shepherdjerred 3y ago
I completely agree, which is why Dagger is perfect for medium/large projects. I think Earthly is better for small projects with a simple build system since it's so much simpler to write and understand. I think Dagger is a worthwhile investment once your Earthfile is >200-400 lines.
- marcosnils 3y ago
  > I completely agree, which is why Dagger is perfect for medium/large projects. Initially that was the case, but with the upcoming work on modules (aka project Zenith), hopefully this changes. After all, IMO the ultimate aspects that will make it super easy to adopt is the ecosystem and learning how to use the tool. The later is mostly tackled by the "pick your favorite language" story; the former is where we hope modules can help users the most.
idoubtit 3y ago
I'm sorry, I can't make sense of what you wrote, those "side benefits that using a programming language [in CI] actually unlocks". Could you please give a concrete example? > Need type checking at build time? done. It's not unlocked by Dagger since every CI can already do this. And it's almost always about launching a (shell) command. How is that line easier in Go/any-language than in yaml? > Need to import functions from other places? done You mean "import functions into the CI code"? Why would I need that? It's usually possible through plugin systems, but I've never felt the need to put complex code into CI actions. > Need a test framework? done. Is that a test framework to test the CI code from within the CI? I hope I'll never want one! I guess I completely misunderstood... On the other side of those "benefits", I can smell problems. For instance, when Docker images and Compose systems are built from the Dagger API. If the need for some advanced features comes in, this extra layer could prove costly.

melezhik 3y ago

Yes,debugging long running /complex / multistage pipelines is a tedious process, however imho it does not make a huge difference if one runs the same pipelines locally (they won’t turn into something more lightweight after that local run), however instead we would rather have simple steps ( scripts) which we can debug individually