21 ms·

Dagger: a new way to build CI/CD pipelines

• simulate-me 5y ago
  What is the benefit of using a new bespoke syntax vs. running docker containers directly? Drone.io does this very well.
  • shykes 5y ago
    One way or the other, you will have to write a declarative configuration describing which containers to run, and how to interconnect them. Then you'll want to add templating to that configuration, perhaps a schema verification tool, some sort of test scaffolding, etc. Our philosophy is that, if you're going to do all those things, it's better to use a dedicated platform with a complete syntax, API and integrated tools, instead of cobbling it together yourself.
• andrewmcwatters 5y ago
  shykes, can you explain or have the folks working on the website explain how is this different or better than just using Jenkins?
  • shykes 5y ago
    Sure, it's better for 3 reasons: 1. You can use the same pipelines in development and automation. Dagger runs on Jenkins, but it also runs on the developer's laptop. This solves a common problem which is drift between the automation available in dev and CI. 2. You're not stuck with Jenkins. If you want to migrate a Jenkins-native pipeline to another CI, you have to rewrite everything. Dagger on the other hand can run on any major CI. This makes migrations much easier, as well as supporting teams with heterogeneous CI setups (which is quite common in larger organizations). 3. You can debug and test your pipelines locally. In theory this is possible with some CI systems. But in practice, the experience is very different. You can actually iterate on your Dagger configuration like you would on regular code: make a change, run, try again, run again. It's quite fun and addictive. EDIT: there is a 4th reason, which is that Dagger uses buildkit under the hood (the hidden backend for "docker build"). So every action in the DAG is automatically cached when possible. This means that your pipeline is tuned for optimal caching out of the box: no manual tweaking necessary.
    • andrewmcwatters 5y ago
      Thank you, I appreciate the brief summary very much.
• mountainriver 5y ago
  We should definitely escape from YAML, but this language isn't it
  • speed_spread 5y ago
    Full YAML is crazy, but the StrictYAML subset I can definitely live with. I regularly go over the main alternatives (TOML, JSON, XML), and they all have serious warts for config IMO. Better formats exists (such as HOCON) but are much less popular. For a public tool I'd still stick with a YAML subset that everyone knows and can work with a minimum of fuss.
    • mountainriver 5y ago
      I agree everyone knows YAML but its only really great as a config language, not as a general purpose turing-complete programming language. The answer here is more likely to be something like what Pulumi did
      • shykes 5y ago
        Pulumi is great and we think of it as complementary to Dagger: one is focused on managing infrastructure, the other on deploying applications on top of the infrastructure. They are more similar in their choice of language than you might think: * Dagger lets you write actions in any language: Go, Python, Shell, it doesn't matter. Then it runs it in containers, wrapped in a declarative configuration. This allows cross-language interoperability. * Pulumi also supports existing languages, although it is more deeply integrated, with language-native SDKs (the tradeoff being less supported languages). Eventually they realized that they needed an additional layer to bridge language siloes, otherwise the Pulumi ecosystem is really an archipelago of incompatible components: Pulumi/Go, Pulumi/TS, etc. To solve this Pulumi introduced... a declarative configuration wrapper. In summary: the best design for this problem requires two layers. One layer for leveraging the existing programming languages with maximum compatibility. And a second to add a declarative configuration model and reduce ecosystem fragmentation. Pulumi and Dagger are both converging on this model, each for their own domain of application. I personally believe we will see many Dagger + Pulumi integrations in the future :)
• justRafi 5y ago
  Been using Drone CI for several years, it has a cli and a command to run locally. Drone Runners allow executing builds on different platforms. CUE is interesting, but I can’t see any major advantages to switching to Dagger.
• vips7L 5y ago
  Not to be confused with https://dagger.dev/ https://dagger.dev/
  • clumsysmurf 5y ago
    When I read the headline, my first thought was dagger dependency injection got a CI/CD feature ... which is ridiculous of course. I think this naming will cause confusion, especially since dagger DI is a large project from google.
  • j2kun 5y ago
    +1 what a strange choice to deliberately conflict with an existing project
    • DangitBobby 5y ago
      There are only so many words.
      • deepsun 5y ago
        We can limit our word space by e.g. bird names (like they do internally at Twitter), and still haven't used most of them. There's a funny paradox: ask people to "name 10 white things", and they do it SLOWER than if you make the task harder by asking to "name 10 white things in your fridge".
  • ComradePhil 5y ago
    Also not to be confused with https://dagster.io/ https://dagster.io/
• xrd 5y ago
  This seems interesting. But, I wish there was some code to show me what makes it so radically different. It seems like this is targeting developers (or is it devops team?) and I'm excited about the new language here, but I don't see any examples of code. Code engages both my head and heart. I am reusing a lot of code in my CI jobs. I have an upload script that I reuse. I have a DEB package script that I reuse across many projects. So, that assertion rings false to me, and seems to indicate there is an unhealthy wall between devops and the developers that prevents shared code. Maybe I misunderstand. The thing that always bites me is that I have trouble debugging a job that depends on the artifacts from a prior job. My e2e job is flakey, and I'm loathe to fix it, because I have to re-run the entire pipeline, test -> build -> installer, etc to get the final artifact used in the e2e job. I've not figured out a way with "gitlab-runner exec" to run a job later in the pipeline and somehow pass in my own artifacts locally. This would be something (albeit very specific to gitlab) that would make me very excited.
  • dijksterhuis 5y ago
    > But, I wish there was some code to show me what makes it so radically different. Sorry to be that guy but maybe try the docs page...? https://docs.dagger.io/ https://docs.dagger.io/ https://docs.dagger.io/1205/container-images https://docs.dagger.io/1205/container-images
    • xrd 5y ago
      Thank you. It isn't hard to find, but my point was that if you say "using an intuitive declarative language" then a developer will get excited by a code snippet that shows that intuitive developer language. It wasn't there, and I think their post could be improved by having less fluffy language and more code, if they are targeting me, that is. This link does show the code, but it took a few clicks to get there from your links: https://docs.dagger.io/1202/plan https://docs.dagger.io/1202/plan At first glance I'm not in love with the language. When I look at the first example, there is a lot I have questions about.
      • verdverm 5y ago
• shykes 5y ago
  Hi everyone, I'm one of the co-founders of Dagger (and before that founder of Docker). This is a big day for us, it's our first time sharing something new since, you know... Docker. If you have any questions, I'll be happy to answer them here!
  • bovermyer 5y ago
    So basically, if I want to not write Jenkinsfiles but still use my company's existing Jenkins installation, I can use Dagger?
    • shykes 5y ago
      Yes :) You can write one last Jenkinsfile that runs Dagger, then do everything inside Dagger. Then you can run the exact same Dagger configuration on another CI system, or on your laptop. All you need is a Docker-compatible runtime (we run everything on buildkit under the hood).
      • bovermyer 5y ago
        Man, I am SO excited for that! Kudos!!
      • chologrande 5y ago
        > You can write one last Jenkinsfile that runs Dagger I'm very confused by this sentiment. This approach loses the best of existing CI tooling does it not? Jenkins users lose understanding of what's actually being executed, what stage they're in, and how long it took. It might provide convenience, and the a (great) benefit of running locally the same as in your CI environment, but it seems to me this would make it difficult for devs to easily understand where/why their build failed, since it just has one megastep.
        • ______-_-______ 5y ago
          I assume if the jenkins-step failed, you'd click a link to the dagger UI to see which dagger-step failed. Alternatively, never open jenkins at all and instead keep a tab open with the dagger UI.
• Svoka 5y ago
  So if it is a "devops engineer" why not call them just "ops" if they don't do software engineering?..
  • bckr 5y ago
    Because the point of their role is to not do the same manual work over and over but to gain leverage over time by developing software to automate ops
  • bovermyer 5y ago
    The name war continues unabated. I'm a "devops engineer." Besides pipeline code and maintaining/building out infrastructure, I also write CLI applications to handle complex infrastructure tasks. At what point is a "devops engineer" not a software engineer? [p.s., I prefer the term "platform engineer," personally]
    • chadcmulligan 5y ago
      So you spend all your time looking after dev infrastructure? Can I ask how many devs there are where you work?
      • bovermyer 5y ago
        Somewhere around 75, if I had to guess.
        • chadcmulligan 5y ago
          fair enough, thanks. Just yourself or a team?
  • philipswood 5y ago
    It’s just software engineering with more obstinate, less tolerant components and without an IDE.
    • 5y ago
• Aeolun 5y ago
  After reading this entire post, I’m still left wondering what problem this solves for me, beyond fluffy promises of ‘everything is going to be better’. At the very least I’d want to see a comparison with what we have now, to show me how this is better. I get that I can try to explore more, but if I don’t get a compelling reason to do so after reading the introductory post, I’m not very motivated to do so Any motivation I do have completely hinges on the words ‘from the creators of docker’, not on the merits of this particular product itself.
  • fideloper 5y ago
    I run a CI application for Laravel developers (Chipper CI). It turns out, the gap between "works locally!" and "works in CI!" is not negligible, especially when you're not sure "about all that server stuff". Getting this working locally with a fast cycle time, and then being able to easily move that into a CI environment of your choice sounds exciting to me. Furthermore, the majority of our customer support is "I can't reproduce this locally but it's broken in CI". Everyone blames the CI tool, but it's almost never the CI tool - just drift between environments. A way to debug locally is a killer feature. Is it worth an entire, funded company? I'm not sure, but I'm excited for them to exist!
    • austinshea 5y ago
      Always nice to have new players in the space, but that gap isn’t even addressed here. Same old problems with configs/secrets, integration with internal/external services, and the details required by your cloud provider. This is the sort of solution I have regularly been hired to untangle, after a company entrenches itself.
  • shykes 5y ago
    That's fair, it can be hard to find the right balance of high-level explanation and technical detail. We tried to solve this by tailoring different parts to different audiences: * The blog post is more high-level. It describes the very real problem of devops engineers being overwhelmed with complexity, and the promise of a more modular system, but does not provide lots of details. * The dagger.io website does provide more technical detail. For example it talks about the 3 most common problems we solve: drift between dev and CI environments; CI lock-in; and local testing and debugging of pipelines. It also features animated code samples. * The documentation at https://docs.dagger.io https://docs.dagger.io go in even more details, and walk you through a concrete example. We do feel that we can do a better job explaining the "meat" of the tool. But we decided to launch and continue improving it incrementally, in the open. If you have any specific suggestions for improvements, please keep them coming!
• chubot 5y ago
  Sigh ... this page is unreadable in Firefox on iPad, and unreadable without JavaScript. Either no text, or black on black
  • shykes 5y ago
    Sorry about that. We'll fix it asap.
  • zefrvtgevtrb 5y ago
    Could you share more details please ? It would be awesome to open an issue on dagger repo : https://github.com/dagger/dagger/issues https://github.com/dagger/dagger/issues. Thanks :)
• Aeolun 5y ago
  This bait and switch in the docs when trying to see an example for anything other than Github feels a bit off to me: > If you would like us to document GitLab next, vote for it here: dagger#1677 If you don’t have an example for a specific tool, just don’t add it to your documentation.
  • vasco 5y ago
    Disagree, it's a perfect way to learn about interest from customers by counting votes or just traffic to that page. This way they can learn which things to implement first with very little impact to you, while giving you a way to help them know you want a Gitlab integration. What's the alternative, you want to have to email them to ask about it or wonder if they're considering it vs Bitbucket or a bunch of alternatives?
  • shykes 5y ago
    To clarify, you can run Dagger on Gitlab today. It just requires some manual configuration that we would like to automate away, for convenience. We have only done this for Github so far, and would like to do it for more. We will look for a way to make this more clear in the documentation.
    • BillSaysThis 5y ago
      What about Bitbucket?
      • shykes 5y ago
        The only dependency for running Dagger is buildkit - which itself requires a docker-compatible container runtime. This could be containerd, runc or of course Docker engine. If your environment can run Docker containers, it can run Dagger. There's another aspect which affects performance: cache persistence. Buildkit needs a place to persist its cache across runs. By default, it relies on local storage. If that is wiped between runs (common in CI environments), everything will still run, it will just be slow - possibly very slow. Luckily, buildkit supports a variety of cache backends, including a Docker registry. In practice, persisting the cache is the most labor-intensive part of integrating in your CI, even though it's technically not a hard dependency. This is the part that we want to automate away for users, but it requires additional engineering work - hence our asking the community for input on what to prioritize.
• daenz 5y ago
  >Write your pipeline once, run anywhere Ha, finally. The timeline for a pipeline that I wrote recently looks like this: 1. Write local test/deploy script 2. Promote scripts to hosted CI system 3. Local scripts rot 3. CI system down, need to use local scripts again 4. Re-write scripts to be current 5. Force CI system to use my local scripts
• nickstinemates 5y ago
  Congrats on the launch - best of luck!
• cebert 5y ago
  Our team has had great success with GitHub Actions and Environments for CI/CD. One nice thing going that route is the build related code is contained within repositories. A large number of developers are already familiar wit GitHub, which makes onboarding new team members easier. I don’t see anything too compelling with dagger.io that is missing with GitHub. You can even use ACT to test workflow changes to builds locally.
  • phphphphp 5y ago
    I'm in the same boat, but I do think there's a prospect of Dagger being a superior option in the long term, if... 1. They invest in building out their catalog of actions to compete with GitHub's. I maintain a few GitHub Actions and despite the GitHub catalog's depth, it's still lacking in many ways and GitHub don't appear to invest in it too much: a "maintainer fund" and creative poaching from Dagger could rapidly bring them up to par. A few million of their raise, well deployed, could crush GitHub's catalog. 2. They invest in tight integrations with platforms. GitHub Actions is great because of composability, yes, but also the deep integration with GitHub itself. Being able to run Dagger on GitHub Actions is one thing, but being able to leverage deployment environments cross-platform would be another. 3. GitHub Actions is great, I am a fan of it, I'll speak highly of it often, but the codebase... it is bad. If Dagger can build out a platform that competes with GitHub Actions on functionality, and it has a pleasant codebase, they'll make huge gains from community participation. Contributing to GitHub Actions is painful. So, I agree with you today, but a year from now, I could see a very different situation and I am optimistic.
    • phphphphp 5y ago
      Also I'm sure he's on their radar given he works for Docker, but they should spend whatever it takes to hire github.com/crazy-max
      • shykes 5y ago
        Fun fact, Crazy Max is the author of the Github Action for Dagger :) https://github.com/dagger/dagger-for-github https://github.com/dagger/dagger-for-github
• billconan 5y ago
  > Run on any Docker-compatible runtime. so does its worker run on a native machine, because we need to build docker images using CI/CD, which we can't do easily within a container? also, is there a gui?
  • shykes 5y ago
    > so does its worker run on a native machine, because we need to build docker images using CI/CD, which we can't do easily within a container? `dagger` is a client tool, which uses buildkit as its worker. Buildkit itself can run on a native machine, or inside a container. Either way, building docker images is one of the most common actions performed by Dagger, and it is very well supported.
  • shykes 5y ago
    > also, is there a gui? Not at the moment, but we plan on offering one (optionally) as a web service.
• nickysielicki 5y ago
  I’m highly jealous of teams that can containerize all their build tools. If you deal with proprietary toolchains that are tens of gigabytes (Windows WDK, Xilinx Vivado/Vitis) it’s just untenable, and that’s before even mentioning licensing. Even Azure doesn’t have a great solution for WDK development. It’s hard to feel like we’re not being left behind. Bind mounting the tools into the container is an option but at that point you’re using the container just because that’s what your CI expects, not because it’s any better or more reproducible than a raw Jenkins shell script.
  • theraven 5y ago
    You can add Apple toolchains to that list also
    • bnert 5y ago
      preach
  • jaaron 5y ago
    100% this. I feel like there's a ton of innovation in the cloud/container CI/CD space, but next to nothing elsewhere. In fact, some of the innovations in CI/CD make things _more_ difficult for those of us developing in other environments (such as game development). There's a lot of low hanging fruit for improving things elsewhere.
    • lazypenguin 5y ago
      Yes! I’m responsible for a game development CI/CD pipeline that needs to run on windows and I feel like it’s harder than it needs to be in 2022.
    • jeremy_k 5y ago
      Do you work in game development? I would love to hear a little bit more about your experience. I play and follow Apex Legends and I'm always so curious about how bugs and regressions seem to make it into every one of their patches. As well as tons of new information that gets data-mined. To me, its like they don't have branches (tons of new code not accessible in the game is released in a patch) and they don't have unit tests that catch bugs (a certain ability has 2 modes, with 2 different activation times. In the most recent patch, the activation times ended up the same) but it's possible that developing a game is far different than developing a SaaS product from a coding perspective. Or it could be that studio just has weird practices.
• cube2222 5y ago
  TLDR this basically looks like portable GitHub Actions + Workflows if I understand correctly.
  • shykes 5y ago
    Yes that is a good summary. It has other advantages beyond its portability - but that is the central feature. Also, importantly, it can run on Github Actions itself!
• EddieRingle 5y ago
  I support the effort to build a platform-agnostic CI/CD pipeline solution, but I don't want it in the form of yet another platform. Rather it needs to be a protocol that any platform can tie in to. I'm especially wary since this is another VC-backed effort that will eventually need to be monetized in some shape or form. Additionally, as someone else here has already mentioned, my mind first went to Dagger, the dependency injection tool (https://dagger.dev https://dagger.dev). That tool in particular was named as a play on DAG (directed acyclic graphs), whereas in this case I don't think it would apply since there may be instances where you'd want cycles in a pipeline. On a whim, I clicked on "Trademark Guidelines" (https://dagger.io/trademark https://dagger.io/trademark) and from that page alone I would recommend avoiding this based on the aggressive language used to try and claim ownership of generic words. According to their own language, it seems I'm violating their guidelines by writing this comment. > Our Marks consist of the following registered, unregistered and/or pending trademarks, service marks and logos which are subject to change without notice: Dagger; Blocklayer; and other designs, logos or marks which may be referred to in your specific license agreement or otherwise. > Blocklayer does not permit using any of our Marks ... to identify non-Blocklayer products, services or technology Which would include Dagger, the dependency injection tool. Other sections of note: > Do Not Use As Nouns (This one just reads amusingly to me, for some reason.) > Do Not Create Composite Marks This section seems to suggest that you can't use "dagger" in any shape or form, even as a smaller part of some other word or body of text. > Websites And Domain Name Uses > > ... Any principal or secondary level domain name should not be identical to or virtually identical to any of our Marks. > > The website owner should not register any domain name containing our Marks and should not claim any trademark or similar proprietary rights in the domain name. For example, “daggertech.com”, “dagger-group.com”, “Meetdagger.com” are not allowed. Any violation of this policy may result in legal action. > > The website should otherwise comply with domain name registry policies and applicable laws regarding trademark infringement and dilution. This would technically include dagger.dev, which again refers to the dependency injection tool. --- Full disclaimer that I'm not a lawyer and there could be totally reasonable explanations for these provisions, but they certainly look scary to a layperson such as myself. All in all, the founders seem to be taking a pretty arrogant approach here, but it unfortunately seems to be a common one. I'm choosing not to support it, however. --- EDIT: formatting
• epage 5y ago
  > Dev/CI drift: instead of writing the same automation twice - once in a CI-specific configuration, and again in a Docker compose file, shell script or Makefile - simply write a Dagger plan once, then run it in all environments. Working on developer tooling, a lot of times I would hear from people that they wanted CI and dev to be 100% the same and wanted a simple "run all CI locally" command to pre-check before posting. Unsure how Dagger is handling this but my concerns with the scenario I described - CI normally divides things up into multiple jobs for speed which breaks the shared "do everything" command - Commands need to scale down to fast iteration as people fix things - Generally people get the best integration by using the underlying wrapped tools directly due to pre-commit or IDE integeration
  • candiddevmike 5y ago
    I don't understand why folks can't write everything in a makefile and then call make targets in CI.
    • shykes 5y ago
      That is a common pattern. Dagger is essentially a more modern iteration on that pattern.
    • __turbobrew__ 5y ago
      This is the way. Writing your CI in groovy is a fast track to hell.
    • francislavoie 5y ago
      I hate makefile syntax even more than I hate yaml
    • dikei 5y ago
      It's mostly for speed, if the tasks are defined separately then a smart CI system can cache the results and avoid rerunning some of them.
  • thundergolfer 5y ago
    > I would hear from people that they wanted CI and dev to be 100% the same and wanted a simple "run all CI locally" command to pre-check before posting. This is exactly what developers should want. It's the most efficient workflow for a dev, because we then don't have to think at all. This is a huge dev efficiency anti-pattern: thinking your code is good to merge, pushing the change, and then finding out 10-20mins later that CI isn't happy for some reason that wasn't natural to check locally. The thesis in the following is the way: https://gregoryszorc.com/blog/2021/04/07/modern-ci-is-too-complex-and-misdirected/ https://gregoryszorc.com/blog/2021/04/07/modern-ci-is-too-co...
• BenoitP 5y ago
  What an unfortunate name collision: https://github.com/google/dagger https://github.com/google/dagger Is it some weird SEO niche-squatting technique? IMHO we see it happen way too often
• gregmac 5y ago
  Your Windows instructions/process needs work. > curl https://dl.dagger.io/dagger/install.ps1 https://dl.dagger.io/dagger/install.ps1 -OutFile install.ps1 This uses the `curl` alias, which is really `Invoke-WebRequest`. It also makes the incorrect assumption I haven't fixed this dumb Microsoft mistake to be an alias to actual curl.exe. > [Windows] We try to move the dagger binary under C:\Windows\System32 Ack, please don't do this! This is similar to installing something under /usr/sbin/. Malware is the only modern thing that would attempt to deploy to that folder. > but in case we miss the necessary permissions, we'll save everything under <your home folder>/dagger > C:\<your home folder>\dagger.exe I'm glad you have a non-admin fallback, but also: yuck. I don't want this polluting my home folder (more importantly: I don't want 100's of other things like this also polluting my home folder). The "Windows way" is to install system-wide to %ProgramFiles%\dagger\ (eg c:\Program files\dagger\dagger.exe), or to install to %LocalAppData%\dagger\ (eg: c:\Users\shykes\AppData\Local\dagger\dagger.exe). The latter is kind of the closest equivalent to $HOME/.dagger on linux. Add whatever folder to the user's PATH environment variable to make it easy to run. Honestly, providing just the .zip is better: then Windows users can muck up their own system however they like. Alternatively, package it with something like Scoop [2] which is a fairly popular dev tool, and provides a fairly easy way to get a sane install with updates, versioning and path stuff all handled. [1] https://docs.dagger.io/ https://docs.dagger.io/ [2] https://scoop.sh/ https://scoop.sh/
  • shykes 5y ago
    Thank you for the feedback! I referenced it in an issue here: https://github.com/dagger/dagger/issues/1946 https://github.com/dagger/dagger/issues/1946
    • vardrop 5y ago
      EDIT: You can now directly install dagger with scoop by running scoop install dagger I have opened an issue and PR with scoop [0], see also [1]. You can directly install dagger with scoop meanwhile by using scoop install https://gist.github.com/vardrop/a25e0c8e2dc055f86a3ff4dd7a7de309/raw/0b7f1e29454d7d2cbe1ae9d6c807ddfcdacb7feb/dagger.json [0] https://github.com/ScoopInstaller/Main/issues/3460 https://github.com/ScoopInstaller/Main/issues/3460 [1] https://github.com/dagger/dagger/issues/1946#issuecomment-1083620665 https://github.com/dagger/dagger/issues/1946#issuecomment-10...
• zinclozenge 5y ago
  harness.io was by far the best CD experience I ever had, and it allowed me to do exactly what I wanted. I can't stand the gitops way of defining every change through a PR and having an operator/controller sync the changes. It's a lot more intuitive for me to say "everything in master ends up in prod" and define a pipeline that moves artifacts between environments. The only other software that seems like it does the same thing is Spinnaker, which i have no desire to try self-hosting.
• time0ut 5y ago
  Congratulations! This looks amazing. I think I am the target audience and I cannot wait to try this. One very important thing for my use case is being able to run steps in parallel distributed across multiple agents. Is it capable of this?
  • shykes 5y ago
    Yes :) We rely entirely on buildkit (https://github.com/moby/buildkit https://github.com/moby/buildkit) to run the pipelines (`dagger` itself acts as a "compiler" and buildkit client). So anything you can do with buildkit, works out of the box with Dagger. That includes clustering. There is a lot of ongoing work in that area in the buildkit ecosystem. For example Netflix has a distributed buildkit farm, which I believe relies on containerd storage backends to share cache data. There are also integrations to run buildkit nodes on a kubernetes cluster. Dagger itself is obviously quite new, but buildkit itself is very mature, thanks to its integration with 'docker build'.
• seeekr 5y ago
  This seems similar to what https://earthly.dev https://earthly.dev is doing?
  • shykes 5y ago
    Yes, I believe that is a fair comparison. Earhly is more focused on builds, whereas Dagger has a wider scope: build, test, deployment, any part of a CI/CD pipeline really. But the overall philosophy is the same: run everything in containers. The choice of buildkit as a runtime is also a key similarity. One big difference is that we use CUE as a configuration language, and Earthly uses YAML. We have a lot of respect (and common friends!) with the Earthly developers, I am confident we can help each other build even better tools, and grow the buildkit ecosystem in the process.
    • seeekr 5y ago
      Thanks for the info, that is wonderful to hear! Best of luck to both you and Earthly, much needed work you're doing!
    • mdaniel 5y ago
      Another *monster* difference is that Dagger is (at least currently) Apache 2: https://github.com/dagger/dagger/blob/v0.2.4/LICENSE https://github.com/dagger/dagger/blob/v0.2.4/LICENSE but Earthly went with BSL: https://github.com/earthly/earthly/blob/v0.6.12/LICENSE https://github.com/earthly/earthly/blob/v0.6.12/LICENSE That means I'm more likely to submit bugs and patches to Dagger, and I won't touch Earthly
      • shykes 5y ago
        I can't speak to Earthly's choice of license (and am not familiar with BSL). But can confirm that we have no intention of changing licenses, and if we did, it would be for another OSI-approved license. Our monetization model follows the same fundamentals as Red Hat before us: open code, strictly enforced trademark rules. "You can modify, use and redistribute the code at will. If you distribute a modified version, please call it something else."
• wdb 5y ago
  Interesting, I should check out if this works with Gitlab CI. As Gitlab CI is quite a pain to debug (guess all CI environments).
• henrydark 5y ago
  Seems like more stuff that Nix solves out of the box
  • deleted 5y ago
    [deleted]
  • speed_spread 5y ago
    Is there a thing that Nix can't solve? This is a CI/CD system. It's not made to configure a single machine from coded configuration, but to build, test, publish software for multiple targets and then manage deployment to different clusters.
    • n42 5y ago
      Nix builds, tests and publishes more than 80,000 packages, continuously on every commit, for multiple target platforms, via one monorepo.. without Docker. It’s not just for building machines, that is NixOS and its module system, which is a library you can use with the build tool. It’s possible to build your own Nix-based monorepo internally at your company, too, and still ship containers in production (that are also built by Nix, again, without Docker). Dagger is not a CI provider, it is a build tool and task runner. Integrating Nix with CI is the same deal as integrating Dagger with CI. Enter my biased opinion, as someone leading a team through solving the problems this tool purports to solve, but by using Nix: this tool will suffer from the same fatal flaws that Docker does, by being built on its foundation (Buildkit). It is abstracting dependency management at the application build level, whereas Nix solves it at the fundamental system dependency level. I would like to be proven wrong, so best of luck!
• politician 5y ago
  I’m ready to be an early (&& paying) adopter of this today, but I have real work to get done so it would be extremely helpful if you could describe what doesn’t work well about your solution today.
• nunez 5y ago
  Hell yes. A build system that I can test locally without hacks (like act) is literally the dream.
• vhanda 5y ago
  How would this handle CI/CD jobs which need to be run on a mac? From what I understand, everything needs to be a container, and that doesn't work for xcode.
• evancordell 5y ago
  I played with a similar idea a while ago: https://github.com/ecordell/cuezel/ https://github.com/ecordell/cuezel/ (cuezel as in: "Bazel but with CUE"), but I was never sure that what I was doing was in the spirit of CUE. CUE pushes nondeterminism into "_tool.cue"[0] files that are allowed to do things like IO and run external processes. Tool files scratch a similar itch to Makefiles, but they lack an integrated plugin system like Bazel (hence why I played with the idea of CUE + Bazel). With Dagger you seem to be restricted to the set of things that the dagger tool can interpret, just like with my Cuezel tool you are limited to what I happened to implement. In CUE `_tool` files you are also limited to the set of things that the tool builtins provide, but the difference is that you know that the rest of the CUE program is deterministic/pure (everything not in a _tool file). There's clearly value in tooling that reads CUE definitions, and dagger is the first commercial interest in CUE that I've seen, which is exciting. But I'm most interested in some CUE-interpreter meta-tool that would allow you to import cue definitions + their interpreters and version them together, but for use in `_tool` files to keep the delineation clear. Maybe this is where dagger is heading? (if so it wasn't clear from the docs) [0]: https://pkg.go.dev/cuelang.org/go@v0.4.2/pkg/tool https://pkg.go.dev/cuelang.org/go@v0.4.2/pkg/tool
  • shykes 5y ago
    I know exactly what you mean. Earlier versions of Dagger actually took this "embedding" approach: any CUE configuration is a valid Dagger configuration, with an extra layer of annotation that describes how to make that configuration runnable. What we learned is that this model is thoroughly confusing to almost everyone except hardcore CUE enthusiasts. Now we use CUE in a more straightforward way: as a superior replacement for YAML. There's a (simple) schema that all Dagger plans must follow; beyond that, you can import any CUE package and apply any definition into your plan. But you don't need to go upstream and annotate these packages with additional Dagger metadata. I'm not sure if my explanation is clear - even I get confused by this embedding business sometimes.
    • evancordell 5y ago
      In the examples it looked as though there is still a CUE unification loop happening during dagger processing: deploy: netlify.#Deploy & { contents: build.contents.output } It looks like dagger is using cue as a bit more than a YAML replacement; it hydrates cue values as it runs - which is cool! - but that's the part that seemed at odds with CUE's philosophy of pushing nondeterminism into clearly marked files.
• deleted 5y ago
  [deleted]
• xmlninja 5y ago
  "A developer’s dream is a devops engineer’s nightmare" Well, the opposite is very much true as well. A devops engineer's dream is usually a developers nightmare.
  • shykes 5y ago
    That is sadly often true today, but not inevitable! When you are overwhelmed with complexity, a natural defense mechanism is to restrict choice. But with better tooling, you can better manage and reduce complexity, which allows you to say "yes" to developers without compromising on reliability and security. That is what we are trying to enable with Dagger.
  • goodpoint 5y ago
    So much for "devops" merging together development and operations. "A developer’s dream is a devops engineer’s nightmare" underines the failure of devops.
  • wseqyrku 5y ago
    I think the opposite cannot be true, because developers come physically before operations. Isn't it actually devops engineers' job to make it so?
• diggs 5y ago
  There's so much more to CI/CD than the build definitions (e.g. dashboarding, log viewing/retention, access control, manual interventions, secret management, test reporting, etc.) and while some of your points resonate very strongly with me (e.g. local builds), I can't help but wonder what the endgame is here? You've raised $27m to what? Give away a free tool for engineers to define their builds in? While they continue to use and pay for another service to actually run the builds? I assume you intend to replace the CI service at some point and move up the stack to monetize? Without more transparency its easy to imagine something like... Step 1. Drive uptake of your tool by selling people on the pitfalls of "CI lock-in" Step 2. Introduce your own CI solution, which people can now easily switch to Step 3. Lock people in
  • shykes 5y ago
    Good question. No, we don't intend to replace the CI service. We think of CI as infrastructure, and there are plenty of great infrastructure providers out there. And you are absolutely correct that for Dagger to succeed, it has to remain agnostic to infrastructure providers, which means we cannot build an infrastructure provider business ourselves. If the experience of hosting Dagger becomes so bad that it affects the developer experience, we might ship such a feature as a stopgap and even charge for it - but it is not a strategic priority. The sooner other infrastructure providers offer a fantastic hosting experience, the better. That is not where we see the biggest opportunity for Dagger. However, there is a great opportunity to help businesses manage their software supply chain. What is running where, and how did it get there? Who is authorized to run which pipelines on which machines, and which which inputs? What is the diff between this dev environment and staging? Staging and production? Etc. Keep in mind this is not limited to your production pipeline; there are staging environments, development environments, all running pipelines all day long. It's hard to simply know everything that is going on. Each time Dagger runs your pipeline, it can produce telemetry about every aspect of your supply chain. Git SHAs, docker image checksums and cryptographic signatures, specific versions of kubernetes and cloudformation configurations, before and after templating, test results, monitoring status... It also integrates with your secrets management backends, your developer's local source code. Basically every node in your supply chain can be a node in your DAG if you want it to. The logical next step is to give you a centralized place to send all this telemetry, and tools to extract insights from it. You could also perhaps manage the configuration of your various Dagger engines in one location. Another product that is often requested is a visual DAG debugger. When a pipeline break, you want to know why, and staring at your CI logs is definitely not the best experience for that. With a web UI, there's a lot we can do there. The business opportunity boils down to this: if CI/CD pipelines are software, there ought to be a platform for developing that software, and an ecosystem of developers creating value around that platform. Dagger aspires to create that missing developer ecosystem. If we succeed, there will be no shortage of business opportunities. If we fail, none of the other features will matter.
• smrtinsert 5y ago
  I remember reading about composable pipelines a while ago. I remember thinking this is going to take the devops world by storm, hopefully Dagger brings us closer to that dream.
• abledon 5y ago
  I'm getting "AmongUs" vibes from this site. Its called 'Dagger', and the illustrations are an anonymously clad astronaut running around a space station.
• andreineculau 5y ago
  I'm happy to see dagger, because it is something that I have been working with since 2016, and ended up creating https://github.com/ysoftwareab/yplatform https://github.com/ysoftwareab/yplatform It uses well-tested software (GNU Make, Bash, Homebrew, Docker) and there's not much to learn really. * Incubated at Tobii (eyetracking, Sweden), * it has been used for several projects of different sizes, * tested on 12+ CIs ! * tested on macOS, Linux and Windows (WSL) ! * tested on 8+ Linux distros ! * comes with prebuilt Docker images ! Early integration with VSCode. And much more. Just read the README :) Happy to help anyone integrate it.
• rendall 5y ago
  Paul Graham just highlighted it on Twitter: https://twitter.com/paulg/status/1509209579414040588 https://twitter.com/paulg/status/1509209579414040588
• oh_sigh 5y ago
  Come on guys...there is an extremely well known and widely used Java library called Dagger. Call yours Cinquedea, Jambiya, Anelace, or Rondel if you really want a knifey name.
• tofflos 5y ago
  Have you given IDEs any thought? I feel like it isn't write once run anywhere until I can use the same configuration inside the IDE, from the command-line, and in CI. The IDE would need to understand what the dependencies are in order to provide things like auto-completion, debugging and syntax-highlighting.
  • shykes 5y ago
    Yes IDE integration are a great idea and we plan on building those too. What IDE would you personally want to integrate, and what actions would you want it to support?
• no_circuit 5y ago
  During the past few months I wrote a CI/CD pipeline using GitHub actions, Terraform, and Kubernetes. I'm not too sure Dagger would have saved that much time for me unless there was already a fully setup service already being sold. The main issue see so far is that I'd also want bringing up a CI instance of the service the same was as new prod (backup) cluster. All of that is written in Terraform already (and GitHub Actions). Why rewrite it at this point? If I need stats I could probably push them to Prometheus. So yeah, ha, sounds like just writing something like Dropbox would be easy on Linux. But it could be great if I wouldn't need to drop down to Bash Run all the time, since a lot of official or community extensions would need to be created: https://docs.dagger.io/1202/plan#plan-structure https://docs.dagger.io/1202/plan#plan-structure. Somewhat reminds me a bit of Earthly: https://earthly.dev/ https://earthly.dev/.
  • icecap12 5y ago
    Glad someone mentioned GH Actions. Recent SaaS issues aside, one of the value-props to me with GH Actions is that I can leverage it in a multi-cloud environment if I'm using GitHub for SCM, which my organization is. There are some other benefits I specifically care about that others might not as well. Infra is still largely a separate concern (as in, it doesn't matter what I use, I have options). I suppose I need to look into Dagger more to understand the value-prop.
• barbs 5y ago
  Not to be confused with Dagger, the dependency injection framework used in Android apps [0]. That's DI, not CI! [0] https://developer.android.com/training/dependency-injection/dagger-basics https://developer.android.com/training/dependency-injection/...
• kristianpaul 5y ago
  Just another CI/CD pipeline... Not saying we should write one [1] but CI/CD are a commodity these days. [1] https://aosabook.org/en/500L/a-continuous-integration-system.html https://aosabook.org/en/500L/a-continuous-integration-system...
• zamalek 5y ago
  I'm not touching anything Docker anymore. Here's the scenario: you're the unfortunate soul who received the first M1 as a new employee, and nothing Docker-related works. Cue multi-arch builds; what a rotten mess. I spent more than a week figuring out the careful orchestration that any build involving `docker manifest` needs. If you aren't within the very fine line that buildx assumes, good luck pal. How long has `docker manifest` been "experimental?" It's abandonware. Then I decided it would be smart to point out that we don't sign our images, and so I had to figure out how to combine the `docker manifest` mess with `docker trust`, another piece of abandonware. Eventually I figured out that the way to do it was with notary[1], another (poorly documented) piece of abandonware. The new shiny thing is notation[2], which does exactly the same thing, but is nowhere near complete. At least Google clearly signals that they are killing something, Docker just lets projects go quiet. I'm now looking at replacing everything I've done for the past 4 months with Podman, Buildah, and Cosign. How long before this project lands up like the rest of them? Coincidentally, we were talking about decoupling our CI from proprietary CI, seeing this was a rollercoaster of emotions. [1]: https://github.com/notaryproject/notary https://github.com/notaryproject/notary [2]: https://github.com/notaryproject/notation https://github.com/notaryproject/notation
  • pc86 5y ago
    Are you me? I just started a new job a few weeks ago, and guess what, I'm the first one (in my small group of ~5 or so) to have an M1. First it was just figuring out what the issue was, then trying UTM and manual port forwarding and getting horrible performance, then Colima which seems passable, but the default 2GB of RAM is useless for anything like kafka, so I end up having to allocate half the RAM of my machine just to docker containers.
    • 411111111111111 5y ago
      I find it hilarious that you're both seemingly blame docker for these issues.
      • pc86 5y ago
        I don't blame anyone for the issue. Well, maybe Apple itself but even then blame is pretty harsh. But from someone coming from 12+ years of software development on Windows, with occasional Mac usage personally, to suddenly doing development on a Mac, it was a big shock (mostly in a good way).
• lyime 5y ago
  How is this different from buildkite
• Xenixo 5y ago
  Ci/CD is transforming right now but I don't think dagger is solving those things or I don't understand it well enough yet. GitHub action is a game changer. CICD finally has a proper UI were it belongs: on the repository. GitHub actions are also much easier than gitlab runner. The basic actions are great and probably solve 90% of all normal use cases. Then we/I see a big trend of going away from self build pipelines. Provide Standard Features through convention over configuration: the docker image build action checks if it need to run on the pipeline and runs isolated by default. The build packs are providing something like this but they are to clumsy/complex. Then you have the real issue no one is solving: - fast sync between stages and builds. 1 gb of source code, maven cache etc. Still need effort to do it fast. Preferable a filesystem which is fast and allows to share snapshots you can mount ober the network (it's not a big issue but puts a latency limit on how fast you can provide build results and I like it fast) - standard building blocks with retry mechanism. When a 3h build fails, restarting is annoying like hell. Pipelines have very little resilience. - unified CICD output: lots of plugins support Jenkins Replacing Jenkins needs a new UI (GitHub has it now) which still is not unified. There is a thing going on to univy on GitHub through GitHub checks. I think in the next few years GitHub will have solved those issues for us. They do exactly what you are looking for. Would love to work with them. They even now inject the repository secret for you. They are slow (hello personal access token alternative we are waiting...) But steady. Btw. Before someone says gitlab: the Auto DevOps thing was shitty and bad supported. A default case with java and postgres was/is not fixed for 3 years. And those vulnerabilities. The last few years? Nope nope nope.
• fpopa 5y ago
  Congrats on the launch. Hopefully this will be the last pipelining language I'll learn.
• throwaway787544 5y ago
  We've really jumped the shark with people not understanding the word DevOps, haven't we? When the Docker people don't get it, we should probably give up. Fine. "DevOps" now means "some something development something something servers something something operations". Are you happy now, tech world? You've made Patrick Debois cry.
  • pid-1 5y ago
    It's simple. Before we had developer and operations teams. Now operations write yaml and we call them devops.
  • liammclennan 5y ago
    I gave up when a friend defined DevOps as "the entire software development lifecycle". There is a trend of this sort of regression to the mean. Vendors want to jump on a keyword, so they fight to expand the definition to include their thing. Pretty soon everything is "DevOps".
  • deleted 5y ago
    [deleted]
  • pnt12 5y ago
    Honest question: how would you define it? This question has come up in my team and everyone has a different definition for it.
    • throwaway787544 5y ago
      It has a definition in the same sense that "Agile" and "Lean" and "Six Sigma" have a definition. The definition isn't important, it's the group of ideas that the word refers to.
• solarengineer 5y ago
  I wonder if there is still opportunity to stop saying “CI/CD”. Here is a talk that I gave last year: https://youtu.be/BJrxYuqG64Q https://youtu.be/BJrxYuqG64Q “Why saying ‘CI/CD’ is not enough”
  • shykes 5y ago
    I agree and I think we will get there. I see CI/CD as one of several “micro-categories” that no longer makes sense standalone, and will gradually merge into a single mega-category. Those include, off the top of my head: continuous integration, continuous deployment , configuration management, infrastructure management, PaaS, IT automation.
• shadycuz 5y ago
  I really wish you would have stuck with a language that is higher in popularity. Python, JavaScript or such.
  • shykes 5y ago
    We did. Each action can be written in the language of your choice (I personally use bash) then wrapped in a declarative CUE config (typically quite short). Then you compose declaratively in cue. Here is an example of a package to deploy to netlify: https://github.com/dagger/dagger/tree/main/pkg/universe.dagger.io/netlify https://github.com/dagger/dagger/tree/main/pkg/universe.dagg...
    • ghostdog20 5y ago
      Have you heard of or explored https://github.com/aws/constructs https://github.com/aws/constructs (related: https://github.com/aws/jsii https://github.com/aws/jsii and https://github.com/aws/aws-cdk https://github.com/aws/aws-cdk)? This is what CDK uses for declarative modeling, but gives the opportunity to use languages/tooling that most devs are already familiar with. CDK8s already uses it as a replacement for yaml (technically, the yaml becomes an implementation detail rather than actually replaced)
• shadycuz 5y ago
  We run most of our CI locally already using vscode devcontainers and precommit. It seems like that is a DIY dagger. Since most of our precommit hooks are just docker containers themselves. I guess dagger does add parallel builds.
  • shykes 5y ago
    Yes, Dagger can be integrated in a devcontainers workflow. How much or how little is done by Dagger is up to you.
• charsi 5y ago
  >.. by composing reusable actions, using an intuitive _declarative language_ and API. Actions can safely be shared and reused thanks to a _complete package management system_ .. That sounds horrible to me. Yet another abstraction layer and set of tools to do something that should be simple. Shell scripts inherently aren't bad. And I really don't see why they'll need more maintenance than using this new tool.
• crossingTime 5y ago
  I thought that maybe the dependency injection tool was expanding into new areas.. https://dagger.dev/ https://dagger.dev/
• mshekow 5y ago
  How would Dagger compare to Toast? https://github.com/stepchowfun/toast https://github.com/stepchowfun/toast
  • shykes 5y ago
    It seems very similar. The main difference (from a cursory look at their docs) is that toast uses YAML as a frontend, and Docker Engine as a backend. Whereas Dagger uses CUE as the frontend, and Buildkit as a backend.
• swozey 5y ago
  Worst company name I've experienced in years. Absolute embarassment. How do I introduce an app named "Dagger" into my stack without sounding caustic or ignorant to my staff? How do you make a service called this without feeling caustic or ignorant. And you have investors? Are you serious? Did you think Mailgun was a good name? edit: Figures Paul Graham is pimping this trash. Good work Sykes, you found a way to float in rich-man-land for another 5 years. I'll never be there but I'm perfectly content living a normal human life. Try it sometime, it may give you some insight into developer tooling and how to profit off of it.
  • cabbagehead 5y ago
  • klohto 5y ago
    It’s not the name that’s the problem with your introduction it seems.
    • swozey 5y ago
      It's not my introduction we're critiquing here. I don't give one fuck what Paul Graham invests in and I can't wait until he realizes nobody younger than him who isn't desperate to be a CEO-bro gives a shit about his opinion.
• pjmlp 5y ago
  Nice choice of naming, really no research done on it? https://dagger.dev/ https://dagger.dev/ https://developer.android.com/training/dependency-injection/dagger-basics https://developer.android.com/training/dependency-injection/... Now good luck searching for using Dagger CI/CD for Android development.
• loevborg 5y ago
  One of the big benefits of using a cloud build server like CircleCI is that it allows me to parallelize tasks. For example: - Build code and lint code at the same time - Run slow end-to-end tests in parallel Is parallelization possible with Dagger, even in principle?
  • chrismatheson 5y ago
    I think this comment from the OP answers your Q ? :) https://news.ycombinator.com/item?id=30859125 https://news.ycombinator.com/item?id=30859125
    • loevborg 5y ago
      Thanks for the pointer! I still don't understand 100%. The link states that tasks are parallelized automatically. At the same time, Dagger AIUI is intended to run on top of existing build servers like Jenkins, Circle CI or Github Actions. But then I would assume that some sort of integration between Dagger and the build server needs to be in place so that tasks are parallelized on multiple worker machines (rather than all running in parallel on a single machine). If everything is running inside of a single CircleCI job, that job doesn't have enough cores to run all the e2e tests in parallel. I guess my question is how this integration works in practice and what kind of complexity it generates.
      • shykes 5y ago
        Good question. Let me provide a more detailed answer in terms of concurrency and parallelism. To paraphrase Rob Pike's excellent explanation: - Concurrency is the breaking down of a program into pieces that can be executed independently. - Parallelism is the simultaneous execution of multiple things (possibly related, possibly not) Dagger is designed to be highly concurrent with minimal development effort. Compared to an equivalent configuration in a traditional CI system, your Dagger pipelines will be more concurrent, and require less lines of code. Because it is highly concurrent, Dagger can be parallelized with relatively little effort. But as you pointed out, you still need to configure parallelization by setting up multiple nodes, etc. Dagger uses buildkit as an excution engine, so parallelizing Dagger boils down to parallelizing buildkit. There is a lot of work in this area (one benefit of building on an existing, mature ecosystem). For example, here is an example of Kubernetes deployment: https://github.com/moby/buildkit/tree/master/examples/kubernetes https://github.com/moby/buildkit/tree/master/examples/kubern... Note that, because of its highly concurrent design and because of built-in caching, a single node may get you further than you might think. For example, in the e2e testing example: instead of running every test for every new commit, like many CI systems do, Dagger will automatically cache tests with unchanged inputs. So, for example, a change in the documentation will not trigger the API tests; a change in the iOS app source code will not trigger the Android tests; etc. It's not uncommong for CI runs to become faster after switching to Dagger, without any additional parallelism. As it turns out, most CIs are very wastesful and leave plenty of low-hanging fruits to pick. Dagger helps you pick them :)
• Annatar 5y ago
• Kalanos 5y ago
  https://github.com/JuliaParallel/Dagger.jl https://github.com/JuliaParallel/Dagger.jl
  • Hasnep 5y ago
    To clarify, this post is unrelated to Dagger.jl which is a parallel computing package for Julia.
• tianzhou 5y ago
  What if we call it "democratize CI/CD" ;)
• hbogert 5y ago
  what highly critical comments so far. Every stipulated problem resonates so well for me. I don't really care what the solution is or will be, they are addressing the right problem imo, which is overlooked in the CI/builder space. So many days are wasted for me because of the ridiculous back and forth to the buildserver when adding or changing automation. My experience is mostly with Gitlab, and I dare to say it's an absolute business strategy for them to lock you in, into the buildserver. They've had the option to use local build runners, but that one lacks fundamental functionality which results in you having to change the Yaml file locally, temporarily, just to extrapolate templates and such. It's horrible. There are so many tickets asking for feature parity for this local-runner and the actual runner on the build server; it's simply ignored or false promises are made. Afaics, Github actions won't do any better. I do expect Dagger integrates well with Gitlab such that I can download logs, artifacts and such from the Gitlab UI. But I'm afraid if something like Dagger catches on, the existing CI platforms will gatekeep functionality through pure yaml instead of allowing API calls from your pipeline. That would mean something like dagger will have to rely on compiling to a backend language, probably YAML. However, when did the 'one-ring-to-rule-them-all' approach ever work? So, basically my question is, what's the incentive for the current CI providers to play nicely? And isn't it trying to follow multiple moving targets?
• godfryd2 5y ago
  Example of integration of Dagger in Kraken CI: https://lab.kraken.ci/runs/4393/jobs https://lab.kraken.ci/runs/4393/jobs