7 ms·

Hi everyone, I'm one of the co-founders of Dagger (and before that founder of Docker). This is a big day for us, it's our first time sharing something new since

by shykes 5y ago

Hi everyone, I'm one of the co-founders of Dagger (and before that founder of Docker). This is a big day for us, it's our first time sharing something new since, you know... Docker. If you have any questions, I'll be happy to answer them here!

bovermyer 5y ago
So basically, if I want to not write Jenkinsfiles but still use my company's existing Jenkins installation, I can use Dagger?
- shykes 5y ago
  Yes :) You can write one last Jenkinsfile that runs Dagger, then do everything inside Dagger. Then you can run the exact same Dagger configuration on another CI system, or on your laptop. All you need is a Docker-compatible runtime (we run everything on buildkit under the hood).
  - bovermyer 5y ago
    Man, I am SO excited for that! Kudos!!
  - chologrande 5y ago
    > You can write one last Jenkinsfile that runs Dagger I'm very confused by this sentiment. This approach loses the best of existing CI tooling does it not? Jenkins users lose understanding of what's actually being executed, what stage they're in, and how long it took. It might provide convenience, and the a (great) benefit of running locally the same as in your CI environment, but it seems to me this would make it difficult for devs to easily understand where/why their build failed, since it just has one megastep.
    - ______-_-______ 5y ago
      I assume if the jenkins-step failed, you'd click a link to the dagger UI to see which dagger-step failed. Alternatively, never open jenkins at all and instead keep a tab open with the dagger UI.
      - shykes 5y ago
        Yes, that’s right. Especially since you’re probably running the same dagger actions all day long in development, upstream from CI.
        chologrande 5y ago
        I guess I'm a little confused where the line here... Where is dagger UI and how does it relate to your CI? I don't see it in docs or cli help. Sounds like Dagger UI in this context (above) is providing little value beyond logging if it's not doing workflow execution. I ask not to talk down on the product, but because I'm actually quite interested. Local execution, plus containerized execution sounds awesome. Just trying to understand the vision.
    - shykes 5y ago
      It can be a “megastep” in jenkins, but it doesn’t have to. It could be one individual step that happens to run on Dagger. Both work equally well. In the “megastep” approach, it boils down to which tool can provide the most useful information. Jenkins is more mature but Dagger has more information about the DAG. So in some cases developers might actually prefer using Jenkins as a “dumb” runner infrastructure. It depends on the situation.
- verdverm 5y ago
  You should be able to drastically simplify you Jenkinsfile(s) and have them just invoke Dagger. The issue you may run into is when you have different Jenkins nodes for different types of work. You could always invoke Dagger on each of these, depending on your setup and needs. Where there is a will, there is a way, with Jenkins :]
topher200 5y ago
Congrats on launching! How mature is this? We have a 20 person team and we're prototyping different pipelines for our next CI/CD pipeline (currently Heroku). Is this ready for production workloads?
- shykes 5y ago
  We consider Dagger to be beta-quality. It definitely has bugs, and APIs are still moving, though we make a big effort to limit breaking changes, and help developers migrate when breaking changes do occur. We aim to be able to guarantee complete peace of mind and continuity for production pipelines, but can't make that guarantee yet. That said, one nice aspect of Dagger is that you don't need to rip and replace your entire system: you can use it in a localized way to solve a specific problem. Then expand over time. It's similar to writing a quick shell script, except it's easier to reuse, refactor and compose over time.
asim 5y ago
Congrats on getting this far with the new venture and good luck. Wishing you all the success in the world!
ghawk1ns 5y ago
Dagger is already a popular dependency injection framework, so why choose a name that will be confusing to people who will likely use both of these frameworks in their projects?
- TimWillebrands 5y ago
  These comments are extremely uninteresting. If you care so much, please just create an issue on their bugtracker or something.
  - deleted 5y ago
    [deleted]
  - kevinb9n 5y ago
    "If you have any questions, I'll be happy to answer them here!"
    - guelo 5y ago
      Except this question.
- guelo 5y ago
  And the reason they're both called Dagger is as a play on using a DAG, directed acyclic graph, to model dependencies. Stealing that wittiness and pretending it's their own is pathetic.
  - hetspookjee 5y ago
    I believe the required creativity for that is not that high as it doesn’t seem too far fetched. I’d rather call it a piece of convergent evolution.
candiddevmike 5y ago
What's the monetization strategy going to be?
- shykes 5y ago
  There will be an optional cloud service (not yet available). Its features will be based on requests from the community. Some problems just can't be solved with a command-line client. For example: visualization of your pipelines; a centralized audit log of all jobs run across all machine; centrally managed access control and policies; etc. We will not rely on unusual licences to restrict competitors from running Dagger as a service. We don't need to, since `dagger` is a client tool. We do encourage cloud providers to run buildkit as a service, though :) Generally, we take inspiration from Red Hat for their balancing of open-source community and business. They are very open with their code, and tightly control how their trademark is used. You can clone their IP and use it to compete with them - but you have to build your own brand, and you can't confuse and fragment the Dagger developer community itself. We think that is a fair model, and we are using it as a general guideline.
  - EddieRingle 5y ago
    > We will not rely on unusual licences to restrict competitors from running Dagger as a service. Your "Trademark Guidelines" appear to contradict you: > Third-party products may not use the Marks to suggest compatibility or interoperability with our platform. For example, the claims “xxx is compatible with Dagger”, “xxx can run your Dagger configurations”, are not allowed. > but you have to build your own brand, and you can't confuse and fragment the Dagger developer community itself If I do an incognito Google search for "dagger", the first result is the Wikipedia page for the knife, and the second result is for Dagger, the dependency injection tool. By naming this "Dagger" you're confusing not just your own developer community but the pre-existing one as well.
    - shykes 5y ago
      > > We will not rely on unusual licences to restrict competitors from running Dagger as a service. > Your "Trademark Guidelines" appear to contradict you: They do not. Software licenses and trademark guidelines are two different things. Some commercial open-source vendors have changed their licenses to restrict use of the software in various ways - typically to limit competition from large cloud providers. We don't do that, and have no intention to. Our license is OSI-approved and we intend to keep it that way. That is what I am referring to. > but you have to build your own brand, and you can't confuse and fragment the Dagger developer community itself This is the intent behind the language in the trademark guideline which you quoted: you can redistribute and modify our code. But if you distribute a modified copy, call it something else. > > Third-party products may not use the Marks to suggest compatibility or interoperability with our platform. For example, the claims “xxx is compatible with Dagger”, “xxx can run your Dagger configurations”, are not allowed. > By naming this "Dagger" you're confusing not just your own developer community but the pre-existing one as well. I disagree. Dagger has existed in private beta for over a year, thousands of engineers have been given access, and I can't remember a single instance of any of them being confused by the name. We have registered the trademark, and nobody has raised an issue.
shredder25 5y ago
Congratulations! I know exactly how this tool will benefit us DevOps Engineers as I knew when you did a demo of Docker at PyCon 2013, wishing you and your team the best!
mhitza 5y ago
Hi! I've browsed the docs quickly, and I have a few questions. Seems to assume that all CI/CD workflows work in a single container at a time pattern. How about testing when I need to spin up an associated database container for my e2e tests. Is it possible, and just omitted from the documentation? Not familiar with cue, but can I import/define a common action that is used across multiple jobs? For example on GitHub I get to duplicate the dependency installation/caching/build across various jobs. (yes, I'm aware that now you can makeshift on GitHub a composite action to reuse) Can you do conditional execution of actions based on passed in input value/env variable? Any public roadmap of upcoming features?
- shykes 5y ago
  > Seems to assume that all CI/CD workflows work in a single container at a time pattern. Dagger runs your workflows as a DAG, where each node is an action running in its own container. The dependency graph is detected automatically, and all containers that can be parallelized (based on their dependencies) will be parallelized. If you specify 10 actions to run, and they don't depend on each other, they will all run in parallel. > How about testing when I need to spin up an associated database container for my e2e tests. Is it possible, and just omitted from the documentation? It is possible, but not yet convenient (you need to connect to an external docker engine, via a docker CLI wrapped in a container) We are working on a more pleasant API that will support long-running containers (like your test DB) and more advanced synchronization primitives (wait for an action; terminate; etc.) This is discussed in the following issues: - https://github.com/dagger/dagger/issues/1337 https://github.com/dagger/dagger/issues/1337 - https://github.com/dagger/dagger/issues/1249 https://github.com/dagger/dagger/issues/1249 - https://github.com/dagger/dagger/issues/1248 https://github.com/dagger/dagger/issues/1248 > Not familiar with cue, but can I import/define a common action that is used across multiple jobs? Yes! That is one of the most important features. CUE has a complete packaging system, and we support it natively. For example here is our "standard library" of CUE packages: https://github.com/dagger/dagger/tree/main/pkg https://github.com/dagger/dagger/tree/main/pkg > For example on GitHub I get to duplicate the dependency installation/caching/build across various jobs. (yes, I'm aware that now you can makeshift on GitHub a composite action to reuse) Yes code reuse across projects is where Dagger really shines, thanks to CUE + the portable nature of the buildkit API. Note: you won't need to configure caching though, because Dagger automatically caches all actions out of the box :) > Can you do conditional execution of actions based on passed in input value/env variable? Yes, that is supported. > Any public roadmap of upcoming features? For now we rely on raw Github issues, with some labels for crude prioritization. But we started using the new Github projects beta (which is a layer over issues), and plan to open that to the community as well. Generally, we develop Dagger in the open. Even as a team, we use public Discord channels (text and voice) by default, unless there is a specific reason not to (confidential information, etc.)
zepn 5y ago
Thanks for answering Qs. Does this compete directly with Tekton ( https://tekton.dev/ https://tekton.dev/ ), or do you imagine a way the two could interoperate? Why choose Dagger over Tekton to power pipelines?
- shykes 5y ago
  You can (and people do) run Dagger on top of Tekton, in the same way that you might run a Makefile or shell script on top of Tekton. The benefit is that you are less tied to a particular runtime environment. The same Dagger pipeline can run on Tekton, Jenkins, or your laptop. This makes local debugging and testing in particular much easier.
drewm1980 5y ago
The carbon footprint of the cloud has exceeded the footprint of air travel. A move away from monolithic statically compiled binaries to constellations of microservices (usually bloated docker containers) is a significant part of the problem. Docker's explosive growth is partly due to the convenience of the abstraction it provides, abstracting the entire linux userspace, putting even OS-wide package managers and language-specific package managers inside another box. This usually breaks any caching / code sharing that the now containerized packages had, resulting in the bloat. The docker image is portable, yes, but disk and RAM efficiency of the systems people are building are awful. It has been the norm for every little microservice to add a few GB of bloat to the overall software system. A dev writes "RUN pip install pytorch" and you have CICD servers pulling down 2GB of pytorch to build the container, every time the software is built, probably forever. Meanwhile species are going extinct and a lot of people are starting to wonder if it's ethical to work in technology at all. What can your team do to reverse this tragedy of the commons? Can you come up with some equally ergonomic tool that can migrate the container ecosystem on to something that has a solid foundation with good caching?
- cakoose 5y ago
  > What can your team do to reverse this tragedy of the commons? It's tragedies all the way down :-) Making a successful tool in a competitive space is hard enough. Asking a creator to somehow factor in environmental impact isn't going to work. A creator that places additional constraints on themselves will more likely lose out to a competitor that doesn't. This is the kind of thing a carbon tax is perfect for. Cost optimization is infused into how businesses and individuals operate. Tax the things you care about and watch all that machinery get to work! Right now, people use Docker (and dynamic languages, bloated JS libraries, etc) because burning electricity is the cheapest route. (Sort of... the market's obviously not perfect.) Make electricity expensive enough and we'll get more efficient systems. And of course taxes aren't the only solution. For example, Google Flights now shows the carbon cost of each flight option, which I think might actually move the needle. But that category of solution might only work in an industry with similar characteristics, e.g. infrequent but large purchases, mature industry that can define measurement standards.
littleroot 5y ago
Are you guys aware of Nix, both the language and the build system? Nix at its core is a build system, but the community pushed the boundary of what a "build" means so hard, now Nix could also be used as one definition language for everything in a CI/CD pipeline (also with a canonical collection of "building blocks" in nixpkgs), from (reproducibly) building artifacts to running automated testing/integrating tasks to automatically deliver the artifacts to whatever the "infrastructure" is. After all in a very general sense the whole CI/CD pipeline could be seen as just another build artifact, which I think resonates a lot with your idea. How do you think your project and Nix would overlap and/or (or both) complement each other?

by shykes 5y ago

bovermyer 5y ago

So basically, if I want to not write Jenkinsfiles but still use my company's existing Jenkins installation, I can use Dagger?

shykes 5y ago
Yes :) You can write one last Jenkinsfile that runs Dagger, then do everything inside Dagger. Then you can run the exact same Dagger configuration on another CI system, or on your laptop. All you need is a Docker-compatible runtime (we run everything on buildkit under the hood).
- bovermyer 5y ago
  Man, I am SO excited for that! Kudos!!
- chologrande 5y ago
  > You can write one last Jenkinsfile that runs Dagger I'm very confused by this sentiment. This approach loses the best of existing CI tooling does it not? Jenkins users lose understanding of what's actually being executed, what stage they're in, and how long it took. It might provide convenience, and the a (great) benefit of running locally the same as in your CI environment, but it seems to me this would make it difficult for devs to easily understand where/why their build failed, since it just has one megastep.
  - ______-_-______ 5y ago
    I assume if the jenkins-step failed, you'd click a link to the dagger UI to see which dagger-step failed. Alternatively, never open jenkins at all and instead keep a tab open with the dagger UI.
    - shykes 5y ago
      Yes, that’s right. Especially since you’re probably running the same dagger actions all day long in development, upstream from CI.
      - chologrande 5y ago
        I guess I'm a little confused where the line here... Where is dagger UI and how does it relate to your CI? I don't see it in docs or cli help. Sounds like Dagger UI in this context (above) is providing little value beyond logging if it's not doing workflow execution. I ask not to talk down on the product, but because I'm actually quite interested. Local execution, plus containerized execution sounds awesome. Just trying to understand the vision.
  - shykes 5y ago
    It can be a “megastep” in jenkins, but it doesn’t have to. It could be one individual step that happens to run on Dagger. Both work equally well. In the “megastep” approach, it boils down to which tool can provide the most useful information. Jenkins is more mature but Dagger has more information about the DAG. So in some cases developers might actually prefer using Jenkins as a “dumb” runner infrastructure. It depends on the situation.
verdverm 5y ago
You should be able to drastically simplify you Jenkinsfile(s) and have them just invoke Dagger. The issue you may run into is when you have different Jenkins nodes for different types of work. You could always invoke Dagger on each of these, depending on your setup and needs. Where there is a will, there is a way, with Jenkins :]

topher200 5y ago

Congrats on launching! How mature is this? We have a 20 person team and we're prototyping different pipelines for our next CI/CD pipeline (currently Heroku). Is this ready for production workloads?

shykes 5y ago
We consider Dagger to be beta-quality. It definitely has bugs, and APIs are still moving, though we make a big effort to limit breaking changes, and help developers migrate when breaking changes do occur. We aim to be able to guarantee complete peace of mind and continuity for production pipelines, but can't make that guarantee yet. That said, one nice aspect of Dagger is that you don't need to rip and replace your entire system: you can use it in a localized way to solve a specific problem. Then expand over time. It's similar to writing a quick shell script, except it's easier to reuse, refactor and compose over time.

asim 5y ago

Congrats on getting this far with the new venture and good luck. Wishing you all the success in the world!

ghawk1ns 5y ago

Dagger is already a popular dependency injection framework, so why choose a name that will be confusing to people who will likely use both of these frameworks in their projects?

TimWillebrands 5y ago
These comments are extremely uninteresting. If you care so much, please just create an issue on their bugtracker or something.
- deleted 5y ago
  [deleted]
- kevinb9n 5y ago
  "If you have any questions, I'll be happy to answer them here!"
  - guelo 5y ago
    Except this question.
guelo 5y ago
And the reason they're both called Dagger is as a play on using a DAG, directed acyclic graph, to model dependencies. Stealing that wittiness and pretending it's their own is pathetic.
- hetspookjee 5y ago
  I believe the required creativity for that is not that high as it doesn’t seem too far fetched. I’d rather call it a piece of convergent evolution.

candiddevmike 5y ago

What's the monetization strategy going to be?

shykes 5y ago
There will be an optional cloud service (not yet available). Its features will be based on requests from the community. Some problems just can't be solved with a command-line client. For example: visualization of your pipelines; a centralized audit log of all jobs run across all machine; centrally managed access control and policies; etc. We will not rely on unusual licences to restrict competitors from running Dagger as a service. We don't need to, since `dagger` is a client tool. We do encourage cloud providers to run buildkit as a service, though :) Generally, we take inspiration from Red Hat for their balancing of open-source community and business. They are very open with their code, and tightly control how their trademark is used. You can clone their IP and use it to compete with them - but you have to build your own brand, and you can't confuse and fragment the Dagger developer community itself. We think that is a fair model, and we are using it as a general guideline.
- EddieRingle 5y ago
  > We will not rely on unusual licences to restrict competitors from running Dagger as a service. Your "Trademark Guidelines" appear to contradict you: > Third-party products may not use the Marks to suggest compatibility or interoperability with our platform. For example, the claims “xxx is compatible with Dagger”, “xxx can run your Dagger configurations”, are not allowed. > but you have to build your own brand, and you can't confuse and fragment the Dagger developer community itself If I do an incognito Google search for "dagger", the first result is the Wikipedia page for the knife, and the second result is for Dagger, the dependency injection tool. By naming this "Dagger" you're confusing not just your own developer community but the pre-existing one as well.
  - shykes 5y ago
    > > We will not rely on unusual licences to restrict competitors from running Dagger as a service. > Your "Trademark Guidelines" appear to contradict you: They do not. Software licenses and trademark guidelines are two different things. Some commercial open-source vendors have changed their licenses to restrict use of the software in various ways - typically to limit competition from large cloud providers. We don't do that, and have no intention to. Our license is OSI-approved and we intend to keep it that way. That is what I am referring to. > but you have to build your own brand, and you can't confuse and fragment the Dagger developer community itself This is the intent behind the language in the trademark guideline which you quoted: you can redistribute and modify our code. But if you distribute a modified copy, call it something else. > > Third-party products may not use the Marks to suggest compatibility or interoperability with our platform. For example, the claims “xxx is compatible with Dagger”, “xxx can run your Dagger configurations”, are not allowed. > By naming this "Dagger" you're confusing not just your own developer community but the pre-existing one as well. I disagree. Dagger has existed in private beta for over a year, thousands of engineers have been given access, and I can't remember a single instance of any of them being confused by the name. We have registered the trademark, and nobody has raised an issue.

shredder25 5y ago

Congratulations! I know exactly how this tool will benefit us DevOps Engineers as I knew when you did a demo of Docker at PyCon 2013, wishing you and your team the best!

mhitza 5y ago

Hi! I've browsed the docs quickly, and I have a few questions. Seems to assume that all CI/CD workflows work in a single container at a time pattern. How about testing when I need to spin up an associated database container for my e2e tests. Is it possible, and just omitted from the documentation? Not familiar with cue, but can I import/define a common action that is used across multiple jobs? For example on GitHub I get to duplicate the dependency installation/caching/build across various jobs. (yes, I'm aware that now you can makeshift on GitHub a composite action to reuse) Can you do conditional execution of actions based on passed in input value/env variable? Any public roadmap of upcoming features?

shykes 5y ago
> Seems to assume that all CI/CD workflows work in a single container at a time pattern. Dagger runs your workflows as a DAG, where each node is an action running in its own container. The dependency graph is detected automatically, and all containers that can be parallelized (based on their dependencies) will be parallelized. If you specify 10 actions to run, and they don't depend on each other, they will all run in parallel. > How about testing when I need to spin up an associated database container for my e2e tests. Is it possible, and just omitted from the documentation? It is possible, but not yet convenient (you need to connect to an external docker engine, via a docker CLI wrapped in a container) We are working on a more pleasant API that will support long-running containers (like your test DB) and more advanced synchronization primitives (wait for an action; terminate; etc.) This is discussed in the following issues: - https://github.com/dagger/dagger/issues/1337 https://github.com/dagger/dagger/issues/1337 - https://github.com/dagger/dagger/issues/1249 https://github.com/dagger/dagger/issues/1249 - https://github.com/dagger/dagger/issues/1248 https://github.com/dagger/dagger/issues/1248 > Not familiar with cue, but can I import/define a common action that is used across multiple jobs? Yes! That is one of the most important features. CUE has a complete packaging system, and we support it natively. For example here is our "standard library" of CUE packages: https://github.com/dagger/dagger/tree/main/pkg https://github.com/dagger/dagger/tree/main/pkg > For example on GitHub I get to duplicate the dependency installation/caching/build across various jobs. (yes, I'm aware that now you can makeshift on GitHub a composite action to reuse) Yes code reuse across projects is where Dagger really shines, thanks to CUE + the portable nature of the buildkit API. Note: you won't need to configure caching though, because Dagger automatically caches all actions out of the box :) > Can you do conditional execution of actions based on passed in input value/env variable? Yes, that is supported. > Any public roadmap of upcoming features? For now we rely on raw Github issues, with some labels for crude prioritization. But we started using the new Github projects beta (which is a layer over issues), and plan to open that to the community as well. Generally, we develop Dagger in the open. Even as a team, we use public Discord channels (text and voice) by default, unless there is a specific reason not to (confidential information, etc.)

zepn 5y ago

Thanks for answering Qs. Does this compete directly with Tekton ( https://tekton.dev/ https://tekton.dev/ ), or do you imagine a way the two could interoperate? Why choose Dagger over Tekton to power pipelines?

shykes 5y ago
You can (and people do) run Dagger on top of Tekton, in the same way that you might run a Makefile or shell script on top of Tekton. The benefit is that you are less tied to a particular runtime environment. The same Dagger pipeline can run on Tekton, Jenkins, or your laptop. This makes local debugging and testing in particular much easier.

drewm1980 5y ago

The carbon footprint of the cloud has exceeded the footprint of air travel. A move away from monolithic statically compiled binaries to constellations of microservices (usually bloated docker containers) is a significant part of the problem. Docker's explosive growth is partly due to the convenience of the abstraction it provides, abstracting the entire linux userspace, putting even OS-wide package managers and language-specific package managers inside another box. This usually breaks any caching / code sharing that the now containerized packages had, resulting in the bloat. The docker image is portable, yes, but disk and RAM efficiency of the systems people are building are awful. It has been the norm for every little microservice to add a few GB of bloat to the overall software system. A dev writes "RUN pip install pytorch" and you have CICD servers pulling down 2GB of pytorch to build the container, every time the software is built, probably forever. Meanwhile species are going extinct and a lot of people are starting to wonder if it's ethical to work in technology at all. What can your team do to reverse this tragedy of the commons? Can you come up with some equally ergonomic tool that can migrate the container ecosystem on to something that has a solid foundation with good caching?

cakoose 5y ago
> What can your team do to reverse this tragedy of the commons? It's tragedies all the way down :-) Making a successful tool in a competitive space is hard enough. Asking a creator to somehow factor in environmental impact isn't going to work. A creator that places additional constraints on themselves will more likely lose out to a competitor that doesn't. This is the kind of thing a carbon tax is perfect for. Cost optimization is infused into how businesses and individuals operate. Tax the things you care about and watch all that machinery get to work! Right now, people use Docker (and dynamic languages, bloated JS libraries, etc) because burning electricity is the cheapest route. (Sort of... the market's obviously not perfect.) Make electricity expensive enough and we'll get more efficient systems. And of course taxes aren't the only solution. For example, Google Flights now shows the carbon cost of each flight option, which I think might actually move the needle. But that category of solution might only work in an industry with similar characteristics, e.g. infrequent but large purchases, mature industry that can define measurement standards.

littleroot 5y ago

Are you guys aware of Nix, both the language and the build system? Nix at its core is a build system, but the community pushed the boundary of what a "build" means so hard, now Nix could also be used as one definition language for everything in a CI/CD pipeline (also with a canonical collection of "building blocks" in nixpkgs), from (reproducibly) building artifacts to running automated testing/integrating tasks to automatically deliver the artifacts to whatever the "infrastructure" is. After all in a very general sense the whole CI/CD pipeline could be seen as just another build artifact, which I think resonates a lot with your idea. How do you think your project and Nix would overlap and/or (or both) complement each other?