4 ms·

> Seems to assume that all CI/CD workflows work in a single container at a time pattern. Dagger runs your workflows as a DAG, where each node is an action runn

by shykes 5y ago

> Seems to assume that all CI/CD workflows work in a single container at a time pattern. Dagger runs your workflows as a DAG, where each node is an action running in its own container. The dependency graph is detected automatically, and all containers that can be parallelized (based on their dependencies) will be parallelized. If you specify 10 actions to run, and they don't depend on each other, they will all run in parallel. > How about testing when I need to spin up an associated database container for my e2e tests. Is it possible, and just omitted from the documentation? It is possible, but not yet convenient (you need to connect to an external docker engine, via a docker CLI wrapped in a container) We are working on a more pleasant API that will support long-running containers (like your test DB) and more advanced synchronization primitives (wait for an action; terminate; etc.) This is discussed in the following issues: - https://github.com/dagger/dagger/issues/1337 https://github.com/dagger/dagger/issues/1337 - https://github.com/dagger/dagger/issues/1249 https://github.com/dagger/dagger/issues/1249 - https://github.com/dagger/dagger/issues/1248 https://github.com/dagger/dagger/issues/1248 > Not familiar with cue, but can I import/define a common action that is used across multiple jobs? Yes! That is one of the most important features. CUE has a complete packaging system, and we support it natively. For example here is our "standard library" of CUE packages: https://github.com/dagger/dagger/tree/main/pkg https://github.com/dagger/dagger/tree/main/pkg > For example on GitHub I get to duplicate the dependency installation/caching/build across various jobs. (yes, I'm aware that now you can makeshift on GitHub a composite action to reuse) Yes code reuse across projects is where Dagger really shines, thanks to CUE + the portable nature of the buildkit API. Note: you won't need to configure caching though, because Dagger automatically caches all actions out of the box :) > Can you do conditional execution of actions based on passed in input value/env variable? Yes, that is supported. > Any public roadmap of upcoming features? For now we rely on raw Github issues, with some labels for crude prioritization. But we started using the new Github projects beta (which is a layer over issues), and plan to open that to the community as well. Generally, we develop Dagger in the open. Even as a team, we use public Discord channels (text and voice) by default, unless there is a specific reason not to (confidential information, etc.)

mhitza 5y ago
Thank you for the detailed response. I appreciate you taking the time. One last question/note. > Note: you won't need to configure caching though, because Dagger automatically caches all actions out of the box :) Is this strictly because it's using Docker underneath and layers can be reused? If so, unless those intermediary layers are somehow pushed/pulled by the dagger github action (or any associated CI/CD tool equivalent), experience on hosting server is going to be slow. Sidenote, around 2013 I've worked on a hacky custom container automation workflow within Jenkins for ~100 projects, and spent considerable effort in setting up policies to prune intermediary images. Thus on certain types of workflows without any prunning a local development machine can be polluted with hundreds of images, unless the user is specifically made aware of stale images. Does/will dagger keep track of the images it builds? I think a command like git gc could make sense.
- shykes 5y ago
  > > Note: you won't need to configure caching though, because Dagger automatically caches all actions out of the box :) > Is this strictly because it's using Docker underneath and layers can be reused? Not exactly: we use Buildkit under the hood, not Docker. When you run a Dagger action, it is compiled to a DAG, and run by buildkit. Each node in the DAG has content-addressed inputs. If the same node has been executed with the same inputs, buildkit will cache it. This is the same mechanism that powers caching in "docker build", but generalized to any operation. The buildkit cache does need to be persisted between runs for this to work. It supports a variety of storage backends, including posix filesystem, a docker registry, or even proprietary key-value services like the Github storage API. If buildkit supports it, Dagger supports it. Don't let the "docker registry" option confuse you: buildkit cache data isn't the same as docker images, so it doesn't carry the same garbage collection and tag pruning problems.
  - chippiewill 5y ago
    > Don't let the "docker registry" option confuse you: buildkit cache data isn't the same as docker images, so it doesn't carry the same garbage collection and tag pruning problems. IIRC doesn't buildkit store its cache data as fake layer blobs + manifest? I don't see how it can avoid the garbage collection and tag pruning problems since those are limitations of the registry implementation itself.

by shykes 5y ago

mhitza 5y ago

Thank you for the detailed response. I appreciate you taking the time. One last question/note. > Note: you won't need to configure caching though, because Dagger automatically caches all actions out of the box :) Is this strictly because it's using Docker underneath and layers can be reused? If so, unless those intermediary layers are somehow pushed/pulled by the dagger github action (or any associated CI/CD tool equivalent), experience on hosting server is going to be slow. Sidenote, around 2013 I've worked on a hacky custom container automation workflow within Jenkins for ~100 projects, and spent considerable effort in setting up policies to prune intermediary images. Thus on certain types of workflows without any prunning a local development machine can be polluted with hundreds of images, unless the user is specifically made aware of stale images. Does/will dagger keep track of the images it builds? I think a command like git gc could make sense.

shykes 5y ago
> > Note: you won't need to configure caching though, because Dagger automatically caches all actions out of the box :) > Is this strictly because it's using Docker underneath and layers can be reused? Not exactly: we use Buildkit under the hood, not Docker. When you run a Dagger action, it is compiled to a DAG, and run by buildkit. Each node in the DAG has content-addressed inputs. If the same node has been executed with the same inputs, buildkit will cache it. This is the same mechanism that powers caching in "docker build", but generalized to any operation. The buildkit cache does need to be persisted between runs for this to work. It supports a variety of storage backends, including posix filesystem, a docker registry, or even proprietary key-value services like the Github storage API. If buildkit supports it, Dagger supports it. Don't let the "docker registry" option confuse you: buildkit cache data isn't the same as docker images, so it doesn't carry the same garbage collection and tag pruning problems.
- chippiewill 5y ago
  > Don't let the "docker registry" option confuse you: buildkit cache data isn't the same as docker images, so it doesn't carry the same garbage collection and tag pruning problems. IIRC doesn't buildkit store its cache data as fake layer blobs + manifest? I don't see how it can avoid the garbage collection and tag pruning problems since those are limitations of the registry implementation itself.