3 ms·
Good question. Let me provide a more detailed answer in terms of concurrency and parallelism. To paraphrase Rob Pike's excellent explanation: - Concurrency is
by shykes 5y ago
Good question. Let me provide a more detailed answer in terms of concurrency and parallelism. To paraphrase Rob Pike's excellent explanation:
- Concurrency is the breaking down of a program into pieces that can be executed independently.
- Parallelism is the simultaneous execution of multiple things (possibly related, possibly not)
Dagger is designed to be highly concurrent with minimal development effort. Compared to an equivalent configuration in a traditional CI system, your Dagger pipelines will be more concurrent, and require less lines of code.
Because it is highly concurrent, Dagger can be parallelized with relatively little effort. But as you pointed out, you still need to configure parallelization by setting up multiple nodes, etc. Dagger uses buildkit as an excution engine, so parallelizing Dagger boils down to parallelizing buildkit. There is a lot of work in this area (one benefit of building on an existing, mature ecosystem). For example, here is an example of Kubernetes deployment: https://github.com/moby/buildkit/tree/master/examples/kubernetes https://github.com/moby/buildkit/tree/master/examples/kubern...
Note that, because of its highly concurrent design and because of built-in caching, a single node may get you further than you might think. For example, in the e2e testing example: instead of running every test for every new commit, like many CI systems do, Dagger will automatically cache tests with unchanged inputs. So, for example, a change in the documentation will not trigger the API tests; a change in the iOS app source code will not trigger the Android tests; etc.
It's not uncommong for CI runs to become faster after switching to Dagger, without any additional parallelism. As it turns out, most CIs are very wastesful and leave plenty of low-hanging fruits to pick. Dagger helps you pick them :)