4 ms·

> Building Docker images in CI today is slow. CI runners are ephemeral, so they must save and load the cache for every build. >...persistent disks significantl

• jeremy_k 4y ago
  Yes, Buildkit allows for this with the `--mount=type=cache` functionality. [0]' Now as an end user you still have to add this to your Dockerfile, but if subsequent builds are able to continually use this cache, build times will drastically improve. - [0] https://docs.docker.com/build/cache/ https://docs.docker.com/build/cache/
  • jacobwg 4y ago
    --mount=type=cache is awesome! One of the reasons we created Depot is that cache mounts aren't supported in GitHub Actions, since each CI run is entirely ephemeral, so the files saved in a cache mount aren't saved across runs. BuildKit doesn't export those cache types via cache-to. There are some manual workarounds creating tarballs of the BuildKit context directory, but we wanted something that just works without needing to save/load tarballs, which can be quite slow.
    • deleted 4y ago
      [deleted]
    • e12e 4y ago
      How do you invalidate the cache / do a clean build? I've previously seen errors sneak in to docker builds due to cache - where perhaps an upstream docker image was gone, or other dependencies had changed (most likely to happen if a license is changed, or something is yanked due to a security issue).
      • jacobwg 4y ago
        We have two ways at the moment: 1. If you need an uncached individual build, you can pass the `--no-cache` flag to build everything from scratch 2. We have a "clear all cache" button in our UI and CLI command, to wipe the entire project's cache disk.
• jacobwg 4y ago
  Yes, all the layer cache is saved on a persistent volume that's reused between jobs. In that respect, it's very similar to running `docker build` on your local laptop, where each build is incremental. But in Depot's case, that incremental experience is shared for all CI builds, and between all users running `depot build` on their local devices.
  • jeremy_k 4y ago
    It sounds like the SSD provides the layer cache with `--cache-to=type=local`? Does Depot support connecting additional VMs to the volume if someone wants to scale up? I've been meaning to look into the new S3 cache to solve this issue.
    • jacobwg 4y ago
      Exactly, we effectively persist BuildKit's state directory, so it doesn't even need to tar and export any of the internal state or layer cache, it gets the same state volume for the next build. Today we're using vertical scaling, so we run BuildKit on larger EC2 instances, tune `max-parallelism`, and let BuildKit handle processing multiple builds / deduplicating steps across builds / etc. What our website calls a "project" basically equates to a new EC2 instance (so two different projects are fully isolated from each other). We'd like to expand into horizontal scaling, probably with some kind of tiered caching, so the builders would be able to use local cache on local SSDs if available, but fall back to copying cache from S3 if not available locally.
  • lxe 4y ago
    Awesome. How does this work in a distributed sense? For example, for multiple parallel builds where each build is building a different git sha?
    • jacobwg 4y ago
      Today we're vertically scaling builders, so multiple builds run on the same large EC2 instance. The instances support processing multiple Docker builds at once, and thanks to BuildKit can even deduplicate work between multiple parallel builds, so like if two simultaneous builds share some steps, it can handle coordinating so those steps run just once. We have plans to expand to more horizontal scaling with tiered caching, so we can keep the speedups we see today but further increase potential parallelism.