11 ms·

I'm so surprised that both Docker's as well as Google's solution to this problem involves 0 caching. Much of our container building includes fetching dependenc

I'm so surprised that both Docker's as well as Google's solution to this problem involves 0 caching. Much of our container building includes fetching dependencies from various package managers which rarely changes across builds, yet takes up the majority of the build time. Most of the time thats good enough, but for cases where we want to immediately deploy a fix, it can be quite frustrating to having to spend all this time seemingly unncessarily. Edit: Docker 1.13 has the new --cache-from option so it seems relatively straight-forward to do I guess?

skj 10y ago
One approach to this problem that I like is storing these dependencies in an intermediate image. Then, you can do an incremental build by referencing (rather than rebuilding) this intermediate image, and pay only the cost of downloading the raw data which should be cheap compared to the normal install process.
- gkop 10y ago
  Why not leverage Docker's built-in image layer caching?
  - kentt 10y ago
    I think the idea here is that the docker level caching isn't well suited for this. Eg for us we install all relevant gems with `bundle install`. If you update one gem, that the whole layer is rebuilt. Since the whole `bundle install` layer is rebuilt, every gem is fetched again from rubygems. Yarn works the same.
    - morgante 10y ago
      Having a separate FROM image doesn't help that situation either, since you have to rebuild it.
      - skj 10y ago
        Couldn't you FROM and then update? I'm not a ruby expert but I'd hope that it could be incremental.
  - puzzle 10y ago
    That might be less effective if you can't guarantee that all the builds end up always on the same node.
kmicklas 10y ago
Some century the industry will discover the NixOS model...
- mickronome 10y ago
  I like what I understand to be the core ideas of NixOS to be, but i have to say: There might be a chance that some century, people both in the industry and outside understand that new syntax, or languages, are rarely necessary, or even beneficial. Use different semantics if you must, but please use some existing syntax for your DSL. I'm so tired of learning a pointless new variety of almost the same thing, only with slightly different syntax, maybe some awkward string quoting, and some rather pointless syntactic sugar. I'm not saying NixOS is the worst offender, but innovating in (mostly) superficial syntax, or using obscure syntax is almost certainly not well spent time except for a few very narrow niches/contexts. Especially not for prospective users. I get it, it's fun inventing both languages and tools, I love doing it myself! However, and in general: Don't invent a language when you need a tool, and vice versa.
  - kmicklas 10y ago
    Agreed, Nix the language is probably unnecessary. (The main benefit I can think of is avoiding depending on the moving target that is basically any other scripting language they could have tried to co-opt.)
    - arghimonmobile0 10y ago
      I feel like I should plug Guix at this point. Pretty much similar to Nix, but the config language is Guile, which is Scheme.
    - dottedmag 10y ago
      To be fair, Nix language is completely different from other languages out there -- it employs only immutable data structures -- so using another language would not be possible. It does not excuse Nix to have such a braindamaged^W esoteric syntax.
jacobparker 10y ago
I'd give this a look: https://github.com/GoogleCloudPlatform/cloud-builders/tree/master/bazel https://github.com/GoogleCloudPlatform/cloud-builders/tree/m... (unfortunately key links to Bazel docs are 404'ing.) Bazel is a general purpose build system that focuses on reproducibility (same build inputs -> same build outputs) and strict dependency specification (at various levels: file, subsystem, external package fetching etc.) enforced by sandboxing which allows for fast incremental builds. Bazel is the open-source version of Google's internal build-system. More info here: https://bazel.build/ https://bazel.build/ How external resources are handled: https://bazel.build/versions/master/docs/external.html https://bazel.build/versions/master/docs/external.html Obviously Bazel requires some buy-in which may be unappealing (at least for existing projects: migration/supporting multiple build systems.) There are, however, serious benefits (and a larger, but nascent, ecosystem of related open-source tooling.) The biggest gap at the moment is supported languages/etc. (I don't work for Google, I just like Bazel.)
- ImJasonH 10y ago
  Thanks for pointing out the broken links, I'll fix those shortly. Until then, https://bazel.build/versions/master/docs/be/docker.html https://bazel.build/versions/master/docs/be/docker.html has the relevant information about Docker in Bazel.
- ravdovid 10y ago
  Thanks for pointing out the broken links, they are now fixed. (I'm on the Google team behind this project.)
- philip1209 10y ago
  We just open sourced our monorepo which uses Bazel - https://GitHub.com/staffjoy/v2 https://GitHub.com/staffjoy/v2 It has some major speed benefits. But it's not turnkey. Setting it up is hard. Its dep management in Go isn't compatible with lint, vet, etc. We never got node modules installing with Bazel.
  - skj 10y ago
    If you send me an email (skelterjohn at google dot com) I'd love to help you get things working with that repo and GCCB.
jzelinskie 10y ago
Full Disclosure: I work on the Quay team at CoreOS The Quay build system has implemented caching. The builder preemptively calculates the hashes of Dockerfile commands and then sends them to an API endpoint that finds the tag for the most similar tree of command hashes in a given repository. This tag is pulled before ever attempting to build the Dockerfile. Google's solution seems far more general than just building Dockerfiles. I'm glad they're pushing for more innovation in this space.

skj 10y ago

One approach to this problem that I like is storing these dependencies in an intermediate image. Then, you can do an incremental build by referencing (rather than rebuilding) this intermediate image, and pay only the cost of downloading the raw data which should be cheap compared to the normal install process.

gkop 10y ago
Why not leverage Docker's built-in image layer caching?
- kentt 10y ago
  I think the idea here is that the docker level caching isn't well suited for this. Eg for us we install all relevant gems with `bundle install`. If you update one gem, that the whole layer is rebuilt. Since the whole `bundle install` layer is rebuilt, every gem is fetched again from rubygems. Yarn works the same.
  - morgante 10y ago
    Having a separate FROM image doesn't help that situation either, since you have to rebuild it.
    - skj 10y ago
      Couldn't you FROM and then update? I'm not a ruby expert but I'd hope that it could be incremental.
- puzzle 10y ago
  That might be less effective if you can't guarantee that all the builds end up always on the same node.

kmicklas 10y ago

Some century the industry will discover the NixOS model...

mickronome 10y ago
I like what I understand to be the core ideas of NixOS to be, but i have to say: There might be a chance that some century, people both in the industry and outside understand that new syntax, or languages, are rarely necessary, or even beneficial. Use different semantics if you must, but please use some existing syntax for your DSL. I'm so tired of learning a pointless new variety of almost the same thing, only with slightly different syntax, maybe some awkward string quoting, and some rather pointless syntactic sugar. I'm not saying NixOS is the worst offender, but innovating in (mostly) superficial syntax, or using obscure syntax is almost certainly not well spent time except for a few very narrow niches/contexts. Especially not for prospective users. I get it, it's fun inventing both languages and tools, I love doing it myself! However, and in general: Don't invent a language when you need a tool, and vice versa.
- kmicklas 10y ago
  Agreed, Nix the language is probably unnecessary. (The main benefit I can think of is avoiding depending on the moving target that is basically any other scripting language they could have tried to co-opt.)
  - arghimonmobile0 10y ago
    I feel like I should plug Guix at this point. Pretty much similar to Nix, but the config language is Guile, which is Scheme.
  - dottedmag 10y ago
    To be fair, Nix language is completely different from other languages out there -- it employs only immutable data structures -- so using another language would not be possible. It does not excuse Nix to have such a braindamaged^W esoteric syntax.

jacobparker 10y ago

I'd give this a look: https://github.com/GoogleCloudPlatform/cloud-builders/tree/master/bazel https://github.com/GoogleCloudPlatform/cloud-builders/tree/m... (unfortunately key links to Bazel docs are 404'ing.) Bazel is a general purpose build system that focuses on reproducibility (same build inputs -> same build outputs) and strict dependency specification (at various levels: file, subsystem, external package fetching etc.) enforced by sandboxing which allows for fast incremental builds. Bazel is the open-source version of Google's internal build-system. More info here: https://bazel.build/ https://bazel.build/ How external resources are handled: https://bazel.build/versions/master/docs/external.html https://bazel.build/versions/master/docs/external.html Obviously Bazel requires some buy-in which may be unappealing (at least for existing projects: migration/supporting multiple build systems.) There are, however, serious benefits (and a larger, but nascent, ecosystem of related open-source tooling.) The biggest gap at the moment is supported languages/etc. (I don't work for Google, I just like Bazel.)

ImJasonH 10y ago
Thanks for pointing out the broken links, I'll fix those shortly. Until then, https://bazel.build/versions/master/docs/be/docker.html https://bazel.build/versions/master/docs/be/docker.html has the relevant information about Docker in Bazel.
ravdovid 10y ago
Thanks for pointing out the broken links, they are now fixed. (I'm on the Google team behind this project.)
philip1209 10y ago
We just open sourced our monorepo which uses Bazel - https://GitHub.com/staffjoy/v2 https://GitHub.com/staffjoy/v2 It has some major speed benefits. But it's not turnkey. Setting it up is hard. Its dep management in Go isn't compatible with lint, vet, etc. We never got node modules installing with Bazel.
- skj 10y ago
  If you send me an email (skelterjohn at google dot com) I'd love to help you get things working with that repo and GCCB.

jzelinskie 10y ago

Full Disclosure: I work on the Quay team at CoreOS The Quay build system has implemented caching. The builder preemptively calculates the hashes of Dockerfile commands and then sends them to an API endpoint that finds the tag for the most similar tree of command hashes in a given repository. This tag is pulled before ever attempting to build the Dockerfile. Google's solution seems far more general than just building Dockerfiles. I'm glad they're pushing for more innovation in this space.