3 ms·

> Recently I put together a simple build rule that strips external dependencies, archives the rest, and uploads it to a cache. Then we can easily run that archi

by jfim 4y ago

> Recently I put together a simple build rule that strips external dependencies, archives the rest, and uploads it to a cache. Then we can easily run that archive against a pre-built container (which contains the external dependencies) in our cluster, enabling a very fast ML iteration loop on beefy cluster machines. Can you tell us more about this? We're using bazel at $day_job and it's about as pleasant as gouging my eyes out with a rusty spoon. Building docker images using bazel takes forever.

aschleck 4y ago
Sure! So the overall goal is to prebuild a runfiles tree containing all the external dependencies into a Docker container, and then when the user wants to run something we build a runfiles tree with all the non-external code. Then in the cluster we want to extract the user's runfiles tree on top of the prebuilt runfiles tree, and then execute the user's code. * I have an archive Starlark function that I use for both this and containers. It sets up a folder structure similar to <target>.runfiles with everything symlinked to the actual location, then it tars the whole thing following symlinks. It has a parameter to include files that start with external/ or not. * This archive function is used by my Bazel container rules, so I simply made a runner.py target that depends on every possible external Python dependency and made a Docker image with it. * I then made a Bazel rule that uses the archive function to archive a given executable without external/ and uploads it to a shared location. * At runtime runner.py is given the location as an argument, downloads it, extracts it, and then execv's it.
gravypod 4y ago
There's lot of improvements needed in rules_docker. I only really have time for approving PRs. Some OSS devs are working on better abstractions on top of things that should speed stuff up (especially for languages that build static binaries like C++ or Go).
radvani13 4y ago
Jim, this is Radhika, and I am the PM for Bazel. I would love to learn more about building docker images using Bazel and your painpoints

by jfim 4y ago

aschleck 4y ago

Sure! So the overall goal is to prebuild a runfiles tree containing all the external dependencies into a Docker container, and then when the user wants to run something we build a runfiles tree with all the non-external code. Then in the cluster we want to extract the user's runfiles tree on top of the prebuilt runfiles tree, and then execute the user's code. * I have an archive Starlark function that I use for both this and containers. It sets up a folder structure similar to <target>.runfiles with everything symlinked to the actual location, then it tars the whole thing following symlinks. It has a parameter to include files that start with external/ or not. * This archive function is used by my Bazel container rules, so I simply made a runner.py target that depends on every possible external Python dependency and made a Docker image with it. * I then made a Bazel rule that uses the archive function to archive a given executable without external/ and uploads it to a shared location. * At runtime runner.py is given the location as an argument, downloads it, extracts it, and then execv's it.

gravypod 4y ago

There's lot of improvements needed in rules_docker. I only really have time for approving PRs. Some OSS devs are working on better abstractions on top of things that should speed stuff up (especially for languages that build static binaries like C++ or Go).

radvani13 4y ago

Jim, this is Radhika, and I am the PM for Bazel. I would love to learn more about building docker images using Bazel and your painpoints