6 ms·

We're using Bazel + Nix together to build Docker images containing Python apps and it's been working surprisingly well. Bazel builds the final image on top of a

We're using Bazel + Nix together to build Docker images containing Python apps and it's been working surprisingly well. Bazel builds the final image on top of a base image built by Nix (containing only Python sourced from Nix's repository). All hermetic and no Dockerfiles. I'm not a huge fan of Nix's expression language, but the system works well.

yjftsjthsd-h 5y ago
Wait, what's bazel giving you if you already use nix?
- jvolkman 5y ago
  Bazel is where I started. As an ex-Google employee it's familiar, and I prefer its build language. I found Nix when looking for ways to make Python builds reproducible across machines. Also, I get the impression that a Nix artifact is generally an entire library/package, whereas a Bazel artifact is often just one or a few files which fits well with how I prefer to use the build system. But I'm no Nix expert and that may not always be true. If Bazel had a better external dependency story (maybe someday [0]) then I'd probably not use Nix. But currently the pair seems to work well. [0] https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3Zu...
  - gravypod 5y ago
    I'm glad you're building containers. Are you using rules_docker or are you using nix to make the containers and just copying files from bazel-out?
    - jvolkman 5y ago
      The former. See an example I made for the sibling thread.
- WraithM 5y ago
  Nix doesn't have incremental builds. It's kinda all or nothing at the package level. This explains clearly why you'd want to combine both: https://www.tweag.io/blog/2018-03-15-bazel-nix/ https://www.tweag.io/blog/2018-03-15-bazel-nix/
xrd 5y ago
Is there any kind of writing about it? I'm very interested.
- jvolkman 5y ago
  Not sure of any writing that ties all of the pieces together, but I made a small example for you: https://github.com/jvolkman/bazel-nix-example https://github.com/jvolkman/bazel-nix-example
  - xrd 5y ago
    It's really fascinating. I filed a bug, but no need to respond unless you find it interesting. This is a really cool direction for me, and I really appreciate you putting this up.
  - wocram 5y ago
    It's striking how heavy the WORKSPACE is compared to everything else.
    - jvolkman 5y ago
      Yeah, I think it's due to the small size of the rest of the example. It wouldn't grow more than a few more lines for a sizeable python app.
thundergolfer 5y ago
We're doing a very similar thing. Having a hell of a time getting a Nix-built CUDA image to work with EKS GPUs though.
- domenkozar 5y ago
  There is data-science channel on Matrix and Discord!
  - dailyanchovy 5y ago
    Oh, care to link?
wocram 5y ago
With this stack, do you ever run binaries outside of the containers? Do you need to use the nix command line very much? I agree that nix is somewhat painful to deal with, but Bazel essentially has no package management story at all. It would be nice to see one of the tools subsume the other, but that doesn't feel like something either project is actively addressing (ie. package management in bazel, or incremental/non-bash-driven builds in nix).
- jvolkman 5y ago
  Sure, we can run commands using `bazel run` or build them using `bazel build` and then run the generated binaries from `bazel-bin` directly. In these cases we also use the same Nix-provided python version, but through a different path. The only time I use the nix CLI tools is when I'm fumbling around trying to make changes to one of our .nix files (we only have a few). Otherwise nix is hidden behind the scenes and I deal only with Bazel.
adev_ 5y ago
I am really not convinced of the need of any incremental build system (like Bazel) over Nix excepted if you inherited of a giant mono-repo mess. Decomposing your dependency tree into small derivation pose no problem with Nix. Mainly due to the fact it solves entirely the ABI problem and the diamond dependency problem. You can also simulate incremental build in Nix-only by using multiple derivations over a single source tree and combine them together. Using a gigantic pile of complexity like Bazel just for that seems kind of an overkill to me. No offense, I am just challenging here :)
- nh2 5y ago
  I think this view is too narrow; there is definitely need for fine-grained incremental build systems, even outside mono-repos. If you need to chase a bug into a more fundamental library, say libc, libpng, or your stack's favourite JSON library, iterating on that code with plain nix ("package-based non-incremental builds") will now take 20 hours per recompilation. With an incremental build system, it takes seconds, which is obviously a game changer. I've had to do this multiple times in the past, and had to employ big hacks to get around this limitation. My project also uses non-incremental nix, and an incremental build for our single-package top-level code. We just tolerate the waiting time, but it's not ideal. This works because our project is relatively small and updates to dependencies are not super frequent, so we can spend the time to build ad-hoc hacks to shortcut those builds each time. We find this better than any other currently available way, but it's not ideal. I'm not convinced that Bazel is the final answer to this (it focused almost entirely on its way to do things, ignoring the package-based world that makes up the open-source eco system). But there's definitely the need for something that addresses both sides, cleanly, and it hasn't been found yet.
  - adev_ 5y ago
    > If you need to chase a bug into a more fundamental library, say libc, libpng, or your stack's favourite JSON library, iterating on that code with plain nix ("package-based non-incremental builds") will now take 20 hours per recompilation. With an incremental build system, it takes seconds, which is obviously a game changer. I see the point but the '20 hour' build seems to me over-exaggerated. - Builds in Nix can be parallelized both on single node (multi-core, multi-socket) and multi-node without problems - Parallel builds 'just works' in Nix with the Nix's sandbox system and the hash-based package identification mechanism. Nothing related to the bug mess that can be a distcc deployment or parallel builds in Bazels. - Binary cache give a pretty efficient way to reuse any successful past build without problem - Modifications in "base" packages generally do not happen everyday scenarios. - Iterations and debugging without majors recompilations can be done with the nix shell directly. Full rebuild are required only for binary shipping. > I've had to do this multiple times in the past, and had to employ big hacks to get around this limitation. I had to recompile entire software stack ( > 1500 software ) for exotic archs (U.S supercomputers) in the past. My way was just to burst it on many nodes: compute time is cheap nowadays (and nodes are not missing on supercomputers :) ) . My time however is not cheap and most incremental builds system are sensible to side-effect (including Bazel) and not as reproducible as they claim to be: Debugging that is in my experience time consuming, much more than a bit a build time. I do see the interest of incremental builds inter-components, but in the current state of the tooling, it is for me not worth the pain. Specially not the pain of Bazel that has a (very) long list of quirks.

yjftsjthsd-h 5y ago

Wait, what's bazel giving you if you already use nix?

jvolkman 5y ago
Bazel is where I started. As an ex-Google employee it's familiar, and I prefer its build language. I found Nix when looking for ways to make Python builds reproducible across machines. Also, I get the impression that a Nix artifact is generally an entire library/package, whereas a Bazel artifact is often just one or a few files which fits well with how I prefer to use the build system. But I'm no Nix expert and that may not always be true. If Bazel had a better external dependency story (maybe someday [0]) then I'd probably not use Nix. But currently the pair seems to work well. [0] https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3Zu...
- gravypod 5y ago
  I'm glad you're building containers. Are you using rules_docker or are you using nix to make the containers and just copying files from bazel-out?
  - jvolkman 5y ago
    The former. See an example I made for the sibling thread.
WraithM 5y ago
Nix doesn't have incremental builds. It's kinda all or nothing at the package level. This explains clearly why you'd want to combine both: https://www.tweag.io/blog/2018-03-15-bazel-nix/ https://www.tweag.io/blog/2018-03-15-bazel-nix/

xrd 5y ago

Is there any kind of writing about it? I'm very interested.

jvolkman 5y ago
Not sure of any writing that ties all of the pieces together, but I made a small example for you: https://github.com/jvolkman/bazel-nix-example https://github.com/jvolkman/bazel-nix-example
- xrd 5y ago
  It's really fascinating. I filed a bug, but no need to respond unless you find it interesting. This is a really cool direction for me, and I really appreciate you putting this up.
- wocram 5y ago
  It's striking how heavy the WORKSPACE is compared to everything else.
  - jvolkman 5y ago
    Yeah, I think it's due to the small size of the rest of the example. It wouldn't grow more than a few more lines for a sizeable python app.

thundergolfer 5y ago

We're doing a very similar thing. Having a hell of a time getting a Nix-built CUDA image to work with EKS GPUs though.

domenkozar 5y ago
There is data-science channel on Matrix and Discord!
- dailyanchovy 5y ago
  Oh, care to link?

wocram 5y ago

With this stack, do you ever run binaries outside of the containers? Do you need to use the nix command line very much? I agree that nix is somewhat painful to deal with, but Bazel essentially has no package management story at all. It would be nice to see one of the tools subsume the other, but that doesn't feel like something either project is actively addressing (ie. package management in bazel, or incremental/non-bash-driven builds in nix).

jvolkman 5y ago
Sure, we can run commands using `bazel run` or build them using `bazel build` and then run the generated binaries from `bazel-bin` directly. In these cases we also use the same Nix-provided python version, but through a different path. The only time I use the nix CLI tools is when I'm fumbling around trying to make changes to one of our .nix files (we only have a few). Otherwise nix is hidden behind the scenes and I deal only with Bazel.

adev_ 5y ago

I am really not convinced of the need of any incremental build system (like Bazel) over Nix excepted if you inherited of a giant mono-repo mess. Decomposing your dependency tree into small derivation pose no problem with Nix. Mainly due to the fact it solves entirely the ABI problem and the diamond dependency problem. You can also simulate incremental build in Nix-only by using multiple derivations over a single source tree and combine them together. Using a gigantic pile of complexity like Bazel just for that seems kind of an overkill to me. No offense, I am just challenging here :)

nh2 5y ago
I think this view is too narrow; there is definitely need for fine-grained incremental build systems, even outside mono-repos. If you need to chase a bug into a more fundamental library, say libc, libpng, or your stack's favourite JSON library, iterating on that code with plain nix ("package-based non-incremental builds") will now take 20 hours per recompilation. With an incremental build system, it takes seconds, which is obviously a game changer. I've had to do this multiple times in the past, and had to employ big hacks to get around this limitation. My project also uses non-incremental nix, and an incremental build for our single-package top-level code. We just tolerate the waiting time, but it's not ideal. This works because our project is relatively small and updates to dependencies are not super frequent, so we can spend the time to build ad-hoc hacks to shortcut those builds each time. We find this better than any other currently available way, but it's not ideal. I'm not convinced that Bazel is the final answer to this (it focused almost entirely on its way to do things, ignoring the package-based world that makes up the open-source eco system). But there's definitely the need for something that addresses both sides, cleanly, and it hasn't been found yet.
- adev_ 5y ago
  > If you need to chase a bug into a more fundamental library, say libc, libpng, or your stack's favourite JSON library, iterating on that code with plain nix ("package-based non-incremental builds") will now take 20 hours per recompilation. With an incremental build system, it takes seconds, which is obviously a game changer. I see the point but the '20 hour' build seems to me over-exaggerated. - Builds in Nix can be parallelized both on single node (multi-core, multi-socket) and multi-node without problems - Parallel builds 'just works' in Nix with the Nix's sandbox system and the hash-based package identification mechanism. Nothing related to the bug mess that can be a distcc deployment or parallel builds in Bazels. - Binary cache give a pretty efficient way to reuse any successful past build without problem - Modifications in "base" packages generally do not happen everyday scenarios. - Iterations and debugging without majors recompilations can be done with the nix shell directly. Full rebuild are required only for binary shipping. > I've had to do this multiple times in the past, and had to employ big hacks to get around this limitation. I had to recompile entire software stack ( > 1500 software ) for exotic archs (U.S supercomputers) in the past. My way was just to burst it on many nodes: compute time is cheap nowadays (and nodes are not missing on supercomputers :) ) . My time however is not cheap and most incremental builds system are sensible to side-effect (including Bazel) and not as reproducible as they claim to be: Debugging that is in my experience time consuming, much more than a bit a build time. I do see the interest of incremental builds inter-components, but in the current state of the tooling, it is for me not worth the pain. Specially not the pain of Bazel that has a (very) long list of quirks.