14 ms·

I am really not convinced of the need of any incremental build system (like Bazel) over Nix excepted if you inherited of a giant mono-repo mess. Decomposing yo

• nh2 5y ago
  I think this view is too narrow; there is definitely need for fine-grained incremental build systems, even outside mono-repos. If you need to chase a bug into a more fundamental library, say libc, libpng, or your stack's favourite JSON library, iterating on that code with plain nix ("package-based non-incremental builds") will now take 20 hours per recompilation. With an incremental build system, it takes seconds, which is obviously a game changer. I've had to do this multiple times in the past, and had to employ big hacks to get around this limitation. My project also uses non-incremental nix, and an incremental build for our single-package top-level code. We just tolerate the waiting time, but it's not ideal. This works because our project is relatively small and updates to dependencies are not super frequent, so we can spend the time to build ad-hoc hacks to shortcut those builds each time. We find this better than any other currently available way, but it's not ideal. I'm not convinced that Bazel is the final answer to this (it focused almost entirely on its way to do things, ignoring the package-based world that makes up the open-source eco system). But there's definitely the need for something that addresses both sides, cleanly, and it hasn't been found yet.
  • adev_ 5y ago
    > If you need to chase a bug into a more fundamental library, say libc, libpng, or your stack's favourite JSON library, iterating on that code with plain nix ("package-based non-incremental builds") will now take 20 hours per recompilation. With an incremental build system, it takes seconds, which is obviously a game changer. I see the point but the '20 hour' build seems to me over-exaggerated. - Builds in Nix can be parallelized both on single node (multi-core, multi-socket) and multi-node without problems - Parallel builds 'just works' in Nix with the Nix's sandbox system and the hash-based package identification mechanism. Nothing related to the bug mess that can be a distcc deployment or parallel builds in Bazels. - Binary cache give a pretty efficient way to reuse any successful past build without problem - Modifications in "base" packages generally do not happen everyday scenarios. - Iterations and debugging without majors recompilations can be done with the nix shell directly. Full rebuild are required only for binary shipping. > I've had to do this multiple times in the past, and had to employ big hacks to get around this limitation. I had to recompile entire software stack ( > 1500 software ) for exotic archs (U.S supercomputers) in the past. My way was just to burst it on many nodes: compute time is cheap nowadays (and nodes are not missing on supercomputers :) ) . My time however is not cheap and most incremental builds system are sensible to side-effect (including Bazel) and not as reproducible as they claim to be: Debugging that is in my experience time consuming, much more than a bit a build time. I do see the interest of incremental builds inter-components, but in the current state of the tooling, it is for me not worth the pain. Specially not the pain of Bazel that has a (very) long list of quirks.
• q3k 5y ago
  > I am really not convinced of the need of any incremental build system (like Bazel) over Nix [...] Without smaller compilation units than the default mkDerivation-per-component-style in Nix you end up having _extremely_ long build/test cycles on every change. You could alleviate this by deferring to whatever build system is called in mkDerivation by providing a short-circuiting shell.nix for every subcomponent (eg. a shell.nix to drop you into somewhere where you can `cargo build`). But IMO that's not nearly as nice of an experience as a `nix-build -A some-component` or `bazel build //some/component`, especially when changes span multiple components. > You can also simulate incremental build in Nix-only by using multiple derivations over a single source tree and combine them together. Yes, you can put in the effort to replicate Bazel-style small-compilation-unit language rules in nix. But then your evaluation time will skyrocket, with nix spending dozen of seconds re-evaluating these tiny derivations before even getting to the build - this already happens when evaluating complex derivations like NixOS configuration builds. This also kills your build/test cycle. Plus, you end up polluting your /nix/store with thousands of tiny out-of-date artifacts that then need to be garbage collected (which also becomes slow when the graph grows). Bazel's build/evaluation model (workspace rules -> targets -> build graph -> action graph -> artifact cache) and the fact that it runs as a server makes it much easier to have snappy rebuilds out of the box without any special consideration for derivation granularity or crutches like shell.nix-per-component. I'm not saying Bazel is perfect, and it's quite difficult to get running in an organization if you don't know it well, but once it's set up correctly it's pretty much the best development workflow I've ever used. Nix is very good for packaging software that you barely change, eg. for packaging software, as it makes it very easy to hermetically build software that comes with its own build system. But from my experience it's not a great experience for building software you develop, as build times rapidly rise to the point of frustration. > [...] excepted if you inherited of a giant mono-repo mess. I'm not sure how to interpret this - both Nix and Bazel shine in monorepo setups (and are IMO a prerequisite to even consider using a monorepo). Even nixpkgs is its own monorepo, and thanks to that is probably the best contributing-to-a-Linux-distribution experience that exists, by a long shot. Are you instead using Nix with flakes accross multiple repositories? Does that alleviate evaluation times?
  • adev_ 5y ago
    > ou could alleviate this by deferring to whatever build system is called in mkDerivation by providing a short-circuiting shell.nix for every subcomponent (eg. a shell.nix to drop you into somewhere where you can `cargo build`). But IMO that's not nearly as nice of an experience as a `nix-build -A some-component` And you describe exactly what we do :) That's not that bad. The multi-components scenario can also be scripted through some specific nix-shell targets that you configure for a local source usage. It is not perfect but it does the job. I do consider that if multi-components modifications happen on regular base, it is anyway a sign of a problem: they should probably not be separated components in the first place. > But then your evaluation time will skyrocket, with nix spending dozen of seconds re-evaluating these tiny derivations before even getting to the build - this already happens when evaluating complex derivations like NixOS configuration builds We do that by using overlays for our package set ( > 250 packages) over the bigger package set (the entire nixpkgs) which is provided through Nix channels. That allows to leverage the Nix internal cache and keep evaluation under few seconds. Also we do use the nix-daemon. > Plus, you end up polluting your /nix/store with thousands of tiny out-of-date artifacts that then need to be garbage collected (which also becomes slow when the graph grows). That is true, but space is cheap :) > [...] I'm not saying Bazel is perfect, and it's quite difficult to get running in an organization if you don't know it well, but once it's set up correctly it's pretty much the best development workflow I've ever used. I understand the point of view but I have the opposite experience. Bazel is one of the worst build system I had to experience. The fact he tries to act both as a package manager and as a build system (CMake, make, configure) makes it conflict with almost everything else in the Open Source world which is not made from day 1 for Bazel: This, for me is unacceptable. I will also pass under silence the long list of quirks it has and the instability of the previous versions. > Even nixpkgs is its own monorepo, and thanks to that is probably the best contributing-to-a-Linux-distribution experience that exists, by a long shot. Nix requires only its configuration (Nix's expression) to be in monorepo. At the opposite, Bazel design is centered around the source code being in monorepo. That is a major difference. One centralize versioning, the other one centralize everything.