3 ms·

> The reason the ecosystem maturity is so important for bazel is because its design encourages complete reimplementations. My experience from before Bazel is t

> The reason the ecosystem maturity is so important for bazel is because its design encourages complete reimplementations. My experience from before Bazel is that reimplementations were the way to go most times you had projects with multiple languages, because the alternatives lead you down dark paths—calling into a separate build system suffers from the same problems as recursive make. What excites me about 1.0 is the possibility that the third-party rules can mature. Its not too bad if you only need a couple sets of third-party rules but my experience is like yours-when a new Bazel release came out, you had to figure out when to upgrade based on your dependencies on third-party rules. If you had several of these, it made upgrading Bazel infeasible.

jonhohle 7y ago
Amazon’s build system handled this gracefully. Native build tool chains were always used, but wrapped in a relatively thin facade. As long as the buildtool could respond to a `release` target and put artifacts in a well known location it didn’t matter what the underlying implementation was. The biggest drawback that I could see was all dependencies and tool chain needed to be modeled, but once they were, incremental builds, dependency rebuilds, etc. all Just Worked™. Getting to that point requires a lot of upfront work, however, which can be hard to justify in smaller groups.
- m0zg 7y ago
  Can it do "bulletproof", repeatable incremental builds? I'm asking because I don't see how it would be able to, unless the toolchain is wired into your build system _at least_ as deeply as it is in Bazel.
  - aidenn0 7y ago
    Slightly OT, but Tup[1] handles this by sanitizing the environment and building in a FUSE filesystem to track reads and writes. It's bulletproof except for system header files (or more properly anything outside the root of your build, which usually is just system header files). Nix[2] accomplishes bulletproof repeatable builds by sandboxing everything, but I don't believe incremental builds are a design goal (though if you modularize everything into different nix expressions you will in fact get annoyingly hard to use bulletproof incremental builds). 1: http://gittup.org/tup/ http://gittup.org/tup/ 2: https://nixos.org/nix/ https://nixos.org/nix/
  - dastbe 7y ago
    So bazel and brazil solve similar but very distinct problems. bazel is solving the problem of fast, repeatable, "bulletproof" builds for (very large) monolithic repos, whereas brazil is solving for fast, repeatable, "bulletproof" builds for (very large) collections of repositories or packages. brazil is EXTREMELY agnostic to what happens within any given package build, with the associated tooling primarily focused on dependency artifact discovery. The (IMO) most important component of brazil is the version set, which is a versioned dependency closure that allows for one-off builds (so your auto-build application packages) as well as merges from upstream version sets (where library vendors release their code). they are the glue that makes the distributed, manyrepo software development model at Amazon feel safe and consistent. If I had my way, I'd combine the single codebase UX of bazel with the multi-code base dependency tracking of brazil, because i think they solve complimentary problems extremely well.
    - m0zg 7y ago
      Google just tests the living Jesus out of everything, and only versions a few core packages such as protobufs, grpc, and other packages used by pretty much everybody (this is called the "crust"). Everything else is tip of tree, and things are automatically re-tested using checked-in tests if they are affected by your changelist. You basically can't submit if you break anything. So in a way, Google doesn't need "versioning". Whatever is currently checked in is good to go. Tests are required, of course, and a Google reviewer won't let you submit anything if your tests suck. This, obviously, precludes the use of such a set-up in, shall we say, "more agile" orgs which don't have good test coverage. Blaze (at Google) is also not just a build system, but also an interface to a much larger distributed build and test backend, which lets you rebuild everything from the kernel upwards in seconds (by caching petabytes of build products at thousands of possible revisions), serves up source code views for developer workstations (code is not stored there either), and sustains the scale of distributed testing needed for this setup to work. As a result, nobody builds or tests on their own workstation, and there's close to zero (or maybe even zero, period) binaries checked into Google3 monorepo. If you need a Haskell compiler and nobody used it in a while, it'll be rebuilt from source and cached for future use. :-) Fundamentally, I think Google got things very, very right with Blaze. Bazel is but a pale shadow of what Blaze is, but even in its present state it is better than most (all?) other build systems.
Too 7y ago
It would be neat if all build systems had a standardized way to return their dependency graph (like gcc -MD) so they can be used recursively by other build systems. I'm not aware of any that does and there's far from a standard which allows easy chaining of multiple different sub build systems. Which unfortunately either leads to recursive make, remodeling the dependencies in two places or forcing everything into the same build system.

jonhohle 7y ago

Amazon’s build system handled this gracefully. Native build tool chains were always used, but wrapped in a relatively thin facade. As long as the buildtool could respond to a `release` target and put artifacts in a well known location it didn’t matter what the underlying implementation was. The biggest drawback that I could see was all dependencies and tool chain needed to be modeled, but once they were, incremental builds, dependency rebuilds, etc. all Just Worked™. Getting to that point requires a lot of upfront work, however, which can be hard to justify in smaller groups.

m0zg 7y ago
Can it do "bulletproof", repeatable incremental builds? I'm asking because I don't see how it would be able to, unless the toolchain is wired into your build system _at least_ as deeply as it is in Bazel.
- aidenn0 7y ago
  Slightly OT, but Tup[1] handles this by sanitizing the environment and building in a FUSE filesystem to track reads and writes. It's bulletproof except for system header files (or more properly anything outside the root of your build, which usually is just system header files). Nix[2] accomplishes bulletproof repeatable builds by sandboxing everything, but I don't believe incremental builds are a design goal (though if you modularize everything into different nix expressions you will in fact get annoyingly hard to use bulletproof incremental builds). 1: http://gittup.org/tup/ http://gittup.org/tup/ 2: https://nixos.org/nix/ https://nixos.org/nix/
- dastbe 7y ago
  So bazel and brazil solve similar but very distinct problems. bazel is solving the problem of fast, repeatable, "bulletproof" builds for (very large) monolithic repos, whereas brazil is solving for fast, repeatable, "bulletproof" builds for (very large) collections of repositories or packages. brazil is EXTREMELY agnostic to what happens within any given package build, with the associated tooling primarily focused on dependency artifact discovery. The (IMO) most important component of brazil is the version set, which is a versioned dependency closure that allows for one-off builds (so your auto-build application packages) as well as merges from upstream version sets (where library vendors release their code). they are the glue that makes the distributed, manyrepo software development model at Amazon feel safe and consistent. If I had my way, I'd combine the single codebase UX of bazel with the multi-code base dependency tracking of brazil, because i think they solve complimentary problems extremely well.
  - m0zg 7y ago
    Google just tests the living Jesus out of everything, and only versions a few core packages such as protobufs, grpc, and other packages used by pretty much everybody (this is called the "crust"). Everything else is tip of tree, and things are automatically re-tested using checked-in tests if they are affected by your changelist. You basically can't submit if you break anything. So in a way, Google doesn't need "versioning". Whatever is currently checked in is good to go. Tests are required, of course, and a Google reviewer won't let you submit anything if your tests suck. This, obviously, precludes the use of such a set-up in, shall we say, "more agile" orgs which don't have good test coverage. Blaze (at Google) is also not just a build system, but also an interface to a much larger distributed build and test backend, which lets you rebuild everything from the kernel upwards in seconds (by caching petabytes of build products at thousands of possible revisions), serves up source code views for developer workstations (code is not stored there either), and sustains the scale of distributed testing needed for this setup to work. As a result, nobody builds or tests on their own workstation, and there's close to zero (or maybe even zero, period) binaries checked into Google3 monorepo. If you need a Haskell compiler and nobody used it in a while, it'll be rebuilt from source and cached for future use. :-) Fundamentally, I think Google got things very, very right with Blaze. Bazel is but a pale shadow of what Blaze is, but even in its present state it is better than most (all?) other build systems.

Too 7y ago

It would be neat if all build systems had a standardized way to return their dependency graph (like gcc -MD) so they can be used recursively by other build systems. I'm not aware of any that does and there's far from a standard which allows easy chaining of multiple different sub build systems. Which unfortunately either leads to recursive make, remodeling the dependencies in two places or forcing everything into the same build system.