5 ms·

Bazel may be 1.0 and the internal abstractions of dag solving etc are rock solid from years of use inside Google, but the ecosystem around Bazel is quite bad at

Bazel may be 1.0 and the internal abstractions of dag solving etc are rock solid from years of use inside Google, but the ecosystem around Bazel is quite bad at the moment. They've been breaking backwards compatibility constantly, so rules authors have struggled to keep up (hopefully done now that it's 1.0). The reason the ecosystem maturity is so important for bazel is because its design encourages complete reimplementations. The rust packaging rules for bazel reimplement a subset of cargo's features from scratch. The docker rules reimplement image building from scratch (they don't actually use docker). The (Google maintained) python rules shell out to pip, but it's got a ton of problems, some packages are unbuildable, and it busts the cache constantly and redownloads everything. If you're using C++ or Java, I can heartily recommend Bazel as an improvement to your current tool. If you're using anything else, hold off until the ecosystem is more mature. I can believe all of these reimplementations are worth it for hermeticity, but I seriously doubt most shops need the extreme hermeticity Google does. It's a thing that really matters at gargantuan scale.

klodolph 7y ago
> The reason the ecosystem maturity is so important for bazel is because its design encourages complete reimplementations. My experience from before Bazel is that reimplementations were the way to go most times you had projects with multiple languages, because the alternatives lead you down dark paths—calling into a separate build system suffers from the same problems as recursive make. What excites me about 1.0 is the possibility that the third-party rules can mature. Its not too bad if you only need a couple sets of third-party rules but my experience is like yours-when a new Bazel release came out, you had to figure out when to upgrade based on your dependencies on third-party rules. If you had several of these, it made upgrading Bazel infeasible.
- jonhohle 7y ago
  Amazon’s build system handled this gracefully. Native build tool chains were always used, but wrapped in a relatively thin facade. As long as the buildtool could respond to a `release` target and put artifacts in a well known location it didn’t matter what the underlying implementation was. The biggest drawback that I could see was all dependencies and tool chain needed to be modeled, but once they were, incremental builds, dependency rebuilds, etc. all Just Worked™. Getting to that point requires a lot of upfront work, however, which can be hard to justify in smaller groups.
  - m0zg 7y ago
    Can it do "bulletproof", repeatable incremental builds? I'm asking because I don't see how it would be able to, unless the toolchain is wired into your build system _at least_ as deeply as it is in Bazel.
    - aidenn0 7y ago
      Slightly OT, but Tup[1] handles this by sanitizing the environment and building in a FUSE filesystem to track reads and writes. It's bulletproof except for system header files (or more properly anything outside the root of your build, which usually is just system header files). Nix[2] accomplishes bulletproof repeatable builds by sandboxing everything, but I don't believe incremental builds are a design goal (though if you modularize everything into different nix expressions you will in fact get annoyingly hard to use bulletproof incremental builds). 1: http://gittup.org/tup/ http://gittup.org/tup/ 2: https://nixos.org/nix/ https://nixos.org/nix/
    - dastbe 7y ago
      So bazel and brazil solve similar but very distinct problems. bazel is solving the problem of fast, repeatable, "bulletproof" builds for (very large) monolithic repos, whereas brazil is solving for fast, repeatable, "bulletproof" builds for (very large) collections of repositories or packages. brazil is EXTREMELY agnostic to what happens within any given package build, with the associated tooling primarily focused on dependency artifact discovery. The (IMO) most important component of brazil is the version set, which is a versioned dependency closure that allows for one-off builds (so your auto-build application packages) as well as merges from upstream version sets (where library vendors release their code). they are the glue that makes the distributed, manyrepo software development model at Amazon feel safe and consistent. If I had my way, I'd combine the single codebase UX of bazel with the multi-code base dependency tracking of brazil, because i think they solve complimentary problems extremely well.
      - m0zg 7y ago
        Google just tests the living Jesus out of everything, and only versions a few core packages such as protobufs, grpc, and other packages used by pretty much everybody (this is called the "crust"). Everything else is tip of tree, and things are automatically re-tested using checked-in tests if they are affected by your changelist. You basically can't submit if you break anything. So in a way, Google doesn't need "versioning". Whatever is currently checked in is good to go. Tests are required, of course, and a Google reviewer won't let you submit anything if your tests suck. This, obviously, precludes the use of such a set-up in, shall we say, "more agile" orgs which don't have good test coverage. Blaze (at Google) is also not just a build system, but also an interface to a much larger distributed build and test backend, which lets you rebuild everything from the kernel upwards in seconds (by caching petabytes of build products at thousands of possible revisions), serves up source code views for developer workstations (code is not stored there either), and sustains the scale of distributed testing needed for this setup to work. As a result, nobody builds or tests on their own workstation, and there's close to zero (or maybe even zero, period) binaries checked into Google3 monorepo. If you need a Haskell compiler and nobody used it in a while, it'll be rebuilt from source and cached for future use. :-) Fundamentally, I think Google got things very, very right with Blaze. Bazel is but a pale shadow of what Blaze is, but even in its present state it is better than most (all?) other build systems.
- Too 7y ago
  It would be neat if all build systems had a standardized way to return their dependency graph (like gcc -MD) so they can be used recursively by other build systems. I'm not aware of any that does and there's far from a standard which allows easy chaining of multiple different sub build systems. Which unfortunately either leads to recursive make, remodeling the dependencies in two places or forcing everything into the same build system.
c4urself 7y ago
Agree. I've been dealing with Python rules for Bazel at work and the default implementation of re-downloading everything with pip became a real pain real fast. Decided to generate the third party py_library with a separate tool instead and much happier for it. As an aside for those looking to start with Bazel, it's worth checking out the BazelBuild Slack -- https://slack.bazel.build/ https://slack.bazel.build/
- dimator 7y ago
  just curious: is that separate tool open source?
miles_matthias 7y ago
They're really excited about 1.0 for this reason, they're tired of breaking other people's builds too. The goal is to limit that with the 1.0 release, so expect this to get a lot better.
- kylecordes 7y ago
  I follow some of the relevant repositories, and there has been a big effort over the last few months to get many many loose ends cleaned up, toward a stable set of tools that will change in a predictable cadence without undue breakage.
kylecordes 7y ago
About the container stuff specifically, I think the different solution provided here versus off-the-shelf Docker is not merely to be compatible with Bazel. Rather it looks like, from having used it modestly, an intentionally different way of thinking about container builds, aiming toward different performance characteristics that scale etc. I can't think of any reason you couldn't call the docker tools in a genrule() though, if you prefer.
- habitue 7y ago
  Yeah, I actually like the speed with which it can build images. But it's a risk that it's an entirely different implementation. If you have a team of people to deal with maintenance, then it's worth it. If you don't, then when an incompatibility crops up, you're stuck until you can hack around it. Having a really well maintained docker ruleset is crucial if you don't have that team
  - weberc2 7y ago
    OCI provides a standard container image format, at least. Assuming your whole toolchain is compatible, then it shouldn't be an issue.
- weberc2 7y ago
  Docker's build system is really strange. The build cache assumes that your dependency tree is linear, and it will invalidate the build cache if any file dependency changes, but a `RUN apt install foo` doesn't invalidate the cache--it's just assumed that every RUN statement is idempotent. Further, the Dockerfile's COPY is close to worthless--if you want to import all of your repo's Python packages, you have to put them all under a single directory or you have to copy them one by one (or you copy the whole repo and deal with the fact that every JS and README change will trigger a complete rebuild). Also, everything depends on a running Docker daemon and last I checked, the performance degraded if you tried to build multiple images in parallel.
  - Quekid5 7y ago
    Strange doesn't begin to cover it :). It's just broken.
lhorie 7y ago
> They've been breaking backwards compatibility constantly I maintain a ruleset and have been on top of Bazel upgrades for a few months. My experience hasn't been so bad. There definitely are a lot of deprecations, and some code changes required, but it was pretty manageable IMHO. With that said, I agree that some stacks are more first class citizens than others. But I feel this is more due to some stacks' departure from the C++ style of build management than an issue of Bazel changes. At our company, the Go org has had a pretty good experience with migrating from Buck to Bazel. For our web org, the main challenges have been that Bazel doesn't really lend itself well to dealing with codebases where inputs and outputs co-exist in the same filesystem tree (and this is incredibly common in the web world with things like babel transpiled files, yarn offline caches, lerna, etc) or where the ecosystem has a bad relationship with symlink handling (e.g. jest, flow, webpack, --preserve-symlinks, etc)
mrighele 7y ago
What are the reasons to prefer Bazel to Maven or Gradle foe Java? I know it is supposed to be faster but on the other side last time i checked it it was much more complex and time consuming (expecially compared to Maven). Are there other good reasons?
- m0zg 7y ago
  I don't work in Java, but just so you know, you can define Maven deps in your WORKSPACE file, and Bazel will automatically download and cache them (and their dependencies) much like Maven would. The rest of the build is trivial to set up.
- oweiler 7y ago
  I think for Java projects Bazel is a very bad choice. Last time I've checked it was not even able to resolve dependencies transitively. And I doubt that it's faster than recent versions of Gradle.
  - jingwen 7y ago
    The Bazel team published official rules for managing transitive dependencies from Maven: https://github.com/bazelbuild/rules_jvm_external https://github.com/bazelbuild/rules_jvm_external There are also community created solutions like https://github.com/johnynek/bazel-deps https://github.com/johnynek/bazel-deps, and https://github.com/square/bazel_maven_repository https://github.com/square/bazel_maven_repository. Disclosure: I maintain rules_jvm_external.
  - swsieber 7y ago
    I very much doubt that it's faster than Gradle, but Gradle does do quite a bit more than Gradle.
    - christiangruber 7y ago
      It's not an absolute, but a contextual thing. For a clean build, on a single machine, under normal circumstances, gradle will often outperform bazel, because bazel pays a cost for isolation, parallelism infrastructure, and trying to max out your machine. OTOH, the abi-based compilation-avoidance, higher sensitivity to caching opportunities, and parallelism can make incremental builds much much faster on Bazel. Add in test-result-caching for non-flaky tests, and it's really a solid win for builds that don't invalidate most of the build graph. So it depends on (a) the shape of your graph, and (b) the nature of the change being built (e.g., does it invalidate a core library and all its downstream, or is it close to the app?). The other thing is remote build execution - the high focus on hermetic builds and parallelism makes building on a cluster of build workers extremely powerful, reducing build times, reducing load on developer machines, etc. And there are out-of-the-box solutions, initially from google on GCP to let you do this yourself. So... is bazel faster? Hell yeah, in some situations. Kinda in others. Not in some others. You need to think through what your development model is, your scale, how you construct (or want to construct) your build graphs, isolation of components, etc.
weberc2 7y ago
> but the ecosystem around Bazel is quite bad at the moment I've noticed this as well. I'd love to use Bazel for our Python 3 project, but as far as I can tell, the advertised Python 3 support doesn't actually work. There are several issues filed for this, but apparently no progress since 2018. Some people reported workarounds, but none that I could reproduce. I believe the core is solid, but language support feels alpha at best.
- thundergolfer 7y ago
  Python support is very much alpha. We use rules_python and the “reported workarounds” include forking rules_python, reimplementing a Bazel pip package manager integration, and adding a few twists to your Python codebase to handle Bazel idiosyncrasies. It’s dozens of hours of work, and even then you quite easily can find yourself not enjoying Bazel’s caching features because of pip wheel being non-deterministic. It’s been a hard road.
  - weberc2 7y ago
    That makes sense. What doesn't make so much sense is that the documentation indicates that Python 3 is supported, but everything I've tried and everyone I've spoken to have indicated that it's just broken. In any case, I hope I can use it eventually. Any idea if deterministic wheels is on the Python project's radar? EDIT: Quick Google search answered my question--the first result is an issue you filed here: https://github.com/pypa/pip/issues/6505 https://github.com/pypa/pip/issues/6505
    - thundergolfer 7y ago
      I haven't been super happy with the prioritisation of that issue in rules_python. In our Java+Scala+Python monorepo we've got our Java+Scala tests caching so that part of our test suite gets a low as 2 minutes in CI. The Python part of our codebase is maybe 5-10% of the codebase but its tests take ~5 minutes in CI because of no caching.
thundergolfer 7y ago
They’re underinvesting in their Python story in my opinion. To my knowledge rules_python still doesn’t support pip3 installs, and from browsing Github, Twitter, and the Bazel Slack it seems everyone is reimplementing their own package management integration because the default is so undercooked. Right now rules_pygen is the best open source option, but it isn’t fully featured and was broken by Bazel 0.28.

klodolph 7y ago

> The reason the ecosystem maturity is so important for bazel is because its design encourages complete reimplementations. My experience from before Bazel is that reimplementations were the way to go most times you had projects with multiple languages, because the alternatives lead you down dark paths—calling into a separate build system suffers from the same problems as recursive make. What excites me about 1.0 is the possibility that the third-party rules can mature. Its not too bad if you only need a couple sets of third-party rules but my experience is like yours-when a new Bazel release came out, you had to figure out when to upgrade based on your dependencies on third-party rules. If you had several of these, it made upgrading Bazel infeasible.

jonhohle 7y ago
Amazon’s build system handled this gracefully. Native build tool chains were always used, but wrapped in a relatively thin facade. As long as the buildtool could respond to a `release` target and put artifacts in a well known location it didn’t matter what the underlying implementation was. The biggest drawback that I could see was all dependencies and tool chain needed to be modeled, but once they were, incremental builds, dependency rebuilds, etc. all Just Worked™. Getting to that point requires a lot of upfront work, however, which can be hard to justify in smaller groups.
- m0zg 7y ago
  Can it do "bulletproof", repeatable incremental builds? I'm asking because I don't see how it would be able to, unless the toolchain is wired into your build system _at least_ as deeply as it is in Bazel.
  - aidenn0 7y ago
    Slightly OT, but Tup[1] handles this by sanitizing the environment and building in a FUSE filesystem to track reads and writes. It's bulletproof except for system header files (or more properly anything outside the root of your build, which usually is just system header files). Nix[2] accomplishes bulletproof repeatable builds by sandboxing everything, but I don't believe incremental builds are a design goal (though if you modularize everything into different nix expressions you will in fact get annoyingly hard to use bulletproof incremental builds). 1: http://gittup.org/tup/ http://gittup.org/tup/ 2: https://nixos.org/nix/ https://nixos.org/nix/
  - dastbe 7y ago
    So bazel and brazil solve similar but very distinct problems. bazel is solving the problem of fast, repeatable, "bulletproof" builds for (very large) monolithic repos, whereas brazil is solving for fast, repeatable, "bulletproof" builds for (very large) collections of repositories or packages. brazil is EXTREMELY agnostic to what happens within any given package build, with the associated tooling primarily focused on dependency artifact discovery. The (IMO) most important component of brazil is the version set, which is a versioned dependency closure that allows for one-off builds (so your auto-build application packages) as well as merges from upstream version sets (where library vendors release their code). they are the glue that makes the distributed, manyrepo software development model at Amazon feel safe and consistent. If I had my way, I'd combine the single codebase UX of bazel with the multi-code base dependency tracking of brazil, because i think they solve complimentary problems extremely well.
    - m0zg 7y ago
      Google just tests the living Jesus out of everything, and only versions a few core packages such as protobufs, grpc, and other packages used by pretty much everybody (this is called the "crust"). Everything else is tip of tree, and things are automatically re-tested using checked-in tests if they are affected by your changelist. You basically can't submit if you break anything. So in a way, Google doesn't need "versioning". Whatever is currently checked in is good to go. Tests are required, of course, and a Google reviewer won't let you submit anything if your tests suck. This, obviously, precludes the use of such a set-up in, shall we say, "more agile" orgs which don't have good test coverage. Blaze (at Google) is also not just a build system, but also an interface to a much larger distributed build and test backend, which lets you rebuild everything from the kernel upwards in seconds (by caching petabytes of build products at thousands of possible revisions), serves up source code views for developer workstations (code is not stored there either), and sustains the scale of distributed testing needed for this setup to work. As a result, nobody builds or tests on their own workstation, and there's close to zero (or maybe even zero, period) binaries checked into Google3 monorepo. If you need a Haskell compiler and nobody used it in a while, it'll be rebuilt from source and cached for future use. :-) Fundamentally, I think Google got things very, very right with Blaze. Bazel is but a pale shadow of what Blaze is, but even in its present state it is better than most (all?) other build systems.
Too 7y ago
It would be neat if all build systems had a standardized way to return their dependency graph (like gcc -MD) so they can be used recursively by other build systems. I'm not aware of any that does and there's far from a standard which allows easy chaining of multiple different sub build systems. Which unfortunately either leads to recursive make, remodeling the dependencies in two places or forcing everything into the same build system.

c4urself 7y ago

Agree. I've been dealing with Python rules for Bazel at work and the default implementation of re-downloading everything with pip became a real pain real fast. Decided to generate the third party py_library with a separate tool instead and much happier for it. As an aside for those looking to start with Bazel, it's worth checking out the BazelBuild Slack -- https://slack.bazel.build/ https://slack.bazel.build/

dimator 7y ago
just curious: is that separate tool open source?

miles_matthias 7y ago

They're really excited about 1.0 for this reason, they're tired of breaking other people's builds too. The goal is to limit that with the 1.0 release, so expect this to get a lot better.

kylecordes 7y ago
I follow some of the relevant repositories, and there has been a big effort over the last few months to get many many loose ends cleaned up, toward a stable set of tools that will change in a predictable cadence without undue breakage.

kylecordes 7y ago

About the container stuff specifically, I think the different solution provided here versus off-the-shelf Docker is not merely to be compatible with Bazel. Rather it looks like, from having used it modestly, an intentionally different way of thinking about container builds, aiming toward different performance characteristics that scale etc. I can't think of any reason you couldn't call the docker tools in a genrule() though, if you prefer.

habitue 7y ago
Yeah, I actually like the speed with which it can build images. But it's a risk that it's an entirely different implementation. If you have a team of people to deal with maintenance, then it's worth it. If you don't, then when an incompatibility crops up, you're stuck until you can hack around it. Having a really well maintained docker ruleset is crucial if you don't have that team
- weberc2 7y ago
  OCI provides a standard container image format, at least. Assuming your whole toolchain is compatible, then it shouldn't be an issue.
weberc2 7y ago
Docker's build system is really strange. The build cache assumes that your dependency tree is linear, and it will invalidate the build cache if any file dependency changes, but a `RUN apt install foo` doesn't invalidate the cache--it's just assumed that every RUN statement is idempotent. Further, the Dockerfile's COPY is close to worthless--if you want to import all of your repo's Python packages, you have to put them all under a single directory or you have to copy them one by one (or you copy the whole repo and deal with the fact that every JS and README change will trigger a complete rebuild). Also, everything depends on a running Docker daemon and last I checked, the performance degraded if you tried to build multiple images in parallel.
- Quekid5 7y ago
  Strange doesn't begin to cover it :). It's just broken.

lhorie 7y ago

> They've been breaking backwards compatibility constantly I maintain a ruleset and have been on top of Bazel upgrades for a few months. My experience hasn't been so bad. There definitely are a lot of deprecations, and some code changes required, but it was pretty manageable IMHO. With that said, I agree that some stacks are more first class citizens than others. But I feel this is more due to some stacks' departure from the C++ style of build management than an issue of Bazel changes. At our company, the Go org has had a pretty good experience with migrating from Buck to Bazel. For our web org, the main challenges have been that Bazel doesn't really lend itself well to dealing with codebases where inputs and outputs co-exist in the same filesystem tree (and this is incredibly common in the web world with things like babel transpiled files, yarn offline caches, lerna, etc) or where the ecosystem has a bad relationship with symlink handling (e.g. jest, flow, webpack, --preserve-symlinks, etc)

mrighele 7y ago

What are the reasons to prefer Bazel to Maven or Gradle foe Java? I know it is supposed to be faster but on the other side last time i checked it it was much more complex and time consuming (expecially compared to Maven). Are there other good reasons?

m0zg 7y ago
I don't work in Java, but just so you know, you can define Maven deps in your WORKSPACE file, and Bazel will automatically download and cache them (and their dependencies) much like Maven would. The rest of the build is trivial to set up.
oweiler 7y ago
I think for Java projects Bazel is a very bad choice. Last time I've checked it was not even able to resolve dependencies transitively. And I doubt that it's faster than recent versions of Gradle.
- jingwen 7y ago
  The Bazel team published official rules for managing transitive dependencies from Maven: https://github.com/bazelbuild/rules_jvm_external https://github.com/bazelbuild/rules_jvm_external There are also community created solutions like https://github.com/johnynek/bazel-deps https://github.com/johnynek/bazel-deps, and https://github.com/square/bazel_maven_repository https://github.com/square/bazel_maven_repository. Disclosure: I maintain rules_jvm_external.
- swsieber 7y ago
  I very much doubt that it's faster than Gradle, but Gradle does do quite a bit more than Gradle.
  - christiangruber 7y ago
    It's not an absolute, but a contextual thing. For a clean build, on a single machine, under normal circumstances, gradle will often outperform bazel, because bazel pays a cost for isolation, parallelism infrastructure, and trying to max out your machine. OTOH, the abi-based compilation-avoidance, higher sensitivity to caching opportunities, and parallelism can make incremental builds much much faster on Bazel. Add in test-result-caching for non-flaky tests, and it's really a solid win for builds that don't invalidate most of the build graph. So it depends on (a) the shape of your graph, and (b) the nature of the change being built (e.g., does it invalidate a core library and all its downstream, or is it close to the app?). The other thing is remote build execution - the high focus on hermetic builds and parallelism makes building on a cluster of build workers extremely powerful, reducing build times, reducing load on developer machines, etc. And there are out-of-the-box solutions, initially from google on GCP to let you do this yourself. So... is bazel faster? Hell yeah, in some situations. Kinda in others. Not in some others. You need to think through what your development model is, your scale, how you construct (or want to construct) your build graphs, isolation of components, etc.

weberc2 7y ago

> but the ecosystem around Bazel is quite bad at the moment I've noticed this as well. I'd love to use Bazel for our Python 3 project, but as far as I can tell, the advertised Python 3 support doesn't actually work. There are several issues filed for this, but apparently no progress since 2018. Some people reported workarounds, but none that I could reproduce. I believe the core is solid, but language support feels alpha at best.

thundergolfer 7y ago
Python support is very much alpha. We use rules_python and the “reported workarounds” include forking rules_python, reimplementing a Bazel pip package manager integration, and adding a few twists to your Python codebase to handle Bazel idiosyncrasies. It’s dozens of hours of work, and even then you quite easily can find yourself not enjoying Bazel’s caching features because of pip wheel being non-deterministic. It’s been a hard road.
- weberc2 7y ago
  That makes sense. What doesn't make so much sense is that the documentation indicates that Python 3 is supported, but everything I've tried and everyone I've spoken to have indicated that it's just broken. In any case, I hope I can use it eventually. Any idea if deterministic wheels is on the Python project's radar? EDIT: Quick Google search answered my question--the first result is an issue you filed here: https://github.com/pypa/pip/issues/6505 https://github.com/pypa/pip/issues/6505
  - thundergolfer 7y ago
    I haven't been super happy with the prioritisation of that issue in rules_python. In our Java+Scala+Python monorepo we've got our Java+Scala tests caching so that part of our test suite gets a low as 2 minutes in CI. The Python part of our codebase is maybe 5-10% of the codebase but its tests take ~5 minutes in CI because of no caching.

thundergolfer 7y ago

They’re underinvesting in their Python story in my opinion. To my knowledge rules_python still doesn’t support pip3 installs, and from browsing Github, Twitter, and the Bazel Slack it seems everyone is reimplementing their own package management integration because the default is so undercooked. Right now rules_pygen is the best open source option, but it isn’t fully featured and was broken by Bazel 0.28.