4 ms·

At what point do you bring in Bazel? I've heard some people say stuff like Python should always use Bazel, or that full stack webapps should use it, but it ofte

At what point do you bring in Bazel? I've heard some people say stuff like Python should always use Bazel, or that full stack webapps should use it, but it often just seems like extra cruft. Why not just have separate repos and use each language's tools?

sa46 2y ago
You'll get wildly different answers depending on who you ask. Most ex-Googlers appreciate Bazel. Folks used to ergonomic single-language builds dislike it. In my experience, Bazel works best when: - You have a monorepo. - You have one engineer who knows it in and out and can stay current. - You use several languages well-supported by Bazel (Protobuf, Go, Java, Kotlin). The benefits are a single way to build and run all tests and binaries across all languages. Bazel CI providers (BuildBuddy, EngFlow) are much nicer than YAML engineering in other CI systems. I have an old comment that's still representative of my experience: https://news.ycombinator.com/item?id=32831555 https://news.ycombinator.com/item?id=32831555
ein0p 2y ago
Large codebase in which you need a hermetic, repeatable, incremental build and test support. Note that in a hermetic build tests can be incremental too: if input and code do not change, there’s no need to rerun the test, and Bazel will skip it.
ali_piccioni 2y ago
We had an integration test that involved many tool chains and languages to prepare. Rust compilation, WASM compilation, Docker builds shell scripts that produced a VM image, Python was in there somehow too, and then a test harness, written in Rust, that took all that to actually run the test. With Bazel setup, and it was a beast to setup, developers could run all that, with remotely cached incremental builds, efficient parallel execution, and all reproducible (and same as CI) from a local developer environment with just one command.
- riku_iki 2y ago
  > Rust compilation, WASM compilation, Docker builds shell scripts that produced a VM image, Python was in there somehow too was it through bazel rules? The worry is if some of those rules will get bug or missing necessary feature, it will be pita to deal with.
  - steveklabnik 2y ago
    This is one reason why I’m quite interested in buck2; there are no built in rules at all, so if you need any changes, you can just make them. Unfortunately the open source set of rules is a bit rough right now, with a decent happy path but a cliff if you’re off of it, but given some time…
  - ali_piccioni 2y ago
    The rules are open source. We did run into some road bumps in Rules Rust, that required patches or clever work arounds. But I believe the community has made great strides to improve Bazel’s Rust support in the last five years. To be frank, I would avoid introduction of Bazel unless you are experiencing major pains with your current system and have at least a small centralized team willing to build expertise and provide support.
    - patmorgan23 2y ago
      Bazel is definitely like kubernetes where you don't need it, until you need it. If you got a big polyglot application its perfect for you. If your app is largely all in one language then you don't need that complexity in you life.
- shanemhansen 2y ago
  I recently left Google and I do miss blaze. I'm working on some stuff that's rust and wasm and typescript and there are containers and I'm very tempted to use bazel for basically correct caching and stuff. But even as a xoogler I think that the common wisdom of "don't use bazel until you can afford a team" or "use bazel if you are the only developer" are probably right. My somewhat janky series of dockerfiles and bash scripts is at least debuggable to my business partner. I'm just not ready to commit our future frontend developers to having to use bazel yet. I'm sort of sticking to a lowest common denominator technology stack for most things except where we are spending our innovation tokens in wasm land. But someday, probably. In the meantime I may setup sccache.
abound 2y ago
For me, it's usually when I have more than three or four languages in a single codebase and there's codegen (e.g. protobuf, GraphQL, etc) involved. It's nice to be able to `bazel test //...` a whole repo, and building minimal, reproducible Docker images becomes comparatively straightforward. That said, I've never made the leap in using Bazel for frontend stuff, I always use NPM/pnpm/yarn without any Bazel trimmings.
greenpizza13 2y ago
We leverage Bazel heavily across multiple very large projects where I work. It greatly simplifies dependencies across languages with different build steps. We also run Bazel in docker so everyone has the same build environment. Finally, we leverage Bazel on cloud build farms so very large builds don't happen locally and can leverage shared caches. This is really where the hermetic nature pays big dividends. Our codebase uses a single build system for CPP, Python, CPP, GoLang, and more.
malkia 2y ago
I'm spoiled, as I worked for 2-3 years at Google, and coming from gamedev - I was wtf is this thing, then when I quit I started missing it. I've used with Java, Python, some C++, some R, bash, sawzall and few other langs. The beauty of it is that it can express dependencies across things from different languages (where this makes sense). Another cool thing (for Java) and FFI C/C++ code is that it can build custom java runner that embeds everything in single binary. So your FFI C/C++ code and the Java code becomes one binary. But basically you can say - in order to build this, you also need to build this in a single language. On top of that you can express tests too. And then it knew how to take/combine unit tests of same "size" and shard them to execute across machines. And because you have to size your tests (e.g. RAM / CPU wise), it knows which to pick, and if your test does not fall anymore in that category it'll tell you to fix it. Might even offer command-line (nowadayws buildozer command-line) how to fix it. (That was ~2014-2016) Then for C++ (and other langs) it'll track whether you've specified your header deps. To think about it, let's say you've said - here is a.cpp and a.cpp includes a.h - but forgot about adding explicitly a.h - Bazel (blaze) can catch that. One of the ways is simply to "symlink" only explicitly placed files in random temp folder, and compile from there - so "a.cpp" won't find "a.h" there and it'll tell - "Here run this command to add a.h to your target". so actually quite the opposite - you want build system that unites all these different languages/runtimes/environments. Not like - python/pip rust/cargo dotnet/nuget javascript/npm java/maven - because with each such different language specific build layer you lose how to express depedencies between them. And not only that - in bazel deps are static, e.g. with a.h missing explicitly in your BUILD file, and bazel would tell you that. In the examples above is healthy mix of everything - from dynamic discovery to who knows what. Then again, I get it - it's not the easiest to get through, the .bzl is dark magic sometimes, but the BUILD files should be easy to use to anyone - engineer, linguist, statistician, producers, tech art, etc.
- hamandcheese 2y ago
  > Not like - python/pip rust/cargo dotnet/nuget javascript/npm java/maven - because with each such different language specific build layer you lose how to express depedencies between them. Its funny you highlight this as a feature, because my #1 piece of feedback for the bazel ecosystem is that it really sucks to have to give up language-native tools. Whether it's LSP servers, documentation that says "add this to your gradle file", "add this to your build.sbt", it's always an uphill battle. And then it's an uphill battle convincing product engineers to forget everything they know, do it the bazel way instead. I get why bazel is cool, but I think it's worth highlighting that for a lot of folks it very well can make their inner development loop worse than without it, particularly if they don't have someone (or several people) dedicated solely to developer experience.

sa46 2y ago

You'll get wildly different answers depending on who you ask. Most ex-Googlers appreciate Bazel. Folks used to ergonomic single-language builds dislike it. In my experience, Bazel works best when: - You have a monorepo. - You have one engineer who knows it in and out and can stay current. - You use several languages well-supported by Bazel (Protobuf, Go, Java, Kotlin). The benefits are a single way to build and run all tests and binaries across all languages. Bazel CI providers (BuildBuddy, EngFlow) are much nicer than YAML engineering in other CI systems. I have an old comment that's still representative of my experience: https://news.ycombinator.com/item?id=32831555 https://news.ycombinator.com/item?id=32831555

ein0p 2y ago

Large codebase in which you need a hermetic, repeatable, incremental build and test support. Note that in a hermetic build tests can be incremental too: if input and code do not change, there’s no need to rerun the test, and Bazel will skip it.

ali_piccioni 2y ago

We had an integration test that involved many tool chains and languages to prepare. Rust compilation, WASM compilation, Docker builds shell scripts that produced a VM image, Python was in there somehow too, and then a test harness, written in Rust, that took all that to actually run the test. With Bazel setup, and it was a beast to setup, developers could run all that, with remotely cached incremental builds, efficient parallel execution, and all reproducible (and same as CI) from a local developer environment with just one command.

riku_iki 2y ago
> Rust compilation, WASM compilation, Docker builds shell scripts that produced a VM image, Python was in there somehow too was it through bazel rules? The worry is if some of those rules will get bug or missing necessary feature, it will be pita to deal with.
- steveklabnik 2y ago
  This is one reason why I’m quite interested in buck2; there are no built in rules at all, so if you need any changes, you can just make them. Unfortunately the open source set of rules is a bit rough right now, with a decent happy path but a cliff if you’re off of it, but given some time…
- ali_piccioni 2y ago
  The rules are open source. We did run into some road bumps in Rules Rust, that required patches or clever work arounds. But I believe the community has made great strides to improve Bazel’s Rust support in the last five years. To be frank, I would avoid introduction of Bazel unless you are experiencing major pains with your current system and have at least a small centralized team willing to build expertise and provide support.
  - patmorgan23 2y ago
    Bazel is definitely like kubernetes where you don't need it, until you need it. If you got a big polyglot application its perfect for you. If your app is largely all in one language then you don't need that complexity in you life.
shanemhansen 2y ago
I recently left Google and I do miss blaze. I'm working on some stuff that's rust and wasm and typescript and there are containers and I'm very tempted to use bazel for basically correct caching and stuff. But even as a xoogler I think that the common wisdom of "don't use bazel until you can afford a team" or "use bazel if you are the only developer" are probably right. My somewhat janky series of dockerfiles and bash scripts is at least debuggable to my business partner. I'm just not ready to commit our future frontend developers to having to use bazel yet. I'm sort of sticking to a lowest common denominator technology stack for most things except where we are spending our innovation tokens in wasm land. But someday, probably. In the meantime I may setup sccache.

abound 2y ago

For me, it's usually when I have more than three or four languages in a single codebase and there's codegen (e.g. protobuf, GraphQL, etc) involved. It's nice to be able to `bazel test //...` a whole repo, and building minimal, reproducible Docker images becomes comparatively straightforward. That said, I've never made the leap in using Bazel for frontend stuff, I always use NPM/pnpm/yarn without any Bazel trimmings.

greenpizza13 2y ago

We leverage Bazel heavily across multiple very large projects where I work. It greatly simplifies dependencies across languages with different build steps. We also run Bazel in docker so everyone has the same build environment. Finally, we leverage Bazel on cloud build farms so very large builds don't happen locally and can leverage shared caches. This is really where the hermetic nature pays big dividends. Our codebase uses a single build system for CPP, Python, CPP, GoLang, and more.

malkia 2y ago

I'm spoiled, as I worked for 2-3 years at Google, and coming from gamedev - I was wtf is this thing, then when I quit I started missing it. I've used with Java, Python, some C++, some R, bash, sawzall and few other langs. The beauty of it is that it can express dependencies across things from different languages (where this makes sense). Another cool thing (for Java) and FFI C/C++ code is that it can build custom java runner that embeds everything in single binary. So your FFI C/C++ code and the Java code becomes one binary. But basically you can say - in order to build this, you also need to build this in a single language. On top of that you can express tests too. And then it knew how to take/combine unit tests of same "size" and shard them to execute across machines. And because you have to size your tests (e.g. RAM / CPU wise), it knows which to pick, and if your test does not fall anymore in that category it'll tell you to fix it. Might even offer command-line (nowadayws buildozer command-line) how to fix it. (That was ~2014-2016) Then for C++ (and other langs) it'll track whether you've specified your header deps. To think about it, let's say you've said - here is a.cpp and a.cpp includes a.h - but forgot about adding explicitly a.h - Bazel (blaze) can catch that. One of the ways is simply to "symlink" only explicitly placed files in random temp folder, and compile from there - so "a.cpp" won't find "a.h" there and it'll tell - "Here run this command to add a.h to your target". so actually quite the opposite - you want build system that unites all these different languages/runtimes/environments. Not like - python/pip rust/cargo dotnet/nuget javascript/npm java/maven - because with each such different language specific build layer you lose how to express depedencies between them. And not only that - in bazel deps are static, e.g. with a.h missing explicitly in your BUILD file, and bazel would tell you that. In the examples above is healthy mix of everything - from dynamic discovery to who knows what. Then again, I get it - it's not the easiest to get through, the .bzl is dark magic sometimes, but the BUILD files should be easy to use to anyone - engineer, linguist, statistician, producers, tech art, etc.

hamandcheese 2y ago
> Not like - python/pip rust/cargo dotnet/nuget javascript/npm java/maven - because with each such different language specific build layer you lose how to express depedencies between them. Its funny you highlight this as a feature, because my #1 piece of feedback for the bazel ecosystem is that it really sucks to have to give up language-native tools. Whether it's LSP servers, documentation that says "add this to your gradle file", "add this to your build.sbt", it's always an uphill battle. And then it's an uphill battle convincing product engineers to forget everything they know, do it the bazel way instead. I get why bazel is cool, but I think it's worth highlighting that for a lot of folks it very well can make their inner development loop worse than without it, particularly if they don't have someone (or several people) dedicated solely to developer experience.