19 ms·

When to use Bazel?

212 points by jvolkman 4y ago

StreamBright 4y ago
Have there been any reviews about Ninja and gn? Are those comparable to Bazel?
jxramos 4y ago
One big gotcha that folks need to appreciate is that you don't get the full power of bazel just by getting baseline bazel to work and build things. You need to really be sensitive to your physical dependencies and not lump similar code together just because they play a similar role and are convenient to place in the same file rather than having that same source sharded out to distinct files. You need to partition source into all its relevant pieces and lump no more functionality in that source than is needed by clients. Bazel encourages the most minute fine grained resolution of dependencies because if you do that and manage it right you unlock the power of the cache. Once you successfully partition all the bits of software to their segregated sections of the build graph DAG then you're positioned to really leverage the test result cache and can start bypassing huge expensive tests that are completely irrelevant to the body of code you're developing on. Bazel is not just a build toolchain it is a test toolchain as well. The problems it solves with tests with respect to running only relevant & applicable tests to the changes being pushed is what many people have dreamed about before or came up with some crude solution to approximate the dependencies. If my test did not print out `(cached)` I know something in the dependency graph was disturbed in some way that may be completely not obvious to me, especially in a gigantic repo. Bazel query is also worth mentioning to visualize dependencies with its xml output and which can also enforce architectural barriers asserting empty `somepath` queries between two targets that should not meet in the middle somewhere. It would be neat one day if bazel integrates in some semantic logical level of dependency checking beyond the physical but that may be too expensive of an operation.
adamgordonbell 4y ago
Author here. I wanted to get my head wrapped around when Bazel was an excellent build solution, so I interviewed six people with a lot of Bazel experience and picked their brains. This somewhat long article hopes to answer questions about Bazel for future people in a similar position to me. If you know a lot of Bazel then you might not learn much but if you’ve vaguely heard of it and are not sure when its the tool that should be reached for I’m hoping this will help.
- jxramos 4y ago
  I particularly enjoyed the history aspect of it, these things give you broad contours of where pools of experience exist and what specific organizations were responsible for driving and advancing things.
- lostdog 4y ago
  This was a great way to pull wisdom from many different people, and get a broad idea of what everyone's experience is. Looking forward to more posts! I'm also surprised at how little experience everyone had with bazel alternatives. I wonder if buck or pants is easier to work with.
  - adamgordonbell 4y ago
    Thanks! Oscar in the article had experience with Pants but preferred Bazel and Bazel certainly has the most usage at this scale.
choeger 4y ago
Very nice work. Worth a read!
lxe 4y ago
When not to use Bazel: when attempting to shoehorn it into an existing JavaScript monorepo.
- spion 4y ago
  Yep. If its just JS, Turborepo would probably be a better choice nowadays.
  - bsimpson 4y ago
    I haven't used Turborepo, but I have really liked wireit. wireit is from the Lit team at Google. They understand both Bazel and JS. As I understand, wireit is their attempt to make the Bazel niceties JS-native.
  - nyadesu 4y ago
    Would you mind explaining why? I use Nx and it's been good, not perfect tho, so at this point I'm very skeptical about toolchaining solutions that claim they 'just work'
    - spion 4y ago
      Last time I looked at Nx it was very opinionated and I couldn't get it to work with an existing repo. In contrast I got running with turborepo in about 15 minutes in a fairly complex workspace project. Its very flexible, but in this case the flexibility is simple: you specify your input globs, output globs and task dependencies for each package.json command you want it to support and it will use that as the basis to detect cache invalidation. Works rather well.
mattboardman 4y ago
Bazel is a great tool, but as mentioned in the article the support from Google is rather limited. If bazel ever got to the level Terraform is with providers it could really take off. Something I'd love to see is a .bazel_mod file where you could pull in other bazel workspaces (similar to go.mod) and reference packages within those workspaces.
- dimator 4y ago
  That's there now: https://bazel.build/build/bzlmod https://bazel.build/build/bzlmod
  - mattboardman 4y ago
    TIL. Thanks for the link!
- n0us 4y ago
  you can already do that with the WORKSPACE file
dietrichepp 4y ago
I've been using Bazel for side projects these days, including small retro game projects for old game consoles. The entry price was high, but it works so well I have a hard time imagining working without it. For retro game projects, the core of your game might be written in C or C++, which you want cross-compiled. That's easily within reach of stuff like makefiles. But then I start adding a bunch of custom tooling--I want to write tools that process sprites, audio, or 3D models. These days I tend to write those tools in Go. I'm also working with other people, and I want to be able to cross-compile these tools for Windows, even though I'm developing on Linux or macOS. My Bazel repository will download the ARM toolchain automatically and do everything it needs to build the target. I don't really need to set up my development environment at all--I just need to install Bazel and a C compiler, and Bazel will handle the rest. I don't need to install anything else on my system, and the C compiler is only needed because I'm using protocol buffers (Bazel downloads and compiles protoc automatically).
- synergy20 4y ago
  Makefile can do all that too, without java dependency and straightforward to read at any time later, what real extra benefit Bazel brings in here.
  - ajconway 4y ago
    Bazel doesn't have any user-facing java dependency. > what real extra benefit Bazel brings in here Builds are hermetic by default, so unless the developer chooses to escape the sandbox, everything is guaranteed to build on other machines with no additional setup. (Also, I genuinely hate when I have to manually install build dependencies system-wide and pray that there will not be any conflicts. Having everything pinned to specific sha256 or git hashes by design is a breath of fresh air)
    - blueblob 4y ago
      I used a M1 Mac to try to build tensorflow and tensorflow-text and it is very untrue that everything is guaranteed to build on other machines with no additional setup. The parenthesized comment is funnier to me because I had to download a specific bazel version to build.
rjsw 4y ago
Unless it becomes easier to port to a new operating system, never.
mplewis9z 4y ago
Personally, I've found Bazel's tooling and dependency management to be extremely aggressive at pushing you to online-only development as your project scales in size. A company I worked for that used it lost multiple person-days for every engineer they had when Covid hit and the VPN went to crap. It's great at being able to offload work to a remote server, but in my opinion that should never be the only way you can get work done. Local development should always be the default, with remote execution being an _option_ when available.
- hamandcheese 4y ago
  > A company I worked for that used it lost multiple person-days for every engineer they had when Covid hit and the VPN went to crap. As far as criticisms go, “we lost a few days of work when a global pandemic shut the world down” seems pretty minor. That said, I too am a fan of offline development and I find that bazel makes that pretty easy for me. I just run bazel fetch before I get on an airplane.
- unicornmama 4y ago
  Bazel fully supports local development. How is it that you needed anything remote to do your work?
- lima 4y ago
  When implemented correctly, it also helps local development. You can use remote cache with local execution for quick compiles locally.
trzy 4y ago
Last year I tried doing a side project with a talented ex-Google buddy who insisted we set up Bazel to replace my simple Makefile. Three weeks later it still wasn’t working on my Windows box. We had a mixed Python and C++ code base and I like to use MinGW64 gcc on Windows. He blamed Windows and tried to get me to switch to Mac (no thanks, lol) and eventually he lost interest and gave up. The project went on to win an OpenCV funded competition and became the basis of a startup — good job, GNU Make! So the answer IMHO to “when to use Bazel” is “never” :)
- dbt00 4y ago
  I've used bazel at medium size (100+ engineers, multiple languages) and tried to use it at a small size (20 engineers, almost all go with a tiny bit of C) and I think the "never" and "unless there's no other way" answers from this interview are pretty good.
- thayne 4y ago
  Bazel probably wasn't a good fit for that project, especially if your makefile was simple. But where it does work well, is when you have a large complex codebase in a monorepo, and you need reliable caching to keep build times down.
  - thechao 4y ago
    C + Makefile + CCache: I just `find . -name "*.h"` and throw that as the dependency of every C file, and then let ccache/make figure out the rest. My code base runs upwards of 10+ million LOC, and incremental builds are in the "hundreds of milliseconds" and "clean" builds (ccache clean, mind you) are about 30s. Truly clean builds (on a scratch system) are ~2 minutes. We do spend time making sure our very long files still compile very quickly -- we're careful to stay away from code that exercises pathological paths through the compiler. I've found that a good laptop can compile ~20kloc/(s-thread) throughput with parallel builds (it varies from 1000–100000loc/(s-thread)).
    - thayne 4y ago
      That's specific to c. Bazel is often used in cases where you have multiple languages in use, and possibly dependencies between projects using different languages. > I just `find . -name "*.h"` That sounds like it would cause problems with not triggering a build if a .h file is deleted. And would trigger unnecessary builds if any .h file is modified. But if your builds are that fast, maybe that isn't a problem.
setheron 4y ago
I can never wrap my head around writing a tool chain in bazel
steeve 4y ago
Trying to learn Bazel by porting an existing project to it is a very hard way to get into this amazing tool. If possible, start fresh. I don’t see myself working without it from now on.
- tdeck 4y ago
  The thing is, something like Bazel only pays dividends on a large complex project. When you're starting from scratch it's unlikely you'll have any problems that Bazel helps to solve, but you'll be spending a lot of time on the ceremony of setting it up.
  - steeve 4y ago
    That's not really true. My personal threshold is if there is more than one language involved.
EdSchouten 4y ago
Regardless of whether you should use Bazel or not, my hope is that any future build systems attempt to adopt Bazel's remote execution protocol (or at least a protocol that is similar in spirit): https://github.com/bazelbuild/remote-apis https://github.com/bazelbuild/remote-apis In my opinion the protocol is fairly well designed.
an_d_rew 4y ago
I will quote my past self (https://news.ycombinator.com/item?id=26064845 https://news.ycombinator.com/item?id=26064845) We use Bazel’s rules_docker as well, and I would caution someone evaluating it with a note from out experience. What Bazel does well (and as well as Bazel fits your use-case) Bazel does extremely well and is a reproducible joy to use. But if you stray off that path even a tiny bit, you’re often in for a surprisingly inexplicable, unavoidable, far-reaching pain. For example, rules_docker is amazing at laying down files in a known base image. Everything is timestamped to the 1970 unix epoch, for reproducibility, but hey, it’s a bit-perfect reproduction. Need to run one teensy executable to do even the smallest thing that’s trivial with a Dockerfile? Bazel mist create the image transfer it to a docker daemon, run the command, transfer it back... your 1 kb change just took 5 minutes and 36 gb of data being tarred, gzipped, and flung around (hopefully-the-local) network. It may not be a dealbreaker, and you may not care, but be forewarned that these little surprises creep up fairly ofen from unexpected quarters! Edit: after 2-ish years of Bazel, I would say that for 99% of developers and organizations, the most likely answer is "never".
grahamplace 4y ago
Worth pointing out that Build Buddy (YC W20) has an enterprise Bazel product that tries to make some of the management of Bazel easier. I haven't used it myself so no commentary, I just have a friend who started there recently so it's on my radar https://www.buildbuddy.io/ https://www.buildbuddy.io/ https://www.ycombinator.com/companies/buildbuddy https://www.ycombinator.com/companies/buildbuddy
miiiiiike 4y ago
I lost a month learning Bazel last year. Never again. Here’s a challenge: Create a Bazel built Angular library using only the publicly available documentation. Here, I’ll save you some time: You can’t. What documentation exists is flawed and what isn’t flawed has massive holes. It was so bad that I had to ask a few Google employees if there was secret internal documentation somewhere. There isn’t, and, they almost all hated using it as well. I went back to Make and had the whole repo building in an afternoon. Bazel is a great idea but like most other Google OSS projects it doesn’t have strong enough documentation to form a community or enough of a community to create good docs. Even if I brute forced my way into using Bazel, I couldn’t ask an employee to learn it. I hear that Blaze is much easier to work with.
- deleted 4y ago
  [deleted]
bmohlenhoff 4y ago
I tried using Bazel a while back but immediately ran into a few issues. The existing codebase I was working with did not lay out its dependencies and files in the manner expected by Bazel so dealing with dependency/include hell was frustrating. Then there was a large portion of the project that depended on generated code for its interfaces (ie. similar to protobuf but slightly different) and trying to dive into the Bazel rules and toolchain with no other Bazel experience was not fun. I attempted to build off of the protobuf implementation but kept finding additional layers to the onion that didn't exactly translate to the non protobuf tooling. The project documentation seemed out of date in this area (ie. major differences in the rules engine between 1.0 and subsequent versions) and I couldn't find many examples to look at other than overly simplified toy examples. All in all a frustrating experience. I could not even get far enough along to compile the interfaces for the project.
- Too 4y ago
  Did you find genrule? It’s a generic escape hatch rule that can be used to just run any program. Of course it still needs the input and output dependencies declared and you are responsible for ensuring it is hermetic.
  - bmohlenhoff 4y ago
    Yes, I started out by using genrule before going down the custom rule path. I was able to properly trigger bazel to generate the interface bindings but then lost the thread when trying to properly configure the output files to act as proper dependencies to other code (ie. generate C++ headers and libraries that could then be properly listed as dependencies by client code.) I think part of the issue was that the generated code from the 3rd party tool did not align with the Bazel expectations about code layout (ie. paths relative to WORKSPACE). Likely solvable but not in the amount of time I had available to dedicate to the effort.
smeagull 4y ago
Haven't yet gotten a single project using Bazel to build on any of my machines. I don't think I'd adopt a build system that is so frustrating.
cstrahan 4y ago
There’s an issue I reported (along with a proof of concept fix) over 4 years ago, that has yet to be fixed: building a mixed source project containing Go & C++ & C++ protocol buffers results in silently broken binaries as rules_go will happily not forward along the linker arguments that the C++ build targets (the protobuf ones, using the built in C++ rules) declare. See https://github.com/bazelbuild/rules_go/issues/1486 https://github.com/bazelbuild/rules_go/issues/1486 Not very confidence inspiring when Google’s build system falls over when you combine three technologies that are used commonly throughout Google’s code base (two of which were created by Google). If you’re Google, sure, use Bazel. Otherwise, I wouldn’t recommend it. Google will cater to their needs and their needs only — putting the code out in the open means you get the privilege of sharing in their tech debt, and if something isn’t working, you can contribute your labor to them for free. No thanks :)
- vlovich123 4y ago
  Does protobufs rely on static initialization or did you link the wrong bug? That would be extremely strange because there’s generally a disallowance of any / all non trivial static initialization in library code Google-wide, especially for protobufs.
badoongi 4y ago
My take - Avoid Bazel as long as you can, for most companies the codebase is not big enough to actually need distributed builds, if you've hit this problem Bazel is probably the best thing you can do today, if you're that big you can probably spare the few dozen headcount needed to make Bazel experience in your company solid. Bazel takes on dependency management, which is probably an improvement for a C++ codebase where there is no de-facto package manager. For modern languages like golang where a package manager is widely adopted by the community it's usually just a pain. e.g Bazel's offering for golang relies on generating "Bazel configurations" for the repositories to fetch, this alternative definition of dependencies is not what all the existing go tooling are expecting, and so to get the dev tooling working properly you end up generating one configuration from the other having 2 sources of truth, and pains when there's somehow a mismatch. Bazel hermeticity is very nice in theory, in practice many of the existing toolchains used by companies that are using Bazel are non-hermetic, resulting in many companies stuck in the process of "migration to Bazel remote execution" forever. Blaze works well in Google's monorepo where all the dependencies are checked in (vendored), the WORKSPACE file was an afterthought when it was opensourced, and the whole process of fetching remote dependencies in practice becomes a pain for big monorepos (I just want to build this small golang utility, `bazel build //simple:simple` and you end up waiting for a whole bunch of python dependencies you don't need to be downloaded). And this is all before talking about Javascript, if your JS codebase wasn't originally designed the way Bazel expects it you're probably up for some fun.
intellix 4y ago
as an angular developer, Bazel was promised to replace webpack behind the scenes and suddenly everything would get fast and all our problems would be solved. Months... Years went by until eventually it became an opt-in solution and I guess they just gave up trying to use it.
deleted 4y ago
[deleted]
xwowsersx 4y ago
Only tangentially related, but can someone give me the pitch for Earthly? I think I'm just missing something basic. Our CI consists of tests, builds and deployments (all docker containers) that are kicked off in bitbucket pipelines. Is there some way that either our local development or CI builds can be improved by using earthly? Appreciate any info!
jsw 4y ago
Bazel is a fully reproducible and hermetic build system. A lot of painstaking work goes into it producing the exact same artifacts build after build. And that provides some interesting properties that you can leverage for artifact caching, deployments, and CICD. We very happily runny a polyglot monorepo w/ 5+ languages, multiple architectures, with fully reproducible artifacts and deployment manifests, all deployed to almost all AWS regions on every build. We update tens of thousands of resources in every environment for every build. The fact that Bazel is creating reproducible artifacts allow us to manage this seamlessly and reliably. Clean builds take an hour+, but our GH self-hosted runners often complete commit to green build for our devs in less than a minute. The core concept of Bazel is very simple: explicitly declare the input you pass to a rule/tool and explicitly declare the output it creates. If that can click, you're half way there.
- jeffbee 4y ago
  Having spent a great deal of time getting bazel set up as you describe, I feel you have given readers a misleading impression. Bazel does not come out of the box that way. It uses whatever toolchain is laying around on the host, by default. It builds with system headers and links with system libraries. It looks at your environment variables. You need to do a lot of surgery on the toolchain to make it hermetic and reproducible.
  - tehlike 4y ago
    Sounds like a job for docker or nix?
    - jeffbee 4y ago
      The bazel way to do it is to put the tools in an archive, refer to the archive and its checksum in your workspace, and execute the build in a completely empty sandbox.
      - hakre 4y ago
        Can you share a bit about the completely empty sandbox? Is this a build-root with it's own user and environment? Or does it build inside worktree, e.g. a subdirectory. Or can both be done?
xvedejas 4y ago
Bazel doesn't allow targeting a lot of platforms (especially embedded) from Rust, even when the Rust ecosystem supports these targets. Something is off with its design if new work needs to be done for every platform that's already available behind an interface that's as consistent as what rustc gives. What is supported needs to be inferred from this file, as far as I can tell: https://github.com/bazelbuild/rules_rust/blob/main/rust/platform/triple_mappings.bzl https://github.com/bazelbuild/rules_rust/blob/main/rust/plat...
- DriftRegion 4y ago
  https://github.com/driftregion/bazel-c-rust-x86_linux-armv7_baremetal/blob/main/readme.md https://github.com/driftregion/bazel-c-rust-x86_linux-armv7_...
  - xvedejas 4y ago
    This doesn't instruct how to target `thumbv7em-none-eabihf`, for example
davikr 4y ago
Kinda tangential, but I still can't believe there are "Modern" CMake books hundreds of pages long. I still end up using it sometimes, but I have no confidence in the rules I write, specially when compared to good old Visual Studio solutions or my favorite tool which is Premake.
cbHXBY1D 4y ago
I use bazel at work and it does make cross language work very easy. I mostly write C++ so it's one of the few good options out there. But I've also used CMAKE with vcpkg for personal projects and found that to be a really good option. Vcpkg is really the best package manager for C++ right now and seems to work seamlessly on Windows, too. Hopefully bazel's bzlmod can take off and get as many libraries as vcpkg.
sa46 4y ago
My experience with Bazel at a startup (also used Blaze at Google): The good: - Amazing for Go backends. I can push a reproducible Docker image to Kubernetes with a single Bazel command. I can run our entire backend with a single command that will work on any developer's box. - Amazing for testing. All of our backends tests use a fully independent Postgres database managed by Bazel. It's really nice not having to worry about shared state in databases across tests. - We can skip Docker on macOS for development which provides on the order of a 10x speedup for tests. - BuildBuddy provides a really nice CI experience with remote execution. Bazel tests are structured so I can see exactly which tests failed without trawling through thousands of lines of log output. I've heard good things about EngFlow but BuildBuddy was free to start. - Really nice for schema driven codegen like protobufs. The bad: - Bazel is much too hard for TypeScript and JavaScript. We don't use Bazel for our frontend. New bundlers like Vite are much faster and have a developer experience that's hard to replicate with Bazel. Aspect.dev is doing some work on this front. One large hurdle is there's not automatic BUILD file dependency updater like Gazelle for Go. - Windows support is too hard largely because most third party dependencies don't work well with Windows. - Third party dependencies are still painful. There's ongoing work with bzlmod but my impression is that it won't be usable for a couple of years. - Getting started was incredibly painful. However, the ongoing maintenance burden is a few hours per month.
- lhorie 4y ago
  > Bazel is much too hard for TypeScript and JavaScript. We don't use Bazel for our frontend. New bundlers like Vite are much faster and have a developer experience that's hard to replicate with Bazel. Aspect.dev is doing some work on this front. One large hurdle is there's not automatic BUILD file dependency updater like Gazelle for Go. I built this[0] for the Bazel + yarn setup that we use at Uber. We currently manage a 1000+ package monorepo with it. [0] https://github.com/uber-web/jazelle https://github.com/uber-web/jazelle
- hamandcheese 4y ago
  > Amazing for testing. All of our backends tests use a fully independent Postgres database managed by Bazel. It's really nice not having to worry about shared state in databases across tests. I would love to know more about this. We use bazel and Postgres, but there is no Postgres isolation between tests currently which is very predictably a large source of flakyness, and also a blocker for increasing test parallelism. I’ve been scheming on perhaps using pg_tmp orchestrated outside of bazel, but if there’s a way to do everything in bazel that would make the transition way easier.
higgins 4y ago
Ctrl-F never
WatchDog 4y ago
While it's not as natural to use for non-jvm projects, gradle has many of the same properties as bazel. It's just a graph of tasks that take inputs in, and produces artifacts out. So long as the inputs to a task don't change then it can be cached.
- xyzzy123 4y ago
  +1 for gradle, not enough people realise it's a very general build system (not just for java / android). The caching is indeed very good if you set it up properly. In the hands of a level 5 gradle wizard it can also be extended to do ANYTHING, with the usual caveat that you may experience regrets when your gradle wizard leaves.
  - nerdyadventurer 4y ago
    Any tips improve build times on Android projects with Gradle?
avodonosov 4y ago
If you're stuck with large Java / Maven repository and want to speedup builds, check this out: https://github.com/avodonosov/hashver-maven-plugin https://github.com/avodonosov/hashver-maven-plugin
raphlinus 4y ago
I find Neil Mitchell's categories of small[1], medium[2], and huge[3] build systems useful. Blaze is absolutely fantastic as a huge build system, as it can correctly specify the exact semantics of systems with lots of components and complexities such as cross-language bindings, autogenerated files, etc. If you are building a huge system, then a small or medium build tool just won't be up to the task. At that kind of scale, somebody is going to be responsible for the build system working, possibly as a full time job, and it makes sense to tap somebody who understands how Bazel works and knows how to put those ideas into practice. Conversely, as many comments here observe, it is terrible as a small build system. There, you want to be able to get started quickly and pull in dependencies without thinking too hard. A simple but easy to understand approach (even one based on make) might work. This is just a different problem than what Bazel solves. My personal opinion (and I should emphasize, I am absolutely not speaking for anyone here, least of all Google) is that the best way forward is to take the ideas of Bazel (hermetic and deterministic builds) and package them as a good small build system, perhaps even compatible with Bazel so you don't have to rewrite build rules all the time. I also think compilers and tools can and should evolve to become good citizens in such a world. But I have no idea how things will go, it's equally plausible the entire space of build systems will continue to suck as they have for decades. [1]: http://neilmitchell.blogspot.com/2021/09/small-project-build-systems.html http://neilmitchell.blogspot.com/2021/09/small-project-build... [2]: https://neilmitchell.blogspot.com/2021/09/reflecting-on-shake-build-system.html https://neilmitchell.blogspot.com/2021/09/reflecting-on-shak... [3]: https://neilmitchell.blogspot.com/2021/09/huge-project-build-systems.html https://neilmitchell.blogspot.com/2021/09/huge-project-build...
- bobsomers 4y ago
  I completely agree with this, having spent an awful lot of time with both Bazel and Make. There is a much tighter, cleaner, simpler build system within Bazel struggling to get out. A judicious second take at it with a minimal focus, while taking some of the best ideas, could be wildly successful I think. It's on my list of things that I will inevitably never get to.
- EdwardDiego 4y ago
  Thanks for the links to a new blog to dive into!
NomDePlum 4y ago
Honestly, even considering this question for any service/codebase that is under 1,000 developers contributing daily commits, or 10,000,000 users is just a vanity project.
infogulch 4y ago
What's the difference between Bazel and the Nix's approach to the problem, Hydra? Are these reasonably comparable?
inoffensivename 4y ago
> Bazel, as Blaze, was created to solve the challenges of Google’s code base, so it’s no surprise that it works well there and that people who’ve spent a lot of time using it make a strong case for everyone using it. I've worked at Google for a long time. Every time I get a chance to work on something outside Google's codebase, it's like a breath of fresh air. I wouldn't recommend Bazel or Blaze to anybody. Sorry to the people who wrote it.
1letterunixname 4y ago
It's painful and doesn't have customization points. It's a dream for enterprise but useless for a hacker. Source: We (MAANG) dumped the thing and rewrote it from scratch.
jesseschalken 4y ago
I migrated a monorepo including C++, Kotlin, Java, TypeScript and Python to Bazel. It's no small feat, and the DX varies widely across languages and platforms, but it's absolutely worth it. `bazel test //...` from a fresh clone builds & tests everything including gRPC/protobuf code generation, custom code generation, downloading packages, toolchains and linters, dependencies between langs, test suites with coverage across multiple languages. Integration testing is a breeze through data dependencies. The reproducibility guarantees means we can reference container image SHAs in our Terraform and if the image didn't change the deploy is a no-op. Bazel is an outstanding build system that handily solves a lot of practical problems in software engineering. Not just "at scale". Practical problems at any scale.
double_nan 4y ago
It took me some time to start understanding bazel properly. But after that using it is very easy and enjoyable (especially inside a monorepo). But learning curve is getting very steep very fast outside of simple BUILD rules. Edit: enjoyable to a point that i half considering doing contracts for bazel based build systems.
dboreham 4y ago
Obviously: never.
sgtnoodle 4y ago
Back in 2016 or so, I was in need of a more capable build system to replace a combination of disorganized cmake and XML IAR project files. Having come from Google, I briefly considered bazel, but it just didn't seem mature enough to build on top of. I went with SCons, and built a bazel-like abstraction layer on top of it. My goal was to abstract away SCons in 90% of use cases. It's still in use 6 years later, and the company's engineering team has grown by at least a factor of 10. People love to complain about build infrastructure, but it's held up pretty well.
oxff 4y ago
Great article
oftenwrong 4y ago
One of the interviewees from this article, Son Luong Ngoc (sluongng), has recently been writing a great series of blog posts on the design of Bazel: https://sluongng.hashnode.dev/ https://sluongng.hashnode.dev/ Tools like Earthly (and other BuildKit HLB frontend languages) will help more teams get some of the main benefits of Bazel with a lower bar of complexity. If you already know how to write Dockerfiles and Makefiles, you can write Earthfiles without much additional learning necessary. It provides a build that is repeatable, incremental, self-contained, never dirty, shared-cacheable, et cetera.
jesseschalken 4y ago
It's worth noting that this is the blog of Earthly which is in many ways a Bazel competitor.
eckesicle 4y ago
I worked at Coinbase for four years, where Bazel is the build system of choice. It's so so much worse than the homegrown software it replaced. Hermetic sounds nice in theory, but doesn't actually matter and comes with a massive cost. 1. Bazel is slow. Like really fucking slow compared to your native build tools. Startup time can be insane if you use multiple rules (languages) in your repo. There's a great feeling when you get to work in the morning to build your go project but you have to install the latest python, nodejs, and ruby toolchains because someone updated them on master. The cache works well, except with a 1000 devs something will always be invalidated on master. 2. The documentation sucks. It's written to explain concepts with how things work, with no examples of how to actually complete tasks you care about. Of course that wouldn't help either because the bazel setup you're using is heavily customised. 3. Everyone on your team now has to learn yet another DSL, and a bunch of commands to run. I would easily spend 5 hours per week either waiting for, or debugging some issue in bazel. All this for what benefit? Everyone's on the same version of a dependency? Not even sure this is a desirable property. Also the monorepo is slow as jelly to work with and many tools or editors struggle to open it. Good luck getting code completion to work well too. It's one of those ideas that are nice in theory, but awful in practice. It's possible that with enough manpower you may be able to use it effectively, but we certainly were not.
- zquestz 4y ago
  I second this. Bazel is more trouble than it's worth.
- traceroute66 4y ago
  > Bazel is slow. Like really fucking slow .... Probably because its written in Java. What sane person writes Java in the 2020s ? I mean, aside from having to dance the JVM maintenance dance, you're also faced with the fact that Java is a memory hog unless you spend half your life tweaking obscure parameters in XML files. The sooner Java dies the better, frankly. Java was great back in the day, even ahead of its time, but today it's just a relic. Use Go, use Rust ... hell, practically anything is better than Java.
  - 4y ago
wging 4y ago
Under "When to Use Bazel - Jason’s Opinion", that is not a particularly good description of how builds (especially code generation) worked at most of Amazon. At least he sort of gestures at the non-universality of his experience by mentioning his experience was in Prime Video and only for 8 months. I have no trouble believing that various pockets of the company found various different wrong ways to do things, I've seen some of those wrong ways myself. But taking a dependency on shared code (for better or for worse) rather than copying something yourself should not have been hard or infrequently done, and wouldn't be for most people using the standard tools. Generating code and having multiple downstream builds depend on that same generated code (e.g. the output artifact in the form of a .jar, in the case of Java) should also be very straightforward, though it was also possible (if a bad idea) for each consumer to run the same generator. Builds at Amazon were definitely not problem-free, but they were different problems that don't really resemble that description.
abiro 4y ago
I looked at Bazel ~5 years ago for a C++, Scala, JS, Python codebase and it looked like the perfect solution, but there seemed to be a lot of breaking changes in Bazel at the time so I passed in the end. Then I started a project this year that will be a Rust, Swift, Kotlin, JS codebase (maybe C# too if Rust for Windows isn’t stable enough) and I had a look at Bazel again, but already the official tutorial from last year didn’t work on the latest stable release so I gave up on it again. Anyone had similar experience and pushed through? Is it worth it? I currently have a bunch of Python scripts and will most likely end up with Gradle which is ok but not great in my experience.
edumucelli 4y ago
I remember struggling with Bazel and Tensorflow for months! I just went to see how things are and it seems that things have not changed much in years, e.g.: https://github.com/bazelbuild/bazel/issues/14639 https://github.com/bazelbuild/bazel/issues/14639 This is kind of project that you won't probably need soon, unless you are really being paid extra money for the suffering.
DriftRegion 4y ago
I mostly do embedded C and my favorite use of bazel not mentioned in the article is the near-seamless swapping of toolchains. It makes unit testing very pleasant. See this example: https://github.com/driftregion/bazel-c-rust-x86_linux-armv7_baremetal/blob/main/readme.md https://github.com/driftregion/bazel-c-rust-x86_linux-armv7_...

212 points by jvolkman 4y ago

Have there been any reviews about Ninja and gn? Are those comparable to Bazel?

One big gotcha that folks need to appreciate is that you don't get the full power of bazel just by getting baseline bazel to work and build things. You need to really be sensitive to your physical dependencies and not lump similar code together just because they play a similar role and are convenient to place in the same file rather than having that same source sharded out to distinct files. You need to partition source into all its relevant pieces and lump no more functionality in that source than is needed by clients. Bazel encourages the most minute fine grained resolution of dependencies because if you do that and manage it right you unlock the power of the cache. Once you successfully partition all the bits of software to their segregated sections of the build graph DAG then you're positioned to really leverage the test result cache and can start bypassing huge expensive tests that are completely irrelevant to the body of code you're developing on. Bazel is not just a build toolchain it is a test toolchain as well. The problems it solves with tests with respect to running only relevant & applicable tests to the changes being pushed is what many people have dreamed about before or came up with some crude solution to approximate the dependencies. If my test did not print out `(cached)` I know something in the dependency graph was disturbed in some way that may be completely not obvious to me, especially in a gigantic repo. Bazel query is also worth mentioning to visualize dependencies with its xml output and which can also enforce architectural barriers asserting empty `somepath` queries between two targets that should not meet in the middle somewhere. It would be neat one day if bazel integrates in some semantic logical level of dependency checking beyond the physical but that may be too expensive of an operation.

adamgordonbell 4y ago

Author here. I wanted to get my head wrapped around when Bazel was an excellent build solution, so I interviewed six people with a lot of Bazel experience and picked their brains. This somewhat long article hopes to answer questions about Bazel for future people in a similar position to me. If you know a lot of Bazel then you might not learn much but if you’ve vaguely heard of it and are not sure when its the tool that should be reached for I’m hoping this will help.

jxramos 4y ago
I particularly enjoyed the history aspect of it, these things give you broad contours of where pools of experience exist and what specific organizations were responsible for driving and advancing things.
lostdog 4y ago
This was a great way to pull wisdom from many different people, and get a broad idea of what everyone's experience is. Looking forward to more posts! I'm also surprised at how little experience everyone had with bazel alternatives. I wonder if buck or pants is easier to work with.
- adamgordonbell 4y ago
  Thanks! Oscar in the article had experience with Pants but preferred Bazel and Bazel certainly has the most usage at this scale.

choeger 4y ago

Very nice work. Worth a read!

lxe 4y ago

When not to use Bazel: when attempting to shoehorn it into an existing JavaScript monorepo.

spion 4y ago
Yep. If its just JS, Turborepo would probably be a better choice nowadays.
- bsimpson 4y ago
  I haven't used Turborepo, but I have really liked wireit. wireit is from the Lit team at Google. They understand both Bazel and JS. As I understand, wireit is their attempt to make the Bazel niceties JS-native.
- nyadesu 4y ago
  Would you mind explaining why? I use Nx and it's been good, not perfect tho, so at this point I'm very skeptical about toolchaining solutions that claim they 'just work'
  - spion 4y ago
    Last time I looked at Nx it was very opinionated and I couldn't get it to work with an existing repo. In contrast I got running with turborepo in about 15 minutes in a fairly complex workspace project. Its very flexible, but in this case the flexibility is simple: you specify your input globs, output globs and task dependencies for each package.json command you want it to support and it will use that as the basis to detect cache invalidation. Works rather well.

mattboardman 4y ago

Bazel is a great tool, but as mentioned in the article the support from Google is rather limited. If bazel ever got to the level Terraform is with providers it could really take off. Something I'd love to see is a .bazel_mod file where you could pull in other bazel workspaces (similar to go.mod) and reference packages within those workspaces.

dimator 4y ago
That's there now: https://bazel.build/build/bzlmod https://bazel.build/build/bzlmod
- mattboardman 4y ago
  TIL. Thanks for the link!
n0us 4y ago
you can already do that with the WORKSPACE file

dietrichepp 4y ago

I've been using Bazel for side projects these days, including small retro game projects for old game consoles. The entry price was high, but it works so well I have a hard time imagining working without it. For retro game projects, the core of your game might be written in C or C++, which you want cross-compiled. That's easily within reach of stuff like makefiles. But then I start adding a bunch of custom tooling--I want to write tools that process sprites, audio, or 3D models. These days I tend to write those tools in Go. I'm also working with other people, and I want to be able to cross-compile these tools for Windows, even though I'm developing on Linux or macOS. My Bazel repository will download the ARM toolchain automatically and do everything it needs to build the target. I don't really need to set up my development environment at all--I just need to install Bazel and a C compiler, and Bazel will handle the rest. I don't need to install anything else on my system, and the C compiler is only needed because I'm using protocol buffers (Bazel downloads and compiles protoc automatically).

synergy20 4y ago
Makefile can do all that too, without java dependency and straightforward to read at any time later, what real extra benefit Bazel brings in here.
- ajconway 4y ago
  Bazel doesn't have any user-facing java dependency. > what real extra benefit Bazel brings in here Builds are hermetic by default, so unless the developer chooses to escape the sandbox, everything is guaranteed to build on other machines with no additional setup. (Also, I genuinely hate when I have to manually install build dependencies system-wide and pray that there will not be any conflicts. Having everything pinned to specific sha256 or git hashes by design is a breath of fresh air)
  - blueblob 4y ago
    I used a M1 Mac to try to build tensorflow and tensorflow-text and it is very untrue that everything is guaranteed to build on other machines with no additional setup. The parenthesized comment is funnier to me because I had to download a specific bazel version to build.

rjsw 4y ago

Unless it becomes easier to port to a new operating system, never.

mplewis9z 4y ago

Personally, I've found Bazel's tooling and dependency management to be extremely aggressive at pushing you to online-only development as your project scales in size. A company I worked for that used it lost multiple person-days for every engineer they had when Covid hit and the VPN went to crap. It's great at being able to offload work to a remote server, but in my opinion that should never be the only way you can get work done. Local development should always be the default, with remote execution being an _option_ when available.

hamandcheese 4y ago
> A company I worked for that used it lost multiple person-days for every engineer they had when Covid hit and the VPN went to crap. As far as criticisms go, “we lost a few days of work when a global pandemic shut the world down” seems pretty minor. That said, I too am a fan of offline development and I find that bazel makes that pretty easy for me. I just run bazel fetch before I get on an airplane.
unicornmama 4y ago
Bazel fully supports local development. How is it that you needed anything remote to do your work?
lima 4y ago
When implemented correctly, it also helps local development. You can use remote cache with local execution for quick compiles locally.

trzy 4y ago

Last year I tried doing a side project with a talented ex-Google buddy who insisted we set up Bazel to replace my simple Makefile. Three weeks later it still wasn’t working on my Windows box. We had a mixed Python and C++ code base and I like to use MinGW64 gcc on Windows. He blamed Windows and tried to get me to switch to Mac (no thanks, lol) and eventually he lost interest and gave up. The project went on to win an OpenCV funded competition and became the basis of a startup — good job, GNU Make! So the answer IMHO to “when to use Bazel” is “never” :)

dbt00 4y ago
I've used bazel at medium size (100+ engineers, multiple languages) and tried to use it at a small size (20 engineers, almost all go with a tiny bit of C) and I think the "never" and "unless there's no other way" answers from this interview are pretty good.
thayne 4y ago
Bazel probably wasn't a good fit for that project, especially if your makefile was simple. But where it does work well, is when you have a large complex codebase in a monorepo, and you need reliable caching to keep build times down.
- thechao 4y ago
  C + Makefile + CCache: I just `find . -name "*.h"` and throw that as the dependency of every C file, and then let ccache/make figure out the rest. My code base runs upwards of 10+ million LOC, and incremental builds are in the "hundreds of milliseconds" and "clean" builds (ccache clean, mind you) are about 30s. Truly clean builds (on a scratch system) are ~2 minutes. We do spend time making sure our very long files still compile very quickly -- we're careful to stay away from code that exercises pathological paths through the compiler. I've found that a good laptop can compile ~20kloc/(s-thread) throughput with parallel builds (it varies from 1000–100000loc/(s-thread)).
  - thayne 4y ago
    That's specific to c. Bazel is often used in cases where you have multiple languages in use, and possibly dependencies between projects using different languages. > I just `find . -name "*.h"` That sounds like it would cause problems with not triggering a build if a .h file is deleted. And would trigger unnecessary builds if any .h file is modified. But if your builds are that fast, maybe that isn't a problem.

setheron 4y ago

I can never wrap my head around writing a tool chain in bazel

steeve 4y ago

Trying to learn Bazel by porting an existing project to it is a very hard way to get into this amazing tool. If possible, start fresh. I don’t see myself working without it from now on.

tdeck 4y ago
The thing is, something like Bazel only pays dividends on a large complex project. When you're starting from scratch it's unlikely you'll have any problems that Bazel helps to solve, but you'll be spending a lot of time on the ceremony of setting it up.
- steeve 4y ago
  That's not really true. My personal threshold is if there is more than one language involved.

EdSchouten 4y ago

Regardless of whether you should use Bazel or not, my hope is that any future build systems attempt to adopt Bazel's remote execution protocol (or at least a protocol that is similar in spirit): https://github.com/bazelbuild/remote-apis https://github.com/bazelbuild/remote-apis In my opinion the protocol is fairly well designed.

an_d_rew 4y ago

I will quote my past self (https://news.ycombinator.com/item?id=26064845 https://news.ycombinator.com/item?id=26064845) We use Bazel’s rules_docker as well, and I would caution someone evaluating it with a note from out experience. What Bazel does well (and as well as Bazel fits your use-case) Bazel does extremely well and is a reproducible joy to use. But if you stray off that path even a tiny bit, you’re often in for a surprisingly inexplicable, unavoidable, far-reaching pain. For example, rules_docker is amazing at laying down files in a known base image. Everything is timestamped to the 1970 unix epoch, for reproducibility, but hey, it’s a bit-perfect reproduction. Need to run one teensy executable to do even the smallest thing that’s trivial with a Dockerfile? Bazel mist create the image transfer it to a docker daemon, run the command, transfer it back... your 1 kb change just took 5 minutes and 36 gb of data being tarred, gzipped, and flung around (hopefully-the-local) network. It may not be a dealbreaker, and you may not care, but be forewarned that these little surprises creep up fairly ofen from unexpected quarters! Edit: after 2-ish years of Bazel, I would say that for 99% of developers and organizations, the most likely answer is "never".

grahamplace 4y ago

Worth pointing out that Build Buddy (YC W20) has an enterprise Bazel product that tries to make some of the management of Bazel easier. I haven't used it myself so no commentary, I just have a friend who started there recently so it's on my radar https://www.buildbuddy.io/ https://www.buildbuddy.io/ https://www.ycombinator.com/companies/buildbuddy https://www.ycombinator.com/companies/buildbuddy

miiiiiike 4y ago

I lost a month learning Bazel last year. Never again. Here’s a challenge: Create a Bazel built Angular library using only the publicly available documentation. Here, I’ll save you some time: You can’t. What documentation exists is flawed and what isn’t flawed has massive holes. It was so bad that I had to ask a few Google employees if there was secret internal documentation somewhere. There isn’t, and, they almost all hated using it as well. I went back to Make and had the whole repo building in an afternoon. Bazel is a great idea but like most other Google OSS projects it doesn’t have strong enough documentation to form a community or enough of a community to create good docs. Even if I brute forced my way into using Bazel, I couldn’t ask an employee to learn it. I hear that Blaze is much easier to work with.

deleted 4y ago
[deleted]

bmohlenhoff 4y ago

I tried using Bazel a while back but immediately ran into a few issues. The existing codebase I was working with did not lay out its dependencies and files in the manner expected by Bazel so dealing with dependency/include hell was frustrating. Then there was a large portion of the project that depended on generated code for its interfaces (ie. similar to protobuf but slightly different) and trying to dive into the Bazel rules and toolchain with no other Bazel experience was not fun. I attempted to build off of the protobuf implementation but kept finding additional layers to the onion that didn't exactly translate to the non protobuf tooling. The project documentation seemed out of date in this area (ie. major differences in the rules engine between 1.0 and subsequent versions) and I couldn't find many examples to look at other than overly simplified toy examples. All in all a frustrating experience. I could not even get far enough along to compile the interfaces for the project.

Too 4y ago
Did you find genrule? It’s a generic escape hatch rule that can be used to just run any program. Of course it still needs the input and output dependencies declared and you are responsible for ensuring it is hermetic.
- bmohlenhoff 4y ago
  Yes, I started out by using genrule before going down the custom rule path. I was able to properly trigger bazel to generate the interface bindings but then lost the thread when trying to properly configure the output files to act as proper dependencies to other code (ie. generate C++ headers and libraries that could then be properly listed as dependencies by client code.) I think part of the issue was that the generated code from the 3rd party tool did not align with the Bazel expectations about code layout (ie. paths relative to WORKSPACE). Likely solvable but not in the amount of time I had available to dedicate to the effort.

smeagull 4y ago

Haven't yet gotten a single project using Bazel to build on any of my machines. I don't think I'd adopt a build system that is so frustrating.

cstrahan 4y ago

There’s an issue I reported (along with a proof of concept fix) over 4 years ago, that has yet to be fixed: building a mixed source project containing Go & C++ & C++ protocol buffers results in silently broken binaries as rules_go will happily not forward along the linker arguments that the C++ build targets (the protobuf ones, using the built in C++ rules) declare. See https://github.com/bazelbuild/rules_go/issues/1486 https://github.com/bazelbuild/rules_go/issues/1486 Not very confidence inspiring when Google’s build system falls over when you combine three technologies that are used commonly throughout Google’s code base (two of which were created by Google). If you’re Google, sure, use Bazel. Otherwise, I wouldn’t recommend it. Google will cater to their needs and their needs only — putting the code out in the open means you get the privilege of sharing in their tech debt, and if something isn’t working, you can contribute your labor to them for free. No thanks :)

vlovich123 4y ago
Does protobufs rely on static initialization or did you link the wrong bug? That would be extremely strange because there’s generally a disallowance of any / all non trivial static initialization in library code Google-wide, especially for protobufs.

badoongi 4y ago

My take - Avoid Bazel as long as you can, for most companies the codebase is not big enough to actually need distributed builds, if you've hit this problem Bazel is probably the best thing you can do today, if you're that big you can probably spare the few dozen headcount needed to make Bazel experience in your company solid. Bazel takes on dependency management, which is probably an improvement for a C++ codebase where there is no de-facto package manager. For modern languages like golang where a package manager is widely adopted by the community it's usually just a pain. e.g Bazel's offering for golang relies on generating "Bazel configurations" for the repositories to fetch, this alternative definition of dependencies is not what all the existing go tooling are expecting, and so to get the dev tooling working properly you end up generating one configuration from the other having 2 sources of truth, and pains when there's somehow a mismatch. Bazel hermeticity is very nice in theory, in practice many of the existing toolchains used by companies that are using Bazel are non-hermetic, resulting in many companies stuck in the process of "migration to Bazel remote execution" forever. Blaze works well in Google's monorepo where all the dependencies are checked in (vendored), the WORKSPACE file was an afterthought when it was opensourced, and the whole process of fetching remote dependencies in practice becomes a pain for big monorepos (I just want to build this small golang utility, `bazel build //simple:simple` and you end up waiting for a whole bunch of python dependencies you don't need to be downloaded). And this is all before talking about Javascript, if your JS codebase wasn't originally designed the way Bazel expects it you're probably up for some fun.

intellix 4y ago

as an angular developer, Bazel was promised to replace webpack behind the scenes and suddenly everything would get fast and all our problems would be solved. Months... Years went by until eventually it became an opt-in solution and I guess they just gave up trying to use it.

deleted 4y ago

[deleted]

xwowsersx 4y ago

Only tangentially related, but can someone give me the pitch for Earthly? I think I'm just missing something basic. Our CI consists of tests, builds and deployments (all docker containers) that are kicked off in bitbucket pipelines. Is there some way that either our local development or CI builds can be improved by using earthly? Appreciate any info!

jsw 4y ago

Bazel is a fully reproducible and hermetic build system. A lot of painstaking work goes into it producing the exact same artifacts build after build. And that provides some interesting properties that you can leverage for artifact caching, deployments, and CICD. We very happily runny a polyglot monorepo w/ 5+ languages, multiple architectures, with fully reproducible artifacts and deployment manifests, all deployed to almost all AWS regions on every build. We update tens of thousands of resources in every environment for every build. The fact that Bazel is creating reproducible artifacts allow us to manage this seamlessly and reliably. Clean builds take an hour+, but our GH self-hosted runners often complete commit to green build for our devs in less than a minute. The core concept of Bazel is very simple: explicitly declare the input you pass to a rule/tool and explicitly declare the output it creates. If that can click, you're half way there.

jeffbee 4y ago
Having spent a great deal of time getting bazel set up as you describe, I feel you have given readers a misleading impression. Bazel does not come out of the box that way. It uses whatever toolchain is laying around on the host, by default. It builds with system headers and links with system libraries. It looks at your environment variables. You need to do a lot of surgery on the toolchain to make it hermetic and reproducible.
- tehlike 4y ago
  Sounds like a job for docker or nix?
  - jeffbee 4y ago
    The bazel way to do it is to put the tools in an archive, refer to the archive and its checksum in your workspace, and execute the build in a completely empty sandbox.
    - hakre 4y ago
      Can you share a bit about the completely empty sandbox? Is this a build-root with it's own user and environment? Or does it build inside worktree, e.g. a subdirectory. Or can both be done?

xvedejas 4y ago

Bazel doesn't allow targeting a lot of platforms (especially embedded) from Rust, even when the Rust ecosystem supports these targets. Something is off with its design if new work needs to be done for every platform that's already available behind an interface that's as consistent as what rustc gives. What is supported needs to be inferred from this file, as far as I can tell: https://github.com/bazelbuild/rules_rust/blob/main/rust/platform/triple_mappings.bzl https://github.com/bazelbuild/rules_rust/blob/main/rust/plat...

DriftRegion 4y ago
https://github.com/driftregion/bazel-c-rust-x86_linux-armv7_baremetal/blob/main/readme.md https://github.com/driftregion/bazel-c-rust-x86_linux-armv7_...
- xvedejas 4y ago
  This doesn't instruct how to target `thumbv7em-none-eabihf`, for example

davikr 4y ago

Kinda tangential, but I still can't believe there are "Modern" CMake books hundreds of pages long. I still end up using it sometimes, but I have no confidence in the rules I write, specially when compared to good old Visual Studio solutions or my favorite tool which is Premake.

cbHXBY1D 4y ago

I use bazel at work and it does make cross language work very easy. I mostly write C++ so it's one of the few good options out there. But I've also used CMAKE with vcpkg for personal projects and found that to be a really good option. Vcpkg is really the best package manager for C++ right now and seems to work seamlessly on Windows, too. Hopefully bazel's bzlmod can take off and get as many libraries as vcpkg.

sa46 4y ago

My experience with Bazel at a startup (also used Blaze at Google): The good: - Amazing for Go backends. I can push a reproducible Docker image to Kubernetes with a single Bazel command. I can run our entire backend with a single command that will work on any developer's box. - Amazing for testing. All of our backends tests use a fully independent Postgres database managed by Bazel. It's really nice not having to worry about shared state in databases across tests. - We can skip Docker on macOS for development which provides on the order of a 10x speedup for tests. - BuildBuddy provides a really nice CI experience with remote execution. Bazel tests are structured so I can see exactly which tests failed without trawling through thousands of lines of log output. I've heard good things about EngFlow but BuildBuddy was free to start. - Really nice for schema driven codegen like protobufs. The bad: - Bazel is much too hard for TypeScript and JavaScript. We don't use Bazel for our frontend. New bundlers like Vite are much faster and have a developer experience that's hard to replicate with Bazel. Aspect.dev is doing some work on this front. One large hurdle is there's not automatic BUILD file dependency updater like Gazelle for Go. - Windows support is too hard largely because most third party dependencies don't work well with Windows. - Third party dependencies are still painful. There's ongoing work with bzlmod but my impression is that it won't be usable for a couple of years. - Getting started was incredibly painful. However, the ongoing maintenance burden is a few hours per month.

lhorie 4y ago
> Bazel is much too hard for TypeScript and JavaScript. We don't use Bazel for our frontend. New bundlers like Vite are much faster and have a developer experience that's hard to replicate with Bazel. Aspect.dev is doing some work on this front. One large hurdle is there's not automatic BUILD file dependency updater like Gazelle for Go. I built this[0] for the Bazel + yarn setup that we use at Uber. We currently manage a 1000+ package monorepo with it. [0] https://github.com/uber-web/jazelle https://github.com/uber-web/jazelle
hamandcheese 4y ago
> Amazing for testing. All of our backends tests use a fully independent Postgres database managed by Bazel. It's really nice not having to worry about shared state in databases across tests. I would love to know more about this. We use bazel and Postgres, but there is no Postgres isolation between tests currently which is very predictably a large source of flakyness, and also a blocker for increasing test parallelism. I’ve been scheming on perhaps using pg_tmp orchestrated outside of bazel, but if there’s a way to do everything in bazel that would make the transition way easier.

higgins 4y ago

Ctrl-F never

WatchDog 4y ago

While it's not as natural to use for non-jvm projects, gradle has many of the same properties as bazel. It's just a graph of tasks that take inputs in, and produces artifacts out. So long as the inputs to a task don't change then it can be cached.

xyzzy123 4y ago
+1 for gradle, not enough people realise it's a very general build system (not just for java / android). The caching is indeed very good if you set it up properly. In the hands of a level 5 gradle wizard it can also be extended to do ANYTHING, with the usual caveat that you may experience regrets when your gradle wizard leaves.
- nerdyadventurer 4y ago
  Any tips improve build times on Android projects with Gradle?

avodonosov 4y ago

If you're stuck with large Java / Maven repository and want to speedup builds, check this out: https://github.com/avodonosov/hashver-maven-plugin https://github.com/avodonosov/hashver-maven-plugin

raphlinus 4y ago

I find Neil Mitchell's categories of small[1], medium[2], and huge[3] build systems useful. Blaze is absolutely fantastic as a huge build system, as it can correctly specify the exact semantics of systems with lots of components and complexities such as cross-language bindings, autogenerated files, etc. If you are building a huge system, then a small or medium build tool just won't be up to the task. At that kind of scale, somebody is going to be responsible for the build system working, possibly as a full time job, and it makes sense to tap somebody who understands how Bazel works and knows how to put those ideas into practice. Conversely, as many comments here observe, it is terrible as a small build system. There, you want to be able to get started quickly and pull in dependencies without thinking too hard. A simple but easy to understand approach (even one based on make) might work. This is just a different problem than what Bazel solves. My personal opinion (and I should emphasize, I am absolutely not speaking for anyone here, least of all Google) is that the best way forward is to take the ideas of Bazel (hermetic and deterministic builds) and package them as a good small build system, perhaps even compatible with Bazel so you don't have to rewrite build rules all the time. I also think compilers and tools can and should evolve to become good citizens in such a world. But I have no idea how things will go, it's equally plausible the entire space of build systems will continue to suck as they have for decades. [1]: http://neilmitchell.blogspot.com/2021/09/small-project-build-systems.html http://neilmitchell.blogspot.com/2021/09/small-project-build... [2]: https://neilmitchell.blogspot.com/2021/09/reflecting-on-shake-build-system.html https://neilmitchell.blogspot.com/2021/09/reflecting-on-shak... [3]: https://neilmitchell.blogspot.com/2021/09/huge-project-build-systems.html https://neilmitchell.blogspot.com/2021/09/huge-project-build...

bobsomers 4y ago
I completely agree with this, having spent an awful lot of time with both Bazel and Make. There is a much tighter, cleaner, simpler build system within Bazel struggling to get out. A judicious second take at it with a minimal focus, while taking some of the best ideas, could be wildly successful I think. It's on my list of things that I will inevitably never get to.
EdwardDiego 4y ago
Thanks for the links to a new blog to dive into!

NomDePlum 4y ago

Honestly, even considering this question for any service/codebase that is under 1,000 developers contributing daily commits, or 10,000,000 users is just a vanity project.

infogulch 4y ago

What's the difference between Bazel and the Nix's approach to the problem, Hydra? Are these reasonably comparable?

inoffensivename 4y ago

> Bazel, as Blaze, was created to solve the challenges of Google’s code base, so it’s no surprise that it works well there and that people who’ve spent a lot of time using it make a strong case for everyone using it. I've worked at Google for a long time. Every time I get a chance to work on something outside Google's codebase, it's like a breath of fresh air. I wouldn't recommend Bazel or Blaze to anybody. Sorry to the people who wrote it.

1letterunixname 4y ago

It's painful and doesn't have customization points. It's a dream for enterprise but useless for a hacker. Source: We (MAANG) dumped the thing and rewrote it from scratch.

jesseschalken 4y ago

I migrated a monorepo including C++, Kotlin, Java, TypeScript and Python to Bazel. It's no small feat, and the DX varies widely across languages and platforms, but it's absolutely worth it. `bazel test //...` from a fresh clone builds & tests everything including gRPC/protobuf code generation, custom code generation, downloading packages, toolchains and linters, dependencies between langs, test suites with coverage across multiple languages. Integration testing is a breeze through data dependencies. The reproducibility guarantees means we can reference container image SHAs in our Terraform and if the image didn't change the deploy is a no-op. Bazel is an outstanding build system that handily solves a lot of practical problems in software engineering. Not just "at scale". Practical problems at any scale.

double_nan 4y ago

It took me some time to start understanding bazel properly. But after that using it is very easy and enjoyable (especially inside a monorepo). But learning curve is getting very steep very fast outside of simple BUILD rules. Edit: enjoyable to a point that i half considering doing contracts for bazel based build systems.

dboreham 4y ago

Obviously: never.

sgtnoodle 4y ago

Back in 2016 or so, I was in need of a more capable build system to replace a combination of disorganized cmake and XML IAR project files. Having come from Google, I briefly considered bazel, but it just didn't seem mature enough to build on top of. I went with SCons, and built a bazel-like abstraction layer on top of it. My goal was to abstract away SCons in 90% of use cases. It's still in use 6 years later, and the company's engineering team has grown by at least a factor of 10. People love to complain about build infrastructure, but it's held up pretty well.

oxff 4y ago

Great article

oftenwrong 4y ago

One of the interviewees from this article, Son Luong Ngoc (sluongng), has recently been writing a great series of blog posts on the design of Bazel: https://sluongng.hashnode.dev/ https://sluongng.hashnode.dev/ Tools like Earthly (and other BuildKit HLB frontend languages) will help more teams get some of the main benefits of Bazel with a lower bar of complexity. If you already know how to write Dockerfiles and Makefiles, you can write Earthfiles without much additional learning necessary. It provides a build that is repeatable, incremental, self-contained, never dirty, shared-cacheable, et cetera.

jesseschalken 4y ago

It's worth noting that this is the blog of Earthly which is in many ways a Bazel competitor.

eckesicle 4y ago

I worked at Coinbase for four years, where Bazel is the build system of choice. It's so so much worse than the homegrown software it replaced. Hermetic sounds nice in theory, but doesn't actually matter and comes with a massive cost. 1. Bazel is slow. Like really fucking slow compared to your native build tools. Startup time can be insane if you use multiple rules (languages) in your repo. There's a great feeling when you get to work in the morning to build your go project but you have to install the latest python, nodejs, and ruby toolchains because someone updated them on master. The cache works well, except with a 1000 devs something will always be invalidated on master. 2. The documentation sucks. It's written to explain concepts with how things work, with no examples of how to actually complete tasks you care about. Of course that wouldn't help either because the bazel setup you're using is heavily customised. 3. Everyone on your team now has to learn yet another DSL, and a bunch of commands to run. I would easily spend 5 hours per week either waiting for, or debugging some issue in bazel. All this for what benefit? Everyone's on the same version of a dependency? Not even sure this is a desirable property. Also the monorepo is slow as jelly to work with and many tools or editors struggle to open it. Good luck getting code completion to work well too. It's one of those ideas that are nice in theory, but awful in practice. It's possible that with enough manpower you may be able to use it effectively, but we certainly were not.

zquestz 4y ago
I second this. Bazel is more trouble than it's worth.
traceroute66 4y ago
> Bazel is slow. Like really fucking slow .... Probably because its written in Java. What sane person writes Java in the 2020s ? I mean, aside from having to dance the JVM maintenance dance, you're also faced with the fact that Java is a memory hog unless you spend half your life tweaking obscure parameters in XML files. The sooner Java dies the better, frankly. Java was great back in the day, even ahead of its time, but today it's just a relic. Use Go, use Rust ... hell, practically anything is better than Java.
- 4y ago

wging 4y ago

Under "When to Use Bazel - Jason’s Opinion", that is not a particularly good description of how builds (especially code generation) worked at most of Amazon. At least he sort of gestures at the non-universality of his experience by mentioning his experience was in Prime Video and only for 8 months. I have no trouble believing that various pockets of the company found various different wrong ways to do things, I've seen some of those wrong ways myself. But taking a dependency on shared code (for better or for worse) rather than copying something yourself should not have been hard or infrequently done, and wouldn't be for most people using the standard tools. Generating code and having multiple downstream builds depend on that same generated code (e.g. the output artifact in the form of a .jar, in the case of Java) should also be very straightforward, though it was also possible (if a bad idea) for each consumer to run the same generator. Builds at Amazon were definitely not problem-free, but they were different problems that don't really resemble that description.

abiro 4y ago

I looked at Bazel ~5 years ago for a C++, Scala, JS, Python codebase and it looked like the perfect solution, but there seemed to be a lot of breaking changes in Bazel at the time so I passed in the end. Then I started a project this year that will be a Rust, Swift, Kotlin, JS codebase (maybe C# too if Rust for Windows isn’t stable enough) and I had a look at Bazel again, but already the official tutorial from last year didn’t work on the latest stable release so I gave up on it again. Anyone had similar experience and pushed through? Is it worth it? I currently have a bunch of Python scripts and will most likely end up with Gradle which is ok but not great in my experience.

edumucelli 4y ago

I remember struggling with Bazel and Tensorflow for months! I just went to see how things are and it seems that things have not changed much in years, e.g.: https://github.com/bazelbuild/bazel/issues/14639 https://github.com/bazelbuild/bazel/issues/14639 This is kind of project that you won't probably need soon, unless you are really being paid extra money for the suffering.

DriftRegion 4y ago

I mostly do embedded C and my favorite use of bazel not mentioned in the article is the near-seamless swapping of toolchains. It makes unit testing very pleasant. See this example: https://github.com/driftregion/bazel-c-rust-x86_linux-armv7_baremetal/blob/main/readme.md https://github.com/driftregion/bazel-c-rust-x86_linux-armv7_...