8 ms·

Last year I tried doing a side project with a talented ex-Google buddy who insisted we set up Bazel to replace my simple Makefile. Three weeks later it still wa

by trzy 4y ago

Last year I tried doing a side project with a talented ex-Google buddy who insisted we set up Bazel to replace my simple Makefile. Three weeks later it still wasn’t working on my Windows box. We had a mixed Python and C++ code base and I like to use MinGW64 gcc on Windows. He blamed Windows and tried to get me to switch to Mac (no thanks, lol) and eventually he lost interest and gave up. The project went on to win an OpenCV funded competition and became the basis of a startup — good job, GNU Make! So the answer IMHO to “when to use Bazel” is “never” :)

dbt00 4y ago
I've used bazel at medium size (100+ engineers, multiple languages) and tried to use it at a small size (20 engineers, almost all go with a tiny bit of C) and I think the "never" and "unless there's no other way" answers from this interview are pretty good.
thayne 4y ago
Bazel probably wasn't a good fit for that project, especially if your makefile was simple. But where it does work well, is when you have a large complex codebase in a monorepo, and you need reliable caching to keep build times down.
- thechao 4y ago
  C + Makefile + CCache: I just `find . -name "*.h"` and throw that as the dependency of every C file, and then let ccache/make figure out the rest. My code base runs upwards of 10+ million LOC, and incremental builds are in the "hundreds of milliseconds" and "clean" builds (ccache clean, mind you) are about 30s. Truly clean builds (on a scratch system) are ~2 minutes. We do spend time making sure our very long files still compile very quickly -- we're careful to stay away from code that exercises pathological paths through the compiler. I've found that a good laptop can compile ~20kloc/(s-thread) throughput with parallel builds (it varies from 1000–100000loc/(s-thread)).
  - thayne 4y ago
    That's specific to c. Bazel is often used in cases where you have multiple languages in use, and possibly dependencies between projects using different languages. > I just `find . -name "*.h"` That sounds like it would cause problems with not triggering a build if a .h file is deleted. And would trigger unnecessary builds if any .h file is modified. But if your builds are that fast, maybe that isn't a problem.
    - thechao 4y ago
      It is specific to C/C++, for sure -- I started with that. ccache is going to be as fast as any checksum dependency tracker --- otherwise you're relying on a timestamp, which isn't any better than "bare" Make. We use `*.d` rules from the `-MD` flag to prevent the "deleted header" case. Considering the brevity of the Makefile (it looks like a "basic" build), performance, and the practical robustness, it's in a pretty great trade-off space.
      - thayne 4y ago
        From what you've described I definitely don't think it would be worth migrating what you have to bazel. My point is just that there are environments where it does provide benefits.
- nsteel 4y ago
  At the risk of looking dumb here, don't incremental builds take care of this, how is caching different? ? Make gives me incremental builds and my makefile lets me define the inputs and the outputs. What am I missing with Bazel?!
  - jvolkman 4y ago
    1. Make uses mtime to determine whether inputs have changed. Bazel uses a hash of all inputs to the target (file contents + other inputs like environment variables) 2. Make lets you define inputs and outputs, but doesn't enforce them. Bazel sandboxes build actions; if you try to import a dependency that you haven't listed as a direct or transitive dependency, the build will fail. This is leads to users never (or at least very rarely) having to run `bazel clean`.
IshKebab 4y ago
That's like saying you should never build a house with concrete foundations because they take so long to dig and pour and the first layer of bricks doesn't need them anyway! Get back to me when you work for a company with a monorepo that has to build and test everything in CI for every change (taking something like 200 CPU hours) because they didn't have the foresight to use Bazel.
- drewcoo 4y ago
  > company with a monorepo that has to build and test everything in CI for every change That smells like covering up an architecture problem (tight coupling) with a build tool. Not a recipe for success.
  - deathanatos 4y ago
    I don't think that's actually the case. Bazel, AIUI, understands the entire dependency tree and can cache steps effectively, which a Makefile cannot (alone) do. (You could certainly build it into one.) CI tooling is orthogonal; your CI tooling can invoke Bazel or Makefile+custom caching, it wouldn't really matter. But the problem in CI is that you lack state; whereas a developer's machine might e.g., have a bunch of .o files from a previous build, CI will be starting from a clean slate¹. So builds take longer. My understanding of Bazel is that it has decent support for caching and can thus makes those rebuilds a lot faster, which it has b/c it understands exactly the inputs to the steps at hand. (Which, if you write your own caching layer, you'd have to figure out.) … but … I've also seen the same thing the top commenter has, with Bazel: endless suggestions to use Bazel … but migrating a large existing codebase is pain, and if you're not willing to put your work where your mouth is … then the status quo prevails. ¹While here CI lacking state is a sort of negative b/c of the time we need to rebuild it, normally, I think this is a good thing: it means you're constantly verifying you can build from scratch. Or, at least, something close to scratch, modulo things like vendoring. (I.e., if you do an "apt-get" from Ubuntu's servers in your CI pipeline … well … your build obviously isn't quite from scratch.)
    - JustLurking2022 4y ago
yamtaddle 4y ago
My experience is that Windows is a constant source of headaches if you're supporting development and builds on multiple platforms and trying not to just write every build-related thing twice, Bazel or no Bazel. Likelihood of it being a PITA goes up fast the more complex the build & tools (probably why Make did better than Bazel). I assume it's OK if it's your only platform—though I got my programming start on Windows, with open-source languages, and distinctly recall how managing the same tools and builds got way easier and more reliable when I switched to Linux. Luckily WSL2 is getting semi-decent and can call out to Windows tools, so it's getting easier to standardize on one or another unixy scripting language for glue, even if some of your tools on Windows are native. Part of the trouble with it, though, is that there are multiple ways to end up with some kind of linux-ish environment on it, and that all of them introduce quirks or oddities. You end up having to wrestle with stupid questions like "which copy of OpenSSH is this tool using?" It's probably less-bad if you're in a position to strictly dictate what Windows dev machines have installed on them, I suppose. Git is installed, but only with options X, Y, and Z checked, no other configurations allowed; WSL2 is installed and has packages A, B, and C installed; and so on, crucially preventing the installation of other things that might try to pile on more weirdness and unpredictability. Those kinds of messes are possible on macOS or Linux or FreeBSD or what have you, but generally don't happen. I think it happens on Windows because every tool's trying to vendor in various Linux-compat dependencies rather than force a particular system configuration on users, or having to try to deal with whatever maybe-not-actually-compatible similar tools the user has installed. So they vendor them in to reduce friction and the rate of spurious bug reports. Basically, the authors of these tools are seeing the same problems as anyone trying to configure cross-platform development with Windows in the mix, and are throwing up their hands and vendoring in their deps, which compounds the problem for anyone trying to coordinate several such tools because there's a tendency for everything to end up with weird configurations that don't play well together.
- trzy 4y ago
  All excellent points but Windows is a very widely used platform (perhaps the most widely used, by some metrics) and it’s actually quite nice for development. It’s hard to take tools seriously that don’t work on it. Unpopular opinion but I think the Unix approach of lots of little tools with arcane configuration files all blaming each other sucks. I get why it exists, why it became popular, and why it remains popular in some scenarios, but I don’t feel beholden to it.
hot_gril 4y ago
In general, if someone wants to beef up the tooling in my project that I know is fine to begin with, I still say "I don't mind, go ahead, but I will continue using my old tooling until yours works with no regression." Half the time, they abandon it. I don't even know how Bazel works cause I've never felt enough pain with makefiles etc to research an alternative. Also, Google tooling knowledge doesn't transfer very well to outside projects. Everything is special on the inside there. They use an internal version of Bazel called Blaze that's totally integrated with everything, with entire teams dedicated to the tooling, and only a few officially supported programming languages, so of course it works smoothly.
- dessant 4y ago
  > Google tooling knowledge doesn't transfer very well to outside projects. A good number of their open source projects fall apart in real-world use. If you visit their repositories you'll often get the sense that beyond working on interesting problems, their engineers have no desire to properly maintain these projects for the next few years. The gamble of relying on a Google product unfortunately also applies to their open source projects.
  - phist_mcgee 4y ago
    Maybe that's because the average tenure at google is so short?
    - UncleMeat 4y ago
      Average tenure at Google is so short because they have grown ridiculously. The actual attrition rates are slower than industry norms.
      - hot_gril 4y ago
        This is correct. People were always shocked how many SWEs were newer than them after working for half a year or so.
    - 4y ago
londons_explore 4y ago
Don't use tools outside what they're designed for... Specifically, Bazel is really at home with Java/C++ and Linux. Sure, it kinda works elsewhere, but you should be considering other options.
lhorie 4y ago
I mean, Python (moreso its ecosystem) is notoriously difficult to get working w/ Bazel idiomatically. Bazel is advertised as a language agnostic system (and to be fair, it's better at it than, say, Buck), but in practice it tends to work better w/ stacks that already have some degree of hermeticity (Go, Java), whereas YMMV very much with the more loosey-goosey stacks (Python, JS). IMHO, Bazel is a classic example of Conway's law[0], and it falls squarely in the "big corp" class of software. You have to be running into issues like 6+ digit CI compute costs or latency SLOs in hundreds-of-teams monorepos before Bazel really starts to make sense as a potential technical solution. [0] https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law
WastingMyTime89 4y ago
You are talking about a two men project becoming a one man project. You are not the target of a complex build system. At that size, you would be fine with basically anything. I’m really glade that Meson and Ninja exist however. I hate GNU Make like few things in my toolbox. M4 really is an awful language.
adastra22 4y ago
I used bazel for my latest project, mainly as an excuse to learn it. I ended up spending waaay too much time debugging bazel instead of working on my code, and I still can’t properly support windows because dependencies don’t build. I will never use bazel again. Not worth the effort required, and doesn’t work out of the box as advertised.
rawoke083600 4y ago
Haha, I like the frankness :) To take it a step further. I never left the "comfort" of bash-build and bash-deploy scripts. In most of my projects (pro and personal) there is a deploy-aws-prod.sh and deploy-aws-dev.sh (or some variation) None is longer than a 10-20 bash commands. These "projects" are webservice, ml-models, batch-processing(think distributed clusters) It's ugly and perfect-enough at the same time. YMMV
Cthulhu_ 4y ago
I mean was there anything wrong about the makefile? Did your product need reproducible builds? This is becoming more and more of a pet peeve of mine, about to become an outright annoyance; the adding or changing of tools without things actually improving, or the purported improvement not actually being relevant or even close to the most important thing that time should be spent on. https://mcfunley.com/choose-boring-technology https://mcfunley.com/choose-boring-technology is a good starting point for moving away from this mindset. I had a Go project up until earlier this year, I could set it up myself. I picked makefiles to build it (not bazel, that would be overkill; not mage, that would mean more coding), and it worked just fine. I never had a compelling reason to move away from it. Actually it was more than fine, it was a relief; my previous experience with tools like that had been Maven (where everything is XML and you need a plugin to do basic things like move or remove a file) and the Javascript ecosystem, from before everything was merged into package.json / nodejs-style. Dependency management too.
- ParetoOptimal 4y ago
  I typically want reproducible builds to never ever have to deal with build issues and to be able to free up that space to think. Boring technologies are easier for people better and making due with what they have and accepting sometimes very large limitations. ADHD makes boring tasks harder for instance, and boring technologies give you no escape from the tedium. With a little fancier technology though, you at least have hope ;)
strikelaserclaw 4y ago
i think its more like for super small startups, the best move is always to keep things as simple as possible (like basic build/deploy scripts) and work more on the product.

by trzy 4y ago

dbt00 4y ago

I've used bazel at medium size (100+ engineers, multiple languages) and tried to use it at a small size (20 engineers, almost all go with a tiny bit of C) and I think the "never" and "unless there's no other way" answers from this interview are pretty good.

thayne 4y ago

Bazel probably wasn't a good fit for that project, especially if your makefile was simple. But where it does work well, is when you have a large complex codebase in a monorepo, and you need reliable caching to keep build times down.

thechao 4y ago
C + Makefile + CCache: I just `find . -name "*.h"` and throw that as the dependency of every C file, and then let ccache/make figure out the rest. My code base runs upwards of 10+ million LOC, and incremental builds are in the "hundreds of milliseconds" and "clean" builds (ccache clean, mind you) are about 30s. Truly clean builds (on a scratch system) are ~2 minutes. We do spend time making sure our very long files still compile very quickly -- we're careful to stay away from code that exercises pathological paths through the compiler. I've found that a good laptop can compile ~20kloc/(s-thread) throughput with parallel builds (it varies from 1000–100000loc/(s-thread)).
- thayne 4y ago
  That's specific to c. Bazel is often used in cases where you have multiple languages in use, and possibly dependencies between projects using different languages. > I just `find . -name "*.h"` That sounds like it would cause problems with not triggering a build if a .h file is deleted. And would trigger unnecessary builds if any .h file is modified. But if your builds are that fast, maybe that isn't a problem.
  - thechao 4y ago
    It is specific to C/C++, for sure -- I started with that. ccache is going to be as fast as any checksum dependency tracker --- otherwise you're relying on a timestamp, which isn't any better than "bare" Make. We use `*.d` rules from the `-MD` flag to prevent the "deleted header" case. Considering the brevity of the Makefile (it looks like a "basic" build), performance, and the practical robustness, it's in a pretty great trade-off space.
    - thayne 4y ago
      From what you've described I definitely don't think it would be worth migrating what you have to bazel. My point is just that there are environments where it does provide benefits.
nsteel 4y ago
At the risk of looking dumb here, don't incremental builds take care of this, how is caching different? ? Make gives me incremental builds and my makefile lets me define the inputs and the outputs. What am I missing with Bazel?!
- jvolkman 4y ago
  1. Make uses mtime to determine whether inputs have changed. Bazel uses a hash of all inputs to the target (file contents + other inputs like environment variables) 2. Make lets you define inputs and outputs, but doesn't enforce them. Bazel sandboxes build actions; if you try to import a dependency that you haven't listed as a direct or transitive dependency, the build will fail. This is leads to users never (or at least very rarely) having to run `bazel clean`.

IshKebab 4y ago

That's like saying you should never build a house with concrete foundations because they take so long to dig and pour and the first layer of bricks doesn't need them anyway! Get back to me when you work for a company with a monorepo that has to build and test everything in CI for every change (taking something like 200 CPU hours) because they didn't have the foresight to use Bazel.

drewcoo 4y ago
> company with a monorepo that has to build and test everything in CI for every change That smells like covering up an architecture problem (tight coupling) with a build tool. Not a recipe for success.
- deathanatos 4y ago
  I don't think that's actually the case. Bazel, AIUI, understands the entire dependency tree and can cache steps effectively, which a Makefile cannot (alone) do. (You could certainly build it into one.) CI tooling is orthogonal; your CI tooling can invoke Bazel or Makefile+custom caching, it wouldn't really matter. But the problem in CI is that you lack state; whereas a developer's machine might e.g., have a bunch of .o files from a previous build, CI will be starting from a clean slate¹. So builds take longer. My understanding of Bazel is that it has decent support for caching and can thus makes those rebuilds a lot faster, which it has b/c it understands exactly the inputs to the steps at hand. (Which, if you write your own caching layer, you'd have to figure out.) … but … I've also seen the same thing the top commenter has, with Bazel: endless suggestions to use Bazel … but migrating a large existing codebase is pain, and if you're not willing to put your work where your mouth is … then the status quo prevails. ¹While here CI lacking state is a sort of negative b/c of the time we need to rebuild it, normally, I think this is a good thing: it means you're constantly verifying you can build from scratch. Or, at least, something close to scratch, modulo things like vendoring. (I.e., if you do an "apt-get" from Ubuntu's servers in your CI pipeline … well … your build obviously isn't quite from scratch.)
  - JustLurking2022 4y ago

yamtaddle 4y ago

My experience is that Windows is a constant source of headaches if you're supporting development and builds on multiple platforms and trying not to just write every build-related thing twice, Bazel or no Bazel. Likelihood of it being a PITA goes up fast the more complex the build & tools (probably why Make did better than Bazel). I assume it's OK if it's your only platform—though I got my programming start on Windows, with open-source languages, and distinctly recall how managing the same tools and builds got way easier and more reliable when I switched to Linux. Luckily WSL2 is getting semi-decent and can call out to Windows tools, so it's getting easier to standardize on one or another unixy scripting language for glue, even if some of your tools on Windows are native. Part of the trouble with it, though, is that there are multiple ways to end up with some kind of linux-ish environment on it, and that all of them introduce quirks or oddities. You end up having to wrestle with stupid questions like "which copy of OpenSSH is this tool using?" It's probably less-bad if you're in a position to strictly dictate what Windows dev machines have installed on them, I suppose. Git is installed, but only with options X, Y, and Z checked, no other configurations allowed; WSL2 is installed and has packages A, B, and C installed; and so on, crucially preventing the installation of other things that might try to pile on more weirdness and unpredictability. Those kinds of messes are possible on macOS or Linux or FreeBSD or what have you, but generally don't happen. I think it happens on Windows because every tool's trying to vendor in various Linux-compat dependencies rather than force a particular system configuration on users, or having to try to deal with whatever maybe-not-actually-compatible similar tools the user has installed. So they vendor them in to reduce friction and the rate of spurious bug reports. Basically, the authors of these tools are seeing the same problems as anyone trying to configure cross-platform development with Windows in the mix, and are throwing up their hands and vendoring in their deps, which compounds the problem for anyone trying to coordinate several such tools because there's a tendency for everything to end up with weird configurations that don't play well together.

trzy 4y ago
All excellent points but Windows is a very widely used platform (perhaps the most widely used, by some metrics) and it’s actually quite nice for development. It’s hard to take tools seriously that don’t work on it. Unpopular opinion but I think the Unix approach of lots of little tools with arcane configuration files all blaming each other sucks. I get why it exists, why it became popular, and why it remains popular in some scenarios, but I don’t feel beholden to it.

hot_gril 4y ago

In general, if someone wants to beef up the tooling in my project that I know is fine to begin with, I still say "I don't mind, go ahead, but I will continue using my old tooling until yours works with no regression." Half the time, they abandon it. I don't even know how Bazel works cause I've never felt enough pain with makefiles etc to research an alternative. Also, Google tooling knowledge doesn't transfer very well to outside projects. Everything is special on the inside there. They use an internal version of Bazel called Blaze that's totally integrated with everything, with entire teams dedicated to the tooling, and only a few officially supported programming languages, so of course it works smoothly.

dessant 4y ago
> Google tooling knowledge doesn't transfer very well to outside projects. A good number of their open source projects fall apart in real-world use. If you visit their repositories you'll often get the sense that beyond working on interesting problems, their engineers have no desire to properly maintain these projects for the next few years. The gamble of relying on a Google product unfortunately also applies to their open source projects.
- phist_mcgee 4y ago
  Maybe that's because the average tenure at google is so short?
  - UncleMeat 4y ago
    Average tenure at Google is so short because they have grown ridiculously. The actual attrition rates are slower than industry norms.
    - hot_gril 4y ago
      This is correct. People were always shocked how many SWEs were newer than them after working for half a year or so.
  - 4y ago

londons_explore 4y ago

Don't use tools outside what they're designed for... Specifically, Bazel is really at home with Java/C++ and Linux. Sure, it kinda works elsewhere, but you should be considering other options.

lhorie 4y ago

I mean, Python (moreso its ecosystem) is notoriously difficult to get working w/ Bazel idiomatically. Bazel is advertised as a language agnostic system (and to be fair, it's better at it than, say, Buck), but in practice it tends to work better w/ stacks that already have some degree of hermeticity (Go, Java), whereas YMMV very much with the more loosey-goosey stacks (Python, JS). IMHO, Bazel is a classic example of Conway's law[0], and it falls squarely in the "big corp" class of software. You have to be running into issues like 6+ digit CI compute costs or latency SLOs in hundreds-of-teams monorepos before Bazel really starts to make sense as a potential technical solution. [0] https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law

WastingMyTime89 4y ago

You are talking about a two men project becoming a one man project. You are not the target of a complex build system. At that size, you would be fine with basically anything. I’m really glade that Meson and Ninja exist however. I hate GNU Make like few things in my toolbox. M4 really is an awful language.

adastra22 4y ago

I used bazel for my latest project, mainly as an excuse to learn it. I ended up spending waaay too much time debugging bazel instead of working on my code, and I still can’t properly support windows because dependencies don’t build. I will never use bazel again. Not worth the effort required, and doesn’t work out of the box as advertised.

rawoke083600 4y ago

Haha, I like the frankness :) To take it a step further. I never left the "comfort" of bash-build and bash-deploy scripts. In most of my projects (pro and personal) there is a deploy-aws-prod.sh and deploy-aws-dev.sh (or some variation) None is longer than a 10-20 bash commands. These "projects" are webservice, ml-models, batch-processing(think distributed clusters) It's ugly and perfect-enough at the same time. YMMV

Cthulhu_ 4y ago

I mean was there anything wrong about the makefile? Did your product need reproducible builds? This is becoming more and more of a pet peeve of mine, about to become an outright annoyance; the adding or changing of tools without things actually improving, or the purported improvement not actually being relevant or even close to the most important thing that time should be spent on. https://mcfunley.com/choose-boring-technology https://mcfunley.com/choose-boring-technology is a good starting point for moving away from this mindset. I had a Go project up until earlier this year, I could set it up myself. I picked makefiles to build it (not bazel, that would be overkill; not mage, that would mean more coding), and it worked just fine. I never had a compelling reason to move away from it. Actually it was more than fine, it was a relief; my previous experience with tools like that had been Maven (where everything is XML and you need a plugin to do basic things like move or remove a file) and the Javascript ecosystem, from before everything was merged into package.json / nodejs-style. Dependency management too.

ParetoOptimal 4y ago
I typically want reproducible builds to never ever have to deal with build issues and to be able to free up that space to think. Boring technologies are easier for people better and making due with what they have and accepting sometimes very large limitations. ADHD makes boring tasks harder for instance, and boring technologies give you no escape from the tedium. With a little fancier technology though, you at least have hope ;)

strikelaserclaw 4y ago

i think its more like for super small startups, the best move is always to keep things as simple as possible (like basic build/deploy scripts) and work more on the product.