7 ms·

Hey, what a surprise, ex-google engineers dragging everyone into bazel... Maybe you have some really complicated use case that makes cargo insufficient, but re

Hey, what a surprise, ex-google engineers dragging everyone into bazel... Maybe you have some really complicated use case that makes cargo insufficient, but really, it's pretty damn nice and it's integrated with everything. There are languages that need bazel, rust isn't one of them

rtpg 4y ago
Even just for running test suites incrementally, Bazel is pretty nice. If you have CI that takes "too long", Bazel is basically the only serious tool that can get you to incremental CI in a pretty rigorous way. Though I seriously wish there were a better option, because there are a _lot_ of usability things with Bazel that make me very frustrated when working with a repo that is "merely" quite large.
bilekas 4y ago
I'm inclined to agree, but mainly because I can't see an issue where cargo is insufficient, it seems to me that would be an issue somewhere else. Not cargo specific.
- charcircuit 4y ago
  One category where it helps are *-sys packages which are lacking a proper build system. Cargo's build.rs is a messy solution and is pretty limited.
- ajconway 4y ago
  Cargo is insufficient if a project has cross-language dependencies.
- duped 4y ago
  Cargo is sufficient if what you're building is an executable or a Rust library. It still doesn't support post build steps, and integrates poorly with other tools that are used for more general builds.
bjackman 4y ago
Lol, Googler here, I can't deny that I have he urge to force Bazel into the world. It's just, when I joined Google I found a build system I didn't hate. This was the first time. I don't actually love Bazel (Blaze internally), but I love that I don't hate it. Every time I have to learn a new build system in open source work I groan. Why are there so many systems for essentially the same thing?
- carlmr 4y ago
  Question for an insider. Bazel code coverage seems to be one of the roughest edges. At least when running C++ tests. Only lcov is supported, and due to the sandboxing it's impossible to get other tools working since they always spit out extra instrumentation data that's neither cached nor available on the next sandbox. Does blaze support code coverage properly? Does Google just not care about code coverage? Am I just missing some obvious thing in the documentation?
  - bjackman 4y ago
    Hmm sorry I'm quite ignorant about that. Code coverage is pretty well supported in the internal ecosystem but I would not be surprised to hear this is because we designed Bazel around the system Google was already using for coverage.
    - simplotek 4y ago
      > Code coverage is pretty well supported in the internal ecosystem (...) What you're actually saying is that internally Google managed to get a single and very specific use case to work and forced all projects to either adopt it or have no code coverage.
  - Game_Ender 4y ago
    You can get coverage working for almost anything by having each test convert its coverage data to LCOV before stopping, in the same sandbox as the test. I forget the exact details but I think you can do this with a custom run_under test wrapper used during coverage. The wrapper runs your test, with whatever custom coverage system you have, then afterwards it converts to LCOV. Sometimes this might require updates to language specific rules because the test runner itself needs to be configured. The bottom line is that Bazel upstream does not put a lot of attention into this, so it’s under developed outside of Google, because Google internal does not use the OSS rules.
carlmr 4y ago
I mean for C++ having the caching just work like you get with Bazel is a huge thing. I agree that Cargo is already near perfect and I wouldn't suggest you switch to Bazel if you have a Rust project. In our case we have a C++ project with some Python, some Rust. Having everything in Bazel here is good for us.
- simplotek 4y ago
  > I mean for C++ having the caching just work like you get with Bazel is a huge thing. It's a good thing then that cmake+ccache work out of the box by passing the caching tool of your choice like ccache to the compiler launchers.
yosefk 4y ago
I never worked for Google nor worked with anyone from Google who said something like "you should move to Bazel" even once. I also did lots of work on in house build system migrations and optimizations. Do you think that the problems with cargo mentioned in TFA are exaggerated or solvable? I think Bazel's support for caching, parallelism and custom rules are real advantages for large projects. What's wrong with this argument wrt Rust/cargo?
- kibwen 4y ago
  > Do you think that the problems with cargo mentioned in TFA are exaggerated or solvable? I'd say they're exaggerated at least slightly, yes. It starts out by saying "Cargo isn't a build system" (which is already the sort of provocative exaggeration designed to increase participation), and then immediately veers off and starts talking about running tests. Cargo is a build system and dependency manager. It also happens to provide `cargo test` for running unit tests and integration tests, but this support is relatively simple, and if you need to do something weird then you're welcome to write a script to serve as your test runner. But a build system is not a test runner; a build system produces artifacts, and a test runner runs tests. The author appears to be annoyed that Cargo is not a build system for web assembly, which is part of their test process, and obviously Cargo is not. This is a somewhat eye-rolling critique, and does little to motivate rewriting the world in Bazel. The second complaint is that Cargo uses file timestamps to decide when to recompile files. I suppose it is possible that Cargo could then proceed to lex any changed file to produce a comment-free token stream, hash the token stream, and then compare that hash to a previously-computed hash that you've stored in a database somewhere. Frankly, this sounds like a lot of work for essentially zero benefit; how often, exactly, do you go into a file and change only a comment? This is a weak justification. It then mentions how changing git branches can invalidate the cache, but I don't think that's true. Git only updates timestamps on files that have changed when changing branches. If a file has changed when changing a branch, that is, again, probably because it has actual changes, and will almost never be because only a comment has changed. It then makes some suggestions about Cargo not properly tracking changes to some global system resource that they have, which is simply too vague to comment on, and the fact that their prior reasons have been so weak makes me think that if this, of all things, is what they're trying to be vague about, then I assume this must be so unique and specific that even they recognize it's a weak critique.
wewxjfq 4y ago
Cargo's build scripts are a black box, especially when they are used to compile C/C++ libraries.
ddevault 4y ago
At the only job I've had where Bazel was in use, we had a team of 4-5 engineers dedicated to maintaining Bazel for a larger team of about 25. Bazel is a nightmare. I wouldn't wish it on my worst enemy.
- ckdarby 4y ago
  I worked at a company where a DevOps team was 5 to support a team of 35. They hired someone incredibly talented, knowledgeable and driven for change. They shrunk in size and are still over capacity and could now be 2 only. Don't look at headcount to gauge complexity, difficulty, etc. What I've experienced is the larger the team the more likely to have dilution of ownership. The less ownership the less likely each person will spend the energy or fully understand solving root causes.
  - taeric 4y ago
    Oddly, I'd say headcount can often be a driver of complexity. In two ways, one being that more people legit need a more complicated system. You want that so that they interact with the system, not with each other. The other being a bit of a Parkinson's Law. That is specifically that you will expand work to fit the allocated time; but I posit that you will also expand work to fit the people doing it. So, more people pushing ideas into the codebase will keep more ideas in the codebase. Even if fewer would work.
- jmmv 4y ago
  Any build system is a nightmare when you don't understand it. I don't know what that team moved to Bazel from. I suppose it was Makefiles, and I'm sure if that's what the team was using, almost all of those 25 people did not understand them either. What parts of Bazel were a nightmare? What problems were fixed by these 4-5 people that were previously impacting all 25 people routinely?
substation13 4y ago
The article explains why Cargo was insufficient for them.
unicornmama 4y ago
> Hey, what a surprise, ex-google engineers dragging everyone into bazel... Hey look another nimwit that make stupid assumptions without understanding the nature of the problem being solved. > and it's integrated with everything. Oh wow Cargo integrates with Python, shell scripts, C++ and node JS? Holly sh*t that’s amazing. /s
echelon 4y ago
> There are languages that need bazel, rust isn't one of them Speak for yourself. Our Rust monorepo has nearly a dozen apps and counting, lots of shared libraries, C++ dependencies, etc. Tests require multiple cargo invocations. We need a proper build system with distributed build caching. Cargo takes forever and doesn't understand the workloads.
WastingMyTime89 4y ago
Then again, the main issue of Bazel is that it’s complex to setup. If you already have the know how internally, it’s a reasonably good and well supported build system. It’s definitely better than cargo when it comes to caching and the breadth of targets available. Plus cargo is strictly limited to rust.
insanitybit 4y ago
> Maybe you have some really complicated use case that makes cargo insufficient, but really, it's pretty damn nice and it's integrated with everything. I feel like our use case at Grapl wasn't particularly crazy. It was just slightly more than "I'm a dev building a library and publishing it" - we had a workspace with multiple devs and CI/CD. CI/CD was really slow - issues with caching, rerunning unnecessary tests, etc. Plus we had some Python code so unifying our builds would have been ncie.
- swsieber 4y ago
  This. As soon as you have to integrate multiple languages, jumping from the native build tool to an integrated build tool is a must. Why? So that you can saturate all your threads in an efficient way.
- twunde 4y ago
  Hey Colin :wave: (I hope you're bringing Rust to DD) To add onto this, Grapl had a moderate sized codebase with less than 10 engineers. I do think we could have done some optimizations in cargo, but it's simply much easier to use a tool like Bazel/pants etc that does the work for you as long as it supports your language of choice.
  - insanitybit 4y ago
    :D Hey
lern_too_spel 4y ago
In addition to the problems stated in the article that you're ignoring here, in any large enough system, you are going to use multiple languages. Cargo isn't built for that. Bazel is.
- simplotek 4y ago
  > In addition to the problems stated in the article that you're ignoring here What problems, exactly? The author acknowledged cargo works but hand-waves over vague claims of "it does not track dependencies well or support arbitrary build graphs", followed by acknowledging those issues are either non-issues or solved problems whose solution was dismissed without any good reason (caching tools). The author's claims boil down to "I wanted to shoehorn Bazel but had no good reason to, so I double-down on Bazel's only selling point while ignoring everything". It's ok if anyone just wants to try stuff without having any rational or coherent argument to justify it, but let's not call that "problems".
  - pie_flavor 4y ago
    The author gave a concrete example of such a build graph and a concrete example of why caching tools can't be used effectively.
    - simplotek 4y ago
      > such a build graph Nonsense. All build systems implement a DAG of build targets. > why caching tools can't be used effectively Nonsense. There are a myriad of caching tools freely available, from ccache, sccache, build cache, etc. To get them to work, all you need to do is pass compiler launchers to them, which means setting a single env/build flag. The author is either trying hard to make a mountain out of irrelevant molehills, or is outright being disingenuous. Bazel is being sold as a solution desperately grasping at straw-like problems, while being feature incomplete.
      - pie_flavor 4y ago
        > All build systems implement a DAG of build targets. And the example they gave is one that doesn't work so well. > ccache, sccache, build cache, etc The example they gave included the build cache misfiring. ccache doesn't work with Rust. sccache does, but can't be used with incremental compilation or when building binaries, making it essentially useless for what they needed, i.e. speeding up CI and local dev. I really have no idea why you're still avoiding reading the article you're dismissing.
kubota 4y ago
Bazel isn't about a particular language needing its build facilities - its about ALL of your organizations languages being built with one tool, ideally in one monorepository. Its very attractive when you have lots of microservices that would otherwise live in disparate repositories. Using a monorepo and Bazel allows you to share api definitions (proto files) across your entire code base (rules_proto, rules_grpc) , so you see immediately what api changes impact existing code across your entire organization, regardless of what languages each component is coded in.It allows you to containerize (rules_docker) and deploy (rules_k8s) only the components that need to be redeployed with every commit. It allows your entire development team to live, code, test, build, and deploy in harmony.
- simplotek 4y ago
  > its about ALL of your organizations languages being built with one tool, Yeah, that's an effective way to sell someone's promotion, but it's always a promise that is never met and instead it's a constant source of problems. Sometimes reinventing the wheel just buys you a wobbly, squeaky wheel.
  - kubota 4y ago
    When I was at a large money center bank a constant source of problems was trying to deploy the 70+ microservices backing our business unit's main application during "release weekend", or for ad-hoc reasons like the log4shell vulnerability patch. It was very frustrating for our build pipelines to compile / test / static scan the entire millions of lines codebase when only a few libraries have changed. This build often took a few hours for our largest application and at least 10 minutes for each microservice. A few times we ran into breaking api changes and only realized it after it was deployed to production. This is the use case Bazel is serving - god like observability across your organizations code base, quickly compiled.
- chatmasta 4y ago
  My problem with Bazel - at least based on my understanding when I evaluated it - is that you can't use the language-specific package managers for each module in your monorepo, so it's all or nothing. I'd prefer a tool that makes it much easier to defer build tasks to my package manager of choice in each project (eg Yarn for TS, Poetry for Python, etc.) and that lets me define rules at one level of abstraction above this. In our monorepo, we ended up just using a Makefile, since it basically does that. Of course the Makefile is now a few hundred lines and we have about 60 *.compose.yml files, but that's another story. Our current solution still uses the Makefile, but we basically split the monorepo into a Python part and a TypeScript part, and each of those has its own monorepo build system (Yarn workspaces for TS, and Poetry with Pants for Python).

rtpg 4y ago

Even just for running test suites incrementally, Bazel is pretty nice. If you have CI that takes "too long", Bazel is basically the only serious tool that can get you to incremental CI in a pretty rigorous way. Though I seriously wish there were a better option, because there are a _lot_ of usability things with Bazel that make me very frustrated when working with a repo that is "merely" quite large.

bilekas 4y ago

I'm inclined to agree, but mainly because I can't see an issue where cargo is insufficient, it seems to me that would be an issue somewhere else. Not cargo specific.

charcircuit 4y ago
One category where it helps are *-sys packages which are lacking a proper build system. Cargo's build.rs is a messy solution and is pretty limited.
ajconway 4y ago
Cargo is insufficient if a project has cross-language dependencies.
duped 4y ago
Cargo is sufficient if what you're building is an executable or a Rust library. It still doesn't support post build steps, and integrates poorly with other tools that are used for more general builds.

bjackman 4y ago

Lol, Googler here, I can't deny that I have he urge to force Bazel into the world. It's just, when I joined Google I found a build system I didn't hate. This was the first time. I don't actually love Bazel (Blaze internally), but I love that I don't hate it. Every time I have to learn a new build system in open source work I groan. Why are there so many systems for essentially the same thing?

carlmr 4y ago
Question for an insider. Bazel code coverage seems to be one of the roughest edges. At least when running C++ tests. Only lcov is supported, and due to the sandboxing it's impossible to get other tools working since they always spit out extra instrumentation data that's neither cached nor available on the next sandbox. Does blaze support code coverage properly? Does Google just not care about code coverage? Am I just missing some obvious thing in the documentation?
- bjackman 4y ago
  Hmm sorry I'm quite ignorant about that. Code coverage is pretty well supported in the internal ecosystem but I would not be surprised to hear this is because we designed Bazel around the system Google was already using for coverage.
  - simplotek 4y ago
    > Code coverage is pretty well supported in the internal ecosystem (...) What you're actually saying is that internally Google managed to get a single and very specific use case to work and forced all projects to either adopt it or have no code coverage.
- Game_Ender 4y ago
  You can get coverage working for almost anything by having each test convert its coverage data to LCOV before stopping, in the same sandbox as the test. I forget the exact details but I think you can do this with a custom run_under test wrapper used during coverage. The wrapper runs your test, with whatever custom coverage system you have, then afterwards it converts to LCOV. Sometimes this might require updates to language specific rules because the test runner itself needs to be configured. The bottom line is that Bazel upstream does not put a lot of attention into this, so it’s under developed outside of Google, because Google internal does not use the OSS rules.

carlmr 4y ago

I mean for C++ having the caching just work like you get with Bazel is a huge thing. I agree that Cargo is already near perfect and I wouldn't suggest you switch to Bazel if you have a Rust project. In our case we have a C++ project with some Python, some Rust. Having everything in Bazel here is good for us.

simplotek 4y ago
> I mean for C++ having the caching just work like you get with Bazel is a huge thing. It's a good thing then that cmake+ccache work out of the box by passing the caching tool of your choice like ccache to the compiler launchers.

yosefk 4y ago

I never worked for Google nor worked with anyone from Google who said something like "you should move to Bazel" even once. I also did lots of work on in house build system migrations and optimizations. Do you think that the problems with cargo mentioned in TFA are exaggerated or solvable? I think Bazel's support for caching, parallelism and custom rules are real advantages for large projects. What's wrong with this argument wrt Rust/cargo?

kibwen 4y ago
> Do you think that the problems with cargo mentioned in TFA are exaggerated or solvable? I'd say they're exaggerated at least slightly, yes. It starts out by saying "Cargo isn't a build system" (which is already the sort of provocative exaggeration designed to increase participation), and then immediately veers off and starts talking about running tests. Cargo is a build system and dependency manager. It also happens to provide `cargo test` for running unit tests and integration tests, but this support is relatively simple, and if you need to do something weird then you're welcome to write a script to serve as your test runner. But a build system is not a test runner; a build system produces artifacts, and a test runner runs tests. The author appears to be annoyed that Cargo is not a build system for web assembly, which is part of their test process, and obviously Cargo is not. This is a somewhat eye-rolling critique, and does little to motivate rewriting the world in Bazel. The second complaint is that Cargo uses file timestamps to decide when to recompile files. I suppose it is possible that Cargo could then proceed to lex any changed file to produce a comment-free token stream, hash the token stream, and then compare that hash to a previously-computed hash that you've stored in a database somewhere. Frankly, this sounds like a lot of work for essentially zero benefit; how often, exactly, do you go into a file and change only a comment? This is a weak justification. It then mentions how changing git branches can invalidate the cache, but I don't think that's true. Git only updates timestamps on files that have changed when changing branches. If a file has changed when changing a branch, that is, again, probably because it has actual changes, and will almost never be because only a comment has changed. It then makes some suggestions about Cargo not properly tracking changes to some global system resource that they have, which is simply too vague to comment on, and the fact that their prior reasons have been so weak makes me think that if this, of all things, is what they're trying to be vague about, then I assume this must be so unique and specific that even they recognize it's a weak critique.

wewxjfq 4y ago

Cargo's build scripts are a black box, especially when they are used to compile C/C++ libraries.

ddevault 4y ago

At the only job I've had where Bazel was in use, we had a team of 4-5 engineers dedicated to maintaining Bazel for a larger team of about 25. Bazel is a nightmare. I wouldn't wish it on my worst enemy.

ckdarby 4y ago
I worked at a company where a DevOps team was 5 to support a team of 35. They hired someone incredibly talented, knowledgeable and driven for change. They shrunk in size and are still over capacity and could now be 2 only. Don't look at headcount to gauge complexity, difficulty, etc. What I've experienced is the larger the team the more likely to have dilution of ownership. The less ownership the less likely each person will spend the energy or fully understand solving root causes.
- taeric 4y ago
  Oddly, I'd say headcount can often be a driver of complexity. In two ways, one being that more people legit need a more complicated system. You want that so that they interact with the system, not with each other. The other being a bit of a Parkinson's Law. That is specifically that you will expand work to fit the allocated time; but I posit that you will also expand work to fit the people doing it. So, more people pushing ideas into the codebase will keep more ideas in the codebase. Even if fewer would work.
jmmv 4y ago
Any build system is a nightmare when you don't understand it. I don't know what that team moved to Bazel from. I suppose it was Makefiles, and I'm sure if that's what the team was using, almost all of those 25 people did not understand them either. What parts of Bazel were a nightmare? What problems were fixed by these 4-5 people that were previously impacting all 25 people routinely?

substation13 4y ago

The article explains why Cargo was insufficient for them.

unicornmama 4y ago

> Hey, what a surprise, ex-google engineers dragging everyone into bazel... Hey look another nimwit that make stupid assumptions without understanding the nature of the problem being solved. > and it's integrated with everything. Oh wow Cargo integrates with Python, shell scripts, C++ and node JS? Holly sh*t that’s amazing. /s

echelon 4y ago

> There are languages that need bazel, rust isn't one of them Speak for yourself. Our Rust monorepo has nearly a dozen apps and counting, lots of shared libraries, C++ dependencies, etc. Tests require multiple cargo invocations. We need a proper build system with distributed build caching. Cargo takes forever and doesn't understand the workloads.

WastingMyTime89 4y ago

Then again, the main issue of Bazel is that it’s complex to setup. If you already have the know how internally, it’s a reasonably good and well supported build system. It’s definitely better than cargo when it comes to caching and the breadth of targets available. Plus cargo is strictly limited to rust.

insanitybit 4y ago

> Maybe you have some really complicated use case that makes cargo insufficient, but really, it's pretty damn nice and it's integrated with everything. I feel like our use case at Grapl wasn't particularly crazy. It was just slightly more than "I'm a dev building a library and publishing it" - we had a workspace with multiple devs and CI/CD. CI/CD was really slow - issues with caching, rerunning unnecessary tests, etc. Plus we had some Python code so unifying our builds would have been ncie.

swsieber 4y ago
This. As soon as you have to integrate multiple languages, jumping from the native build tool to an integrated build tool is a must. Why? So that you can saturate all your threads in an efficient way.
twunde 4y ago
Hey Colin :wave: (I hope you're bringing Rust to DD) To add onto this, Grapl had a moderate sized codebase with less than 10 engineers. I do think we could have done some optimizations in cargo, but it's simply much easier to use a tool like Bazel/pants etc that does the work for you as long as it supports your language of choice.
- insanitybit 4y ago
  :D Hey

lern_too_spel 4y ago

In addition to the problems stated in the article that you're ignoring here, in any large enough system, you are going to use multiple languages. Cargo isn't built for that. Bazel is.

simplotek 4y ago
> In addition to the problems stated in the article that you're ignoring here What problems, exactly? The author acknowledged cargo works but hand-waves over vague claims of "it does not track dependencies well or support arbitrary build graphs", followed by acknowledging those issues are either non-issues or solved problems whose solution was dismissed without any good reason (caching tools). The author's claims boil down to "I wanted to shoehorn Bazel but had no good reason to, so I double-down on Bazel's only selling point while ignoring everything". It's ok if anyone just wants to try stuff without having any rational or coherent argument to justify it, but let's not call that "problems".
- pie_flavor 4y ago
  The author gave a concrete example of such a build graph and a concrete example of why caching tools can't be used effectively.
  - simplotek 4y ago
    > such a build graph Nonsense. All build systems implement a DAG of build targets. > why caching tools can't be used effectively Nonsense. There are a myriad of caching tools freely available, from ccache, sccache, build cache, etc. To get them to work, all you need to do is pass compiler launchers to them, which means setting a single env/build flag. The author is either trying hard to make a mountain out of irrelevant molehills, or is outright being disingenuous. Bazel is being sold as a solution desperately grasping at straw-like problems, while being feature incomplete.
    - pie_flavor 4y ago
      > All build systems implement a DAG of build targets. And the example they gave is one that doesn't work so well. > ccache, sccache, build cache, etc The example they gave included the build cache misfiring. ccache doesn't work with Rust. sccache does, but can't be used with incremental compilation or when building binaries, making it essentially useless for what they needed, i.e. speeding up CI and local dev. I really have no idea why you're still avoiding reading the article you're dismissing.

kubota 4y ago

Bazel isn't about a particular language needing its build facilities - its about ALL of your organizations languages being built with one tool, ideally in one monorepository. Its very attractive when you have lots of microservices that would otherwise live in disparate repositories. Using a monorepo and Bazel allows you to share api definitions (proto files) across your entire code base (rules_proto, rules_grpc) , so you see immediately what api changes impact existing code across your entire organization, regardless of what languages each component is coded in.It allows you to containerize (rules_docker) and deploy (rules_k8s) only the components that need to be redeployed with every commit. It allows your entire development team to live, code, test, build, and deploy in harmony.

simplotek 4y ago
> its about ALL of your organizations languages being built with one tool, Yeah, that's an effective way to sell someone's promotion, but it's always a promise that is never met and instead it's a constant source of problems. Sometimes reinventing the wheel just buys you a wobbly, squeaky wheel.
- kubota 4y ago
  When I was at a large money center bank a constant source of problems was trying to deploy the 70+ microservices backing our business unit's main application during "release weekend", or for ad-hoc reasons like the log4shell vulnerability patch. It was very frustrating for our build pipelines to compile / test / static scan the entire millions of lines codebase when only a few libraries have changed. This build often took a few hours for our largest application and at least 10 minutes for each microservice. A few times we ran into breaking api changes and only realized it after it was deployed to production. This is the use case Bazel is serving - god like observability across your organizations code base, quickly compiled.
chatmasta 4y ago
My problem with Bazel - at least based on my understanding when I evaluated it - is that you can't use the language-specific package managers for each module in your monorepo, so it's all or nothing. I'd prefer a tool that makes it much easier to defer build tasks to my package manager of choice in each project (eg Yarn for TS, Poetry for Python, etc.) and that lets me define rules at one level of abstraction above this. In our monorepo, we ended up just using a Makefile, since it basically does that. Of course the Makefile is now a few hundred lines and we have about 60 *.compose.yml files, but that's another story. Our current solution still uses the Makefile, but we basically split the monorepo into a Python part and a TypeScript part, and each of those has its own monorepo build system (Yarn workspaces for TS, and Poetry with Pants for Python).