18 ms·

Evaluating Bazel for building Firefox

194 points by dochtman 7y ago

NieDzejkob 7y ago
I'm surprised to see no mention of Tup [1] in the discussion. Verifying the dependency graph for maximizing parallelism is its major feature. [1]: http://gittup.org/tup/examples.html http://gittup.org/tup/examples.html
- theamk 7y ago
  It seems dead? Latest commit was over a year ago, and there are over 100 open issues seemingly with no activity.
- swsieber 7y ago
  Does Tup do more than C? Firefox needs to build some Rust things too.
  - andrewflnr 7y ago
    It's a very general tool. IIRC from playing with it, it doesn't give a fig about whether it's running a compiler at all. I think I was using it for a toy compiler+nasm, actually...
- staktrace 7y ago
  Firefox currently can be built with tup, although it's still experimental and not used in production.
- IshKebab 7y ago
  What is different about Tup? It's home page contains a lot of text but zero information.
  - beagle3 7y ago
    Tup runs what is effectively a bash build script, but does so under ptrace (or equivalent) so it knows about every file that has been checked (and IIRC even those that were missing). So you basically write a "from scratch" build script and that's it. To rebuild, it will rerun the script, except that it knows which file it reads (and which mustn't exist), so if nothing is changed, it can skip that stage. There is no inherent difference between the compiler binary (say, /usr/bin/gcc), the system include (/usr/include/stdio.h), and the project source. It makes for a super simple build script, which, after one successful build, gives tup enough information about dependencies and parallelization opportunities, while at the same time also tracks toolchains better than you can manually (does /usr/bin/gcc shell to /usr/bin/x86-gcc or /usr/bin/amd64-gcc ? did one of them change but not the other? was it the one we need? tup knows) Unfortunately, it is nontrivial; it can uses ptrace or fuse to trace file access on Linux, IIRC has some other hack on Windows - and little to no support for other systems. Also, the overhead is in the single digit % last time I checked.
    - IshKebab 7y ago
      I suspected as much but why doesn't the page supposedly explaining what Tup is actually say any of that?
- adgasf 7y ago
  Windows support seems to be very lacking.
  - dblohm7 7y ago
    That would be pretty easy to fix, TBH. Tup's author works for Mozilla, and Mozilla has the expertise to add that functionality. As always, it's a matter of will and resourcing.
m0zg 7y ago
Bazel is excellent on Linux and macOS, but it's pretty bad on Windows. I'm not sure what (if anything) they're going to do about that. Maybe they'll fix it on Windows though. That'd be pretty cool.
- Onanymous 7y ago
  Never tried it on Windows. For me it was pretty bad on Linux and macOS.
- NickGerleman 7y ago
  Microsoft internally uses something pretty similar to Bazel. I'm not familiar enough with the two to fully understand motivations for the divergence. It supposedly initially ran poorly on Mac, just due to differences in what is cheap on different platforms. I wonder what kind of inherent performance differences you would find in something "Windows first" vs "Linux/MacOS first" https://github.com/microsoft/BuildXL https://github.com/microsoft/BuildXL
  - deleted 7y ago
    [deleted]
  - m0zg 7y ago
    I think there's nothing insurmountable in getting Bazel to work well on Windows. It's just a chicken and egg problem: few people use it there, so it's not up to par, so few people use it there because it's not up to par.
  - EricBurnett 7y ago
    The differences between bazel and BuildXL are less about platform and more about philosophy. Bazel is about opinionated builds - declare everything, and you get fast correct builds. Hard to adopt, but super powerful. BuildXL is more about meeting codebases where they are - take an existing msbuild|cmake|dscript build with its under-specified dependencies, and figure out how to parallelize it safely. This delivers quick wins, but comes with real costs as well (notably: no caching; labour-intensive magic to guess an initial seed of deps, with a performance cliff when something changes). (Disclaimer: Googler, work closely with the bazel team. I like bazel; I like BuildXL for expanding the space of build ideas, but have long-term concerns it settles in a local maxima that'll be hard to get out of).
txtsd 7y ago
Is meson not good?
- adgasf 7y ago
  Meson does not have the same feature set as Bazel. Most notably sandboxing and remote execution.
  - qznc 7y ago
    Other build systems (CMake, Meson, etc) should be able to gain the sandboxing feature. The remote execution feels more fundamental to me and not so easy to replicate.
adgasf 7y ago
Before it comes up: Bazel ships with its own copy of the JVM so you do not need to install Java to use it.
- qhwudbebd 7y ago
  ...if you're happy running random third party binaries instead of building from source.
  - adgasf 7y ago
    Do you review all of the source-code that you run? If you trust the source-code due to its origin, why not trust a binary?
    - qhwudbebd 7y ago
      It doesn't matter whether I trust it or not: these binaries are typically linked against a different C library, assume a different filesystem layout and dynamic linker location, and are often only available for x86-64 in any case. I build binaries locally to get something that's marginally more likely to work on my system, rather than because I believe I could somehow catch nasties buried in the source code even if I read every line. Too many years of seeing the ingenuity of IOCCC entries has taught me I won't succeed at that. :)
- account42 7y ago
  Is this supposed to be a good thing?
  - PedroBatista 7y ago
    Probably better than Nodejs Gulp Yo man ugabuga npm here's 3.4 million files for you to download or Python is it 2.7 or 3.1? Pip too and Ansible let's make that terminal a disco night of text! or some Go "Works great for my static file blog" build tool.
    - 7y ago
rjsw 7y ago
Firefox will currently build on more platforms than Bazel does. Getting fed up of the endless list of new build systems.
- qhwudbebd 7y ago
  I agree, and inflicting Bazel's horrible Java dependency on Firefox would be a tragedy.
  - mda 7y ago
    How is it different than depending on e.g. Python?
    - rjsw 7y ago
      Python doesn't currently require a working Python binary to build itself. Both Bazel and Java do.
      - new_realist 7y ago
        A C compiler, which Python needs, is arguably as complex as either a Python or Java runtime.
        jcranmer 7y ago
        A C compiler is a functional requirement of any operating system these days.
        pjmlp 7y ago
        You won't go very far with it on Android, ChromeOS, iOS, iPadOS, mbed, Arduino, console OSes,...
        creatornator 7y ago
hartror 7y ago
TIL Thunderbird is still a thing.
- pja 7y ago
  It’s still one of the best IMAP clients IMO.
  - CrLf 7y ago
    That says a lot about the state of IMAP clients.
  - gspr 7y ago
    mbsync/isync changed my email life. http://isync.sourceforge.net/mbsync.html http://isync.sourceforge.net/mbsync.html
    - NilsIRL 7y ago
      I what way? What do you use it with?
boris 7y ago
I wonder if they considered build2[1]? It seems to tick most of their boxes. In particular, it has first-class Windows support and, in the upcoming release, even native Clang-targeting-MSVC support (not the clang-cl wrapper)[2]. While some of the features they are looking for (like faster build times which I assume means distributed compilation and caching) are not there yet, we are working on them (and would welcome help)[3]. [1] https://build2.org https://build2.org [2] https://stage.build2.org/?builds=&cf=windows_10-clang* https://stage.build2.org/?builds=&cf=windows_10-clang* [3] https://github.com/build2/README/blob/master/roadmap.md https://github.com/build2/README/blob/master/roadmap.md
- dochtman 7y ago
  One of the important parts in the post seems to be that it is not just a C++ build system, there are many other tasks executed through the build system. In that sense, it seems like build2 isn't a great fit.
  - boris 7y ago
    On the contrary, build2 is a language-agnostic, general-purpose build system. There is the bash module[1], for example. And now that we have support for dynamically buildable/loadable build system modules, we can develop support for more languages, code generators, etc., without bundling it with the build system core. I myself am itching to take a stab at Rust. Hopefully will find time after the release. [1] https://build2.org/build2/doc/build2-build-system-manual.xhtml#module-bash https://build2.org/build2/doc/build2-build-system-manual.xht...
    - qznc 7y ago
      The frontpage starts with "build2 is an open source (MIT), cross-platform build toolchain for developing and packaging C and C++ code." My impression about the modules you linked to: A secondary feature because many C/C++ project need some additional scripting.
      - boris 7y ago
steveklabnik 7y ago
> The typical solution is to become more conservative in how you build things such that you can be sure that Y is always built before X…but typically by making the dependency implicit by, say, ordering the build commands Just So, and not by actually making the dependency explicit to make itself. Maybe specifying the explicit dependency is rather difficult, or maybe somebody just wants to make things work. After several rounds of these kind of fixes, you wind up with Makefiles that are (probably) correct, but probably not as fast as it could be, because you’ve likely serialized build steps that could have been executed in parallel. And untangling such systems to the point that you can properly parallelize things and that you don’t regress correctness can be…challenging. We had a particularly "fun" instance of this sort of Makefile weirdness back when we used Make for Rust... https://github.com/rust-lang/rust/blob/efec34a95a76212b2324d98f3f6d94a1397c2544/mk/prepare.mk#L44-L51 https://github.com/rust-lang/rust/blob/efec34a95a76212b2324d... (Discussed on HN: https://news.ycombinator.com/item?id=13655081 https://news.ycombinator.com/item?id=13655081)
ndesaulniers 7y ago
Of all the build systems I've used, Bazel/Blaze has sucked the least.
munificent 7y ago
Hermetic declarative build systems like Bazel feel exactly like sound statically typed languages to me. It's really hard to get all of your dependencies precisely specified declaratively. The urge to hack in a little imperative code or jam in a build step that you just "know" needs to happen in a certain order is always there and it can be very difficult to satisfy the taskmaster that is the build system. When you have weird cases like a script that produces a variable set of output files based on the content of some input file, it's tough to get that working in the build system. But... if you climb that cliff, there are many rewards at the top. Like the article notes, you can reuse intermediate build artifacts between developers (because it's not possible for those artifacts to accidentally depend on developer-specific state), you can automatically parallelize (because the build system can tell declaratively which tasks do no depend on each other) and you can ask it interesting queries about the dependency graph. That sounds an awful lot like the runtime safety, performance, and static analysis static types give you. The challenge is that for small programs, dynamic types are just easier to get going with. And every build script starts pretty small. So you have a plethora of simple, hackable build systems like make that work fine at first but then get more and more painful as the program grows. With software, there is at least some customer-visible pain that can enable you to justify a rewrite in a statically-typed language. But build infrastructure is always a hidden cost so it's really hard to get the resources to migrate to a different build language. Maybe the lesson here is to bite the bullet and start with a hermetic build system at first.
- adgasf 7y ago
  Except the Bazel syntax is really good. Compiling a few C++ files into a binary is just: cc_binary( name = "app", srcs = glob([ "*.hpp", "*.cpp" ]), ) That's more concise than most build systems AND it's cross-platform.
  - joshuamorton 7y ago
    Although I'd strongly suggest not doing that bazel works better if you have 1 library per header (or source file for non c langs). It helps to have tools to manage deps for your though.
jayd16 7y ago
I'm trying to wrap my head around whether Bazel is worth looking into for the kind of work loads I typically work in. There doesn't seem like that much benefit if you have single language code base and especially if you already have a single build. I suppose I see the benefit of this setup if you keep many libraries and want to pin to the latest instead of, an explicit lib version. If you're in a multi-repo system you're probably not doing this but I can see how Bazel could actually make this possible. Can someone shed some light on the CI story? Can Bazel do code deployment as well? Is that a smart thing to do with Bazel?
- lima 7y ago
  Bazel is worth looking into as soon as you have dependency graphs that your language's native build system can't deal with efficiently, or you have multiple languages that have dependencies on each other's artifacts. A pure Go codebase? Not worth it. Personally, I use Bazel even for small projects as soon as they involve things like generated code or gRPC. Yes, there are multiple rulesets for deployments, like rules_k8s[1] and rules_docker[2]. Of course, you can easily build your own custom deployment pipeline. [1]: https://github.com/bazelbuild/rules_k8s https://github.com/bazelbuild/rules_k8s [2]: https://github.com/bazelbuild/rules_docker https://github.com/bazelbuild/rules_docker
xvilka 7y ago
The problem of Bazel is the JVM requirement. Why not to use something native and more lightweight?
- pjmlp 7y ago
  You can compile Java to native code. And in the case you don't want to, it is hardly any different from shipping a bunch of .so/.dll with the application.

194 points by dochtman 7y ago

I'm surprised to see no mention of Tup [1] in the discussion. Verifying the dependency graph for maximizing parallelism is its major feature. [1]: http://gittup.org/tup/examples.html http://gittup.org/tup/examples.html

theamk 7y ago
It seems dead? Latest commit was over a year ago, and there are over 100 open issues seemingly with no activity.
swsieber 7y ago
Does Tup do more than C? Firefox needs to build some Rust things too.
- andrewflnr 7y ago
  It's a very general tool. IIRC from playing with it, it doesn't give a fig about whether it's running a compiler at all. I think I was using it for a toy compiler+nasm, actually...
staktrace 7y ago
Firefox currently can be built with tup, although it's still experimental and not used in production.
IshKebab 7y ago
What is different about Tup? It's home page contains a lot of text but zero information.
- beagle3 7y ago
  Tup runs what is effectively a bash build script, but does so under ptrace (or equivalent) so it knows about every file that has been checked (and IIRC even those that were missing). So you basically write a "from scratch" build script and that's it. To rebuild, it will rerun the script, except that it knows which file it reads (and which mustn't exist), so if nothing is changed, it can skip that stage. There is no inherent difference between the compiler binary (say, /usr/bin/gcc), the system include (/usr/include/stdio.h), and the project source. It makes for a super simple build script, which, after one successful build, gives tup enough information about dependencies and parallelization opportunities, while at the same time also tracks toolchains better than you can manually (does /usr/bin/gcc shell to /usr/bin/x86-gcc or /usr/bin/amd64-gcc ? did one of them change but not the other? was it the one we need? tup knows) Unfortunately, it is nontrivial; it can uses ptrace or fuse to trace file access on Linux, IIRC has some other hack on Windows - and little to no support for other systems. Also, the overhead is in the single digit % last time I checked.
  - IshKebab 7y ago
    I suspected as much but why doesn't the page supposedly explaining what Tup is actually say any of that?
adgasf 7y ago
Windows support seems to be very lacking.
- dblohm7 7y ago
  That would be pretty easy to fix, TBH. Tup's author works for Mozilla, and Mozilla has the expertise to add that functionality. As always, it's a matter of will and resourcing.

m0zg 7y ago

Bazel is excellent on Linux and macOS, but it's pretty bad on Windows. I'm not sure what (if anything) they're going to do about that. Maybe they'll fix it on Windows though. That'd be pretty cool.

Onanymous 7y ago
Never tried it on Windows. For me it was pretty bad on Linux and macOS.
NickGerleman 7y ago
Microsoft internally uses something pretty similar to Bazel. I'm not familiar enough with the two to fully understand motivations for the divergence. It supposedly initially ran poorly on Mac, just due to differences in what is cheap on different platforms. I wonder what kind of inherent performance differences you would find in something "Windows first" vs "Linux/MacOS first" https://github.com/microsoft/BuildXL https://github.com/microsoft/BuildXL
- deleted 7y ago
  [deleted]
- m0zg 7y ago
  I think there's nothing insurmountable in getting Bazel to work well on Windows. It's just a chicken and egg problem: few people use it there, so it's not up to par, so few people use it there because it's not up to par.
- EricBurnett 7y ago
  The differences between bazel and BuildXL are less about platform and more about philosophy. Bazel is about opinionated builds - declare everything, and you get fast correct builds. Hard to adopt, but super powerful. BuildXL is more about meeting codebases where they are - take an existing msbuild|cmake|dscript build with its under-specified dependencies, and figure out how to parallelize it safely. This delivers quick wins, but comes with real costs as well (notably: no caching; labour-intensive magic to guess an initial seed of deps, with a performance cliff when something changes). (Disclaimer: Googler, work closely with the bazel team. I like bazel; I like BuildXL for expanding the space of build ideas, but have long-term concerns it settles in a local maxima that'll be hard to get out of).

txtsd 7y ago

Is meson not good?

adgasf 7y ago
Meson does not have the same feature set as Bazel. Most notably sandboxing and remote execution.
- qznc 7y ago
  Other build systems (CMake, Meson, etc) should be able to gain the sandboxing feature. The remote execution feels more fundamental to me and not so easy to replicate.

adgasf 7y ago

Before it comes up: Bazel ships with its own copy of the JVM so you do not need to install Java to use it.

qhwudbebd 7y ago
...if you're happy running random third party binaries instead of building from source.
- adgasf 7y ago
  Do you review all of the source-code that you run? If you trust the source-code due to its origin, why not trust a binary?
  - qhwudbebd 7y ago
    It doesn't matter whether I trust it or not: these binaries are typically linked against a different C library, assume a different filesystem layout and dynamic linker location, and are often only available for x86-64 in any case. I build binaries locally to get something that's marginally more likely to work on my system, rather than because I believe I could somehow catch nasties buried in the source code even if I read every line. Too many years of seeing the ingenuity of IOCCC entries has taught me I won't succeed at that. :)
account42 7y ago
Is this supposed to be a good thing?
- PedroBatista 7y ago
  Probably better than Nodejs Gulp Yo man ugabuga npm here's 3.4 million files for you to download or Python is it 2.7 or 3.1? Pip too and Ansible let's make that terminal a disco night of text! or some Go "Works great for my static file blog" build tool.
  - 7y ago

rjsw 7y ago

Firefox will currently build on more platforms than Bazel does. Getting fed up of the endless list of new build systems.

qhwudbebd 7y ago
I agree, and inflicting Bazel's horrible Java dependency on Firefox would be a tragedy.
- mda 7y ago
  How is it different than depending on e.g. Python?
  - rjsw 7y ago
    Python doesn't currently require a working Python binary to build itself. Both Bazel and Java do.
    - new_realist 7y ago
      A C compiler, which Python needs, is arguably as complex as either a Python or Java runtime.
      - jcranmer 7y ago
        A C compiler is a functional requirement of any operating system these days.
        pjmlp 7y ago
        You won't go very far with it on Android, ChromeOS, iOS, iPadOS, mbed, Arduino, console OSes,...
        creatornator 7y ago

hartror 7y ago

TIL Thunderbird is still a thing.

pja 7y ago
It’s still one of the best IMAP clients IMO.
- CrLf 7y ago
  That says a lot about the state of IMAP clients.
- gspr 7y ago
  mbsync/isync changed my email life. http://isync.sourceforge.net/mbsync.html http://isync.sourceforge.net/mbsync.html
  - NilsIRL 7y ago
    I what way? What do you use it with?

boris 7y ago

I wonder if they considered build2[1]? It seems to tick most of their boxes. In particular, it has first-class Windows support and, in the upcoming release, even native Clang-targeting-MSVC support (not the clang-cl wrapper)[2]. While some of the features they are looking for (like faster build times which I assume means distributed compilation and caching) are not there yet, we are working on them (and would welcome help)[3]. [1] https://build2.org https://build2.org [2] https://stage.build2.org/?builds=&cf=windows_10-clang* https://stage.build2.org/?builds=&cf=windows_10-clang* [3] https://github.com/build2/README/blob/master/roadmap.md https://github.com/build2/README/blob/master/roadmap.md

dochtman 7y ago
One of the important parts in the post seems to be that it is not just a C++ build system, there are many other tasks executed through the build system. In that sense, it seems like build2 isn't a great fit.
- boris 7y ago
  On the contrary, build2 is a language-agnostic, general-purpose build system. There is the bash module[1], for example. And now that we have support for dynamically buildable/loadable build system modules, we can develop support for more languages, code generators, etc., without bundling it with the build system core. I myself am itching to take a stab at Rust. Hopefully will find time after the release. [1] https://build2.org/build2/doc/build2-build-system-manual.xhtml#module-bash https://build2.org/build2/doc/build2-build-system-manual.xht...
  - qznc 7y ago
    The frontpage starts with "build2 is an open source (MIT), cross-platform build toolchain for developing and packaging C and C++ code." My impression about the modules you linked to: A secondary feature because many C/C++ project need some additional scripting.
    - boris 7y ago

steveklabnik 7y ago

> The typical solution is to become more conservative in how you build things such that you can be sure that Y is always built before X…but typically by making the dependency implicit by, say, ordering the build commands Just So, and not by actually making the dependency explicit to make itself. Maybe specifying the explicit dependency is rather difficult, or maybe somebody just wants to make things work. After several rounds of these kind of fixes, you wind up with Makefiles that are (probably) correct, but probably not as fast as it could be, because you’ve likely serialized build steps that could have been executed in parallel. And untangling such systems to the point that you can properly parallelize things and that you don’t regress correctness can be…challenging. We had a particularly "fun" instance of this sort of Makefile weirdness back when we used Make for Rust... https://github.com/rust-lang/rust/blob/efec34a95a76212b2324d98f3f6d94a1397c2544/mk/prepare.mk#L44-L51 https://github.com/rust-lang/rust/blob/efec34a95a76212b2324d... (Discussed on HN: https://news.ycombinator.com/item?id=13655081 https://news.ycombinator.com/item?id=13655081)

ndesaulniers 7y ago

Of all the build systems I've used, Bazel/Blaze has sucked the least.

munificent 7y ago

Hermetic declarative build systems like Bazel feel exactly like sound statically typed languages to me. It's really hard to get all of your dependencies precisely specified declaratively. The urge to hack in a little imperative code or jam in a build step that you just "know" needs to happen in a certain order is always there and it can be very difficult to satisfy the taskmaster that is the build system. When you have weird cases like a script that produces a variable set of output files based on the content of some input file, it's tough to get that working in the build system. But... if you climb that cliff, there are many rewards at the top. Like the article notes, you can reuse intermediate build artifacts between developers (because it's not possible for those artifacts to accidentally depend on developer-specific state), you can automatically parallelize (because the build system can tell declaratively which tasks do no depend on each other) and you can ask it interesting queries about the dependency graph. That sounds an awful lot like the runtime safety, performance, and static analysis static types give you. The challenge is that for small programs, dynamic types are just easier to get going with. And every build script starts pretty small. So you have a plethora of simple, hackable build systems like make that work fine at first but then get more and more painful as the program grows. With software, there is at least some customer-visible pain that can enable you to justify a rewrite in a statically-typed language. But build infrastructure is always a hidden cost so it's really hard to get the resources to migrate to a different build language. Maybe the lesson here is to bite the bullet and start with a hermetic build system at first.

adgasf 7y ago
Except the Bazel syntax is really good. Compiling a few C++ files into a binary is just: cc_binary( name = "app", srcs = glob([ "*.hpp", "*.cpp" ]), ) That's more concise than most build systems AND it's cross-platform.
- joshuamorton 7y ago
  Although I'd strongly suggest not doing that bazel works better if you have 1 library per header (or source file for non c langs). It helps to have tools to manage deps for your though.

jayd16 7y ago

I'm trying to wrap my head around whether Bazel is worth looking into for the kind of work loads I typically work in. There doesn't seem like that much benefit if you have single language code base and especially if you already have a single build. I suppose I see the benefit of this setup if you keep many libraries and want to pin to the latest instead of, an explicit lib version. If you're in a multi-repo system you're probably not doing this but I can see how Bazel could actually make this possible. Can someone shed some light on the CI story? Can Bazel do code deployment as well? Is that a smart thing to do with Bazel?

lima 7y ago
Bazel is worth looking into as soon as you have dependency graphs that your language's native build system can't deal with efficiently, or you have multiple languages that have dependencies on each other's artifacts. A pure Go codebase? Not worth it. Personally, I use Bazel even for small projects as soon as they involve things like generated code or gRPC. Yes, there are multiple rulesets for deployments, like rules_k8s[1] and rules_docker[2]. Of course, you can easily build your own custom deployment pipeline. [1]: https://github.com/bazelbuild/rules_k8s https://github.com/bazelbuild/rules_k8s [2]: https://github.com/bazelbuild/rules_docker https://github.com/bazelbuild/rules_docker

xvilka 7y ago

The problem of Bazel is the JVM requirement. Why not to use something native and more lightweight?

pjmlp 7y ago
You can compile Java to native code. And in the case you don't want to, it is hardly any different from shipping a bunch of .so/.dll with the application.