4 ms·

One of the major time savers of bazel is Remote Build Execution (RBE), which allows you to build modules in parallel in the cloud. So if you have 1000 CPUs, yo

One of the major time savers of bazel is Remote Build Execution (RBE), which allows you to build modules in parallel in the cloud. So if you have 1000 CPUs, you can really just have a client do `bazel build -j 1000 //...` and you can get a huge speed-up. Remote (and local) builds all happen in a sandbox, so you don't have to worry about e.g. preparing a docker image with the worker / build slave environment. (You do, however, have to register your dependencies with Bazel, which can be a hard at first). To add to this, bazel also has a remote global cache which can benefit large teams. For fairly large C++ codebases, RBE is really a competitive advantage. I've seen RBE cut down iteration time by an order of magnitude. I love CMake, and CMake can get you plenty of parallelism, but CMake doesn't really provide a tool for building several CMake sub-projects in parallel, and bazel handles this really well. Sadly Bazel RBE is still primarily a Google Cloud product. Also, GCE is slow to work on supporting auto-scale, so you have to pay for unused workers. (Like most products in Google Cloud, Google is ripping you off with alpha-quality stuff). There's some very rough open source RBE stuff on Github that you can run yourself, but nothing really production-grade yet. gg ( https://github.com/StanfordSNR/gg https://github.com/StanfordSNR/gg ) is a promising-looking alternative. It's research code, but it might be the community's best hope for a non-Google alternative (that e.g. supports AWS Lambda for parallelism). Bazel is great, but without independence (e.g. what Kubernetes achieved) it's difficult to see bazel as dependable as make or CMake long term.

9nGQluzmnq3M 7y ago
> GCE is slow to work on supporting auto-scale, so you have to pay for unused workers. Can you expand on what you mean here? GCE has had autoscaling for years and it's quite configurable, I don't see why you would need to keep unused workers around unless your build load is extremely spiky.
- choppaface 7y ago
  Not auto-scaling VM compute instances, auto-scaling the Google Cloud RBE service (I think it's called Cloud Build right now?). If you have a worker pool of size 1,000, you have to pay for that even if your dev team only uses it at full capacity 8-10hrs per day. Or maybe the Google Cloud sales people working with my company are misinformed? IDK, but my experience with TPUs, GPUs, AI Engine, Dataproc, BigQuery, etc... every GCloud feature I've used (besides vanilla Compute VMs) have gotchas and/or time sinks that are costly unless you have a load of free credits. For AI Engine, it took like 6 months to upgrade their internal Kubernetes cluster; in the meantime, crashed jobs would burn an extra 15 minutes of GPU dollars. There's a lot of good stuff in Google Cloud, but be sure to get Google to pay you to wade through the construction with free credits.
  - merb 7y ago
    in Google Cloud Build you pay per minute, not per instance.
  - shusson 7y ago
    > Cloud Build If you're talking about [1], it is billed in minutes. You can have 10 concurrent workers. Custom Workers is in alpha. [1] https://cloud.google.com/cloud-build/ https://cloud.google.com/cloud-build/
  - philwo 7y ago
    Cloud Build and Cloud RBE are two different products. The former is Google Cloud's CI service (https://cloud.google.com/cloud-build/ https://cloud.google.com/cloud-build/), while the latter is a hosted backend for Bazel's remote execution feature (some infos are here: https://groups.google.com/forum/#!forum/rbe-alpha-customers https://groups.google.com/forum/#!forum/rbe-alpha-customers). They are not related. Disclaimer: I work on the Bazel team at Google.
Nullabillity 7y ago
Nix also supports using any SSH server with Nix installed as a remote builder[0]. Performance gains aside, it's also very useful for building for other platforms where you don't have a cross-toolchain set up. For example, I've had some colleagues use nix-docker to build packages for Linux from their Macs, even if the packages weren't cross-aware[1]. [0]: https://nixos.wiki/wiki/Distributed_build https://nixos.wiki/wiki/Distributed_build [1]: https://github.com/LnL7/nix-docker#running-as-a-remote-builder https://github.com/LnL7/nix-docker#running-as-a-remote-build...
arberx 7y ago
Check out recc: https://gitlab.com/bloomberg/recc https://gitlab.com/bloomberg/recc It's a remote execution client, that you can use with your current cmake project, along with buildgrid: https://gitlab.com/BuildGrid/buildgrid https://gitlab.com/BuildGrid/buildgrid which is a basically a FOSS RBE. Along with workers: https://gitlab.com/BuildGrid/buildbox/buildbox-worker https://gitlab.com/BuildGrid/buildbox/buildbox-worker With these FOSS alternatives, you can tailor/scale your builds yourself without relying on RBE. All of these utilize the Remote Execution protocol, so you can plug and play with other clients/workers/servers that use the protocol.
mikepurvis 7y ago
It isn't a native CMake solution, but the ROS (Robot Operating System) community has been dealing with the "build many CMake/python/other projects in parallel" problem for a long time, and has produced several tools and which address this. The most recent effort is called colcon, which you can find here: https://colcon.readthedocs.io/en/released/ https://colcon.readthedocs.io/en/released/ In the past (for example with catkin_tools, colcon's predecessor), the biggest barrier to use with conventional CMake packages has always been that they don't export dependency information in an easily machine-consumable way, so any regular CMake package brought into the workspace needed to have a package.xml file patched into it. Colcon still has the ability to bring in extra metadata for things like ensuring environment variables and so on are set up correctly in the final workspace, but it's gotten a lot better at parsing/interpreting the native expressions of dependency and figuring out what to do. See for example how you build the multiple projects which make up the Gazebo simulator with colcon: https://colcon.readthedocs.io/en/released/user/quick-start.html#build-gazebo-and-the-ignition-packages https://colcon.readthedocs.io/en/released/user/quick-start.h... Of course, colcon also supports adding python/setuptools repos, and includes experimental support for bazel, gradle, cargo, and other buildsystems which can do an "install" step into a reasonably sane FHS tree. So in that way it's kind of like a cross-language supercharged virtualenv— it gives you the benefits of building multiple pieces of a system from source, semi-isolated, without the hassle of dealing with containers or VMs, where you also have to have all your tools in the isolated environment.
shantly 7y ago
Used to be able to do something similar with Gentoo back in, like, 2002, I think. Distribute build jobs to multiple remote machines.
- jvolkman 7y ago
  https://github.com/icecc/icecream https://github.com/icecc/icecream
razeh 7y ago
For C++ we've had better luck, on our local cluster, with distcc than with bazel's remote execution, and we've found the the cache slowed things down. Has your experience all been in the google cloud?

9nGQluzmnq3M 7y ago

> GCE is slow to work on supporting auto-scale, so you have to pay for unused workers. Can you expand on what you mean here? GCE has had autoscaling for years and it's quite configurable, I don't see why you would need to keep unused workers around unless your build load is extremely spiky.

choppaface 7y ago
Not auto-scaling VM compute instances, auto-scaling the Google Cloud RBE service (I think it's called Cloud Build right now?). If you have a worker pool of size 1,000, you have to pay for that even if your dev team only uses it at full capacity 8-10hrs per day. Or maybe the Google Cloud sales people working with my company are misinformed? IDK, but my experience with TPUs, GPUs, AI Engine, Dataproc, BigQuery, etc... every GCloud feature I've used (besides vanilla Compute VMs) have gotchas and/or time sinks that are costly unless you have a load of free credits. For AI Engine, it took like 6 months to upgrade their internal Kubernetes cluster; in the meantime, crashed jobs would burn an extra 15 minutes of GPU dollars. There's a lot of good stuff in Google Cloud, but be sure to get Google to pay you to wade through the construction with free credits.
- merb 7y ago
  in Google Cloud Build you pay per minute, not per instance.
- shusson 7y ago
  > Cloud Build If you're talking about [1], it is billed in minutes. You can have 10 concurrent workers. Custom Workers is in alpha. [1] https://cloud.google.com/cloud-build/ https://cloud.google.com/cloud-build/
- philwo 7y ago
  Cloud Build and Cloud RBE are two different products. The former is Google Cloud's CI service (https://cloud.google.com/cloud-build/ https://cloud.google.com/cloud-build/), while the latter is a hosted backend for Bazel's remote execution feature (some infos are here: https://groups.google.com/forum/#!forum/rbe-alpha-customers https://groups.google.com/forum/#!forum/rbe-alpha-customers). They are not related. Disclaimer: I work on the Bazel team at Google.

Nullabillity 7y ago

Nix also supports using any SSH server with Nix installed as a remote builder[0]. Performance gains aside, it's also very useful for building for other platforms where you don't have a cross-toolchain set up. For example, I've had some colleagues use nix-docker to build packages for Linux from their Macs, even if the packages weren't cross-aware[1]. [0]: https://nixos.wiki/wiki/Distributed_build https://nixos.wiki/wiki/Distributed_build [1]: https://github.com/LnL7/nix-docker#running-as-a-remote-builder https://github.com/LnL7/nix-docker#running-as-a-remote-build...

arberx 7y ago

Check out recc: https://gitlab.com/bloomberg/recc https://gitlab.com/bloomberg/recc It's a remote execution client, that you can use with your current cmake project, along with buildgrid: https://gitlab.com/BuildGrid/buildgrid https://gitlab.com/BuildGrid/buildgrid which is a basically a FOSS RBE. Along with workers: https://gitlab.com/BuildGrid/buildbox/buildbox-worker https://gitlab.com/BuildGrid/buildbox/buildbox-worker With these FOSS alternatives, you can tailor/scale your builds yourself without relying on RBE. All of these utilize the Remote Execution protocol, so you can plug and play with other clients/workers/servers that use the protocol.

mikepurvis 7y ago

It isn't a native CMake solution, but the ROS (Robot Operating System) community has been dealing with the "build many CMake/python/other projects in parallel" problem for a long time, and has produced several tools and which address this. The most recent effort is called colcon, which you can find here: https://colcon.readthedocs.io/en/released/ https://colcon.readthedocs.io/en/released/ In the past (for example with catkin_tools, colcon's predecessor), the biggest barrier to use with conventional CMake packages has always been that they don't export dependency information in an easily machine-consumable way, so any regular CMake package brought into the workspace needed to have a package.xml file patched into it. Colcon still has the ability to bring in extra metadata for things like ensuring environment variables and so on are set up correctly in the final workspace, but it's gotten a lot better at parsing/interpreting the native expressions of dependency and figuring out what to do. See for example how you build the multiple projects which make up the Gazebo simulator with colcon: https://colcon.readthedocs.io/en/released/user/quick-start.html#build-gazebo-and-the-ignition-packages https://colcon.readthedocs.io/en/released/user/quick-start.h... Of course, colcon also supports adding python/setuptools repos, and includes experimental support for bazel, gradle, cargo, and other buildsystems which can do an "install" step into a reasonably sane FHS tree. So in that way it's kind of like a cross-language supercharged virtualenv— it gives you the benefits of building multiple pieces of a system from source, semi-isolated, without the hassle of dealing with containers or VMs, where you also have to have all your tools in the isolated environment.

shantly 7y ago

Used to be able to do something similar with Gentoo back in, like, 2002, I think. Distribute build jobs to multiple remote machines.

jvolkman 7y ago
https://github.com/icecc/icecream https://github.com/icecc/icecream

razeh 7y ago

For C++ we've had better luck, on our local cluster, with distcc than with bazel's remote execution, and we've found the the cache slowed things down. Has your experience all been in the google cloud?