5 ms·

I've been using Bazel for side projects these days, including small retro game projects for old game consoles. The entry price was high, but it works so well I

I've been using Bazel for side projects these days, including small retro game projects for old game consoles. The entry price was high, but it works so well I have a hard time imagining working without it. For retro game projects, the core of your game might be written in C or C++, which you want cross-compiled. That's easily within reach of stuff like makefiles. But then I start adding a bunch of custom tooling--I want to write tools that process sprites, audio, or 3D models. These days I tend to write those tools in Go. I'm also working with other people, and I want to be able to cross-compile these tools for Windows, even though I'm developing on Linux or macOS. My Bazel repository will download the ARM toolchain automatically and do everything it needs to build the target. I don't really need to set up my development environment at all--I just need to install Bazel and a C compiler, and Bazel will handle the rest. I don't need to install anything else on my system, and the C compiler is only needed because I'm using protocol buffers (Bazel downloads and compiles protoc automatically).

synergy20 4y ago
Makefile can do all that too, without java dependency and straightforward to read at any time later, what real extra benefit Bazel brings in here.
- ajconway 4y ago
  Bazel doesn't have any user-facing java dependency. > what real extra benefit Bazel brings in here Builds are hermetic by default, so unless the developer chooses to escape the sandbox, everything is guaranteed to build on other machines with no additional setup. (Also, I genuinely hate when I have to manually install build dependencies system-wide and pray that there will not be any conflicts. Having everything pinned to specific sha256 or git hashes by design is a breath of fresh air)
  - blueblob 4y ago
    I used a M1 Mac to try to build tensorflow and tensorflow-text and it is very untrue that everything is guaranteed to build on other machines with no additional setup. The parenthesized comment is funnier to me because I had to download a specific bazel version to build.
    - jvolkman 4y ago
      > The parenthesized comment is funnier to me because I had to download a specific bazel version to build. If you use bazelisk to provide your `bazel` command, it'll download the appropriate Bazel version for the repo you're trying to build. https://github.com/bazelbuild/bazelisk https://github.com/bazelbuild/bazelisk
      - dmitriid 4y ago
        So, more tools on top of your tools. And this couldn't be a part of bazel proper for the dame reason MS built a separate tool to discover paths for their tools: overengineering
    - ajconway 4y ago
      That is true, Bazel itself is still evolving, and there have been breaking changes between versions. Sometimes the required version number is placed in a .bazelversion file, which makes Bazelisk your top-level dependency. I'd expect Tensorflow to have some non-hermetic build actions, but if choosing a specific Bazel version was the only thing that was required to build it, that's awesome!
      - blueblob 4y ago
        It was definitely not the only thing that was required to build, but that may be as much Apple's fault as bazel's.
- kldx 4y ago
  Can make do incremental builds based on checksums instead of modification time? How about distributing the build across a cluster in a language agnostic way?
- jpollock 4y ago
  Bazel is more strict about what constitutes a dependency change. It doesn't use mtime on the file, it relies on the checksum. It also considers the command line flags to be part of the cache key. So, spurious changes (touching a file) will result in a cache hit, while hidden changes (changing an environment flag used by Make) are caught. This is particularly important if verifiable builds are needed for SoX compliance.
  - aaaaaaaaaaab 4y ago
    What if I use the __DATE__ macro?
    - jvolkman 4y ago
      You'll get "redacted" https://github.com/bazelbuild/bazel/blob/34ce6a23f5a2be58bb59661dd5fc9ab586ea1703/tools/cpp/unix_cc_configure.bzl#L628-L631 https://github.com/bazelbuild/bazel/blob/34ce6a23f5a2be58bb5...
    - jpollock 4y ago
      Yes, you can go out of your way to make your builds non-deterministic. That should break in Make as well. I don't know enough about bazel to answer your question definitively, but "stamping" is what you want to search for. https://bazel.build/docs/user-manual https://bazel.build/docs/user-manual
    - kmarc 4y ago
      The better question is, what if you change a comment? It checksums the preprocessed files, so neither `__DATE__`, no comment changes should affect your build times.
      - jpollock 4y ago
        I think __DATE__ without redaction pushes the dependency change 1 level down, to the .o file. The .a/.so/.exe would no longer match the inputs (the .o files), causing a re-link.
  - 4y ago
- IshKebab 4y ago
  The benefits that it brings is the whole reason it exists: precise dependency tracking. That means you can't have missing dependency links (very common and hard to debug in Make or CMake for example). You can know what outputs a change affects so you can only build/test a subset of the project in CI. Incremental builds become totally reliable. Etc.
- yata69420 4y ago
  The benefit is that it keeps a huge graph of all dependencies, so everything from the toolchain through your application can be built with hashing and caching. Sort of like make, but decentralized and based on hashed file contents. Once you have bazel, you can distribute the workload so that you could have thousands of machines each producing artifacts to be shared with other build machines. Then you can set up your dev machine to rely on those caches, so your local builds either use everything directly from cache or instruct a remote builder to produce the artifact for you. No matter what you change, because the dependencies are graphed precisely, you only need to rebuild a very tiny set of artifacts impacted by your change. Of course, this doesn't actually work in practice. Your builds probably aren't deterministic, so your graph of artifacts won't be either, causing lots of stuff to rebuild. Also, it may work great for something like Java that produces class files, but not provide any caching at all for ruby. Debugging and supporting it is a full time job for a team of engineers that dramatically outweighs the cost of keeping your projects sensibly sized. You might think that as the project gets larger, it's totally worth it to use bazel for that sweet caching. But in reality the graph construction and querying will become so bloated that just figuring out which targets need to rebuilt becomes a full time engineering effort that breaks constantly with all tooling upgrades. Also, the plugin ecosystem is just poor. Bazel is the perfect storm of computer scientists loving big graphs, Google exporting an open source project and then rebuilding it internally, and inexperienced engineers being sold on a tech as being obviously right because all the big players use it.
  - bobsomers 4y ago
    This completely mirrors my experience with Bazel as well. As long as, like Google, you have plenty of engineering effort to waste on nothing but maintaining your build system, sure why not. But if you actually have to get something done for your business to exist, it's a lot of unrelated work to keep the beast fed and happy.
- dietrichepp 4y ago
  > Makefile can do all that too, I guess it's theoretically possible, but why would you do this? This sounds like some kind of "I want to prove it can be done" project like running a web server on a Commodore 64, or making a kitchen knife out of cardboard. Yes, you could figure out a way to download and install dependencies using Make and Curl, and maybe you could build the same library for multiple targets using a bunch of variable expansions, possibly by invoking make multiple times, or including the same makefile multiple times. Make sucks; it sucks a lot; I'd be miserable. And then if you aren't very careful, you'll end up forgetting to declare some dependency, or some flag change will invalidate your build but Make won't do anything, or you'll make a change to your makefile and forget to clean, and then you spend another hour or day debugging some build that was made out of stale parts. I've done this before, which is why I avoid make for everything but the smallest projects. > without java dependency Bazel does not have a Java dependency. I think you might be assuming that because Bazel is written in Java, somehow that means that Java must be installed on your computer. This is not true. Bazel is a self-contained executable you can drop in /usr/local/bin. I really don't understand your perspective here at all. I've seen people defend make, but make is full of so many traps and gotchas--I wonder how someone could use make and then decide that these traps and gotchas are okay. The projects that successfully use it tend to be small projects, use generated makefiles (automake, cmake, etc), or tend to be a bit simpler. I use make myself, but as a rule of thumb, only for projects with a few files.
- jen20 4y ago
  > straightforward to read at any time later Any makefile doing anything remotely complex is definitely not readable later.
- baby 4y ago
  Honestly Makefiles that are more than 10 lines should not exist
robbintt 4y ago
Do you have anything OSS that I could try to graft?
- dietrichepp 4y ago
  I made a demo project that you can build with Bazel. I've tested it with Bazel 5. It should build on macOS and Linux with just devtools + Bazel 5. https://github.com/depp/bazel-gba-example https://github.com/depp/bazel-gba-example Cross-compiling toolchains are a pain to set up in Bazel. Once you get them working, it's nice.

synergy20 4y ago

Makefile can do all that too, without java dependency and straightforward to read at any time later, what real extra benefit Bazel brings in here.

ajconway 4y ago
Bazel doesn't have any user-facing java dependency. > what real extra benefit Bazel brings in here Builds are hermetic by default, so unless the developer chooses to escape the sandbox, everything is guaranteed to build on other machines with no additional setup. (Also, I genuinely hate when I have to manually install build dependencies system-wide and pray that there will not be any conflicts. Having everything pinned to specific sha256 or git hashes by design is a breath of fresh air)
- blueblob 4y ago
  I used a M1 Mac to try to build tensorflow and tensorflow-text and it is very untrue that everything is guaranteed to build on other machines with no additional setup. The parenthesized comment is funnier to me because I had to download a specific bazel version to build.
  - jvolkman 4y ago
    > The parenthesized comment is funnier to me because I had to download a specific bazel version to build. If you use bazelisk to provide your `bazel` command, it'll download the appropriate Bazel version for the repo you're trying to build. https://github.com/bazelbuild/bazelisk https://github.com/bazelbuild/bazelisk
    - dmitriid 4y ago
      So, more tools on top of your tools. And this couldn't be a part of bazel proper for the dame reason MS built a separate tool to discover paths for their tools: overengineering
  - ajconway 4y ago
    That is true, Bazel itself is still evolving, and there have been breaking changes between versions. Sometimes the required version number is placed in a .bazelversion file, which makes Bazelisk your top-level dependency. I'd expect Tensorflow to have some non-hermetic build actions, but if choosing a specific Bazel version was the only thing that was required to build it, that's awesome!
    - blueblob 4y ago
      It was definitely not the only thing that was required to build, but that may be as much Apple's fault as bazel's.
kldx 4y ago
Can make do incremental builds based on checksums instead of modification time? How about distributing the build across a cluster in a language agnostic way?
jpollock 4y ago
Bazel is more strict about what constitutes a dependency change. It doesn't use mtime on the file, it relies on the checksum. It also considers the command line flags to be part of the cache key. So, spurious changes (touching a file) will result in a cache hit, while hidden changes (changing an environment flag used by Make) are caught. This is particularly important if verifiable builds are needed for SoX compliance.
- aaaaaaaaaaab 4y ago
  What if I use the __DATE__ macro?
  - jvolkman 4y ago
    You'll get "redacted" https://github.com/bazelbuild/bazel/blob/34ce6a23f5a2be58bb59661dd5fc9ab586ea1703/tools/cpp/unix_cc_configure.bzl#L628-L631 https://github.com/bazelbuild/bazel/blob/34ce6a23f5a2be58bb5...
  - jpollock 4y ago
    Yes, you can go out of your way to make your builds non-deterministic. That should break in Make as well. I don't know enough about bazel to answer your question definitively, but "stamping" is what you want to search for. https://bazel.build/docs/user-manual https://bazel.build/docs/user-manual
  - kmarc 4y ago
    The better question is, what if you change a comment? It checksums the preprocessed files, so neither `__DATE__`, no comment changes should affect your build times.
    - jpollock 4y ago
      I think __DATE__ without redaction pushes the dependency change 1 level down, to the .o file. The .a/.so/.exe would no longer match the inputs (the .o files), causing a re-link.
- 4y ago
IshKebab 4y ago
The benefits that it brings is the whole reason it exists: precise dependency tracking. That means you can't have missing dependency links (very common and hard to debug in Make or CMake for example). You can know what outputs a change affects so you can only build/test a subset of the project in CI. Incremental builds become totally reliable. Etc.
yata69420 4y ago
The benefit is that it keeps a huge graph of all dependencies, so everything from the toolchain through your application can be built with hashing and caching. Sort of like make, but decentralized and based on hashed file contents. Once you have bazel, you can distribute the workload so that you could have thousands of machines each producing artifacts to be shared with other build machines. Then you can set up your dev machine to rely on those caches, so your local builds either use everything directly from cache or instruct a remote builder to produce the artifact for you. No matter what you change, because the dependencies are graphed precisely, you only need to rebuild a very tiny set of artifacts impacted by your change. Of course, this doesn't actually work in practice. Your builds probably aren't deterministic, so your graph of artifacts won't be either, causing lots of stuff to rebuild. Also, it may work great for something like Java that produces class files, but not provide any caching at all for ruby. Debugging and supporting it is a full time job for a team of engineers that dramatically outweighs the cost of keeping your projects sensibly sized. You might think that as the project gets larger, it's totally worth it to use bazel for that sweet caching. But in reality the graph construction and querying will become so bloated that just figuring out which targets need to rebuilt becomes a full time engineering effort that breaks constantly with all tooling upgrades. Also, the plugin ecosystem is just poor. Bazel is the perfect storm of computer scientists loving big graphs, Google exporting an open source project and then rebuilding it internally, and inexperienced engineers being sold on a tech as being obviously right because all the big players use it.
- bobsomers 4y ago
  This completely mirrors my experience with Bazel as well. As long as, like Google, you have plenty of engineering effort to waste on nothing but maintaining your build system, sure why not. But if you actually have to get something done for your business to exist, it's a lot of unrelated work to keep the beast fed and happy.
dietrichepp 4y ago
> Makefile can do all that too, I guess it's theoretically possible, but why would you do this? This sounds like some kind of "I want to prove it can be done" project like running a web server on a Commodore 64, or making a kitchen knife out of cardboard. Yes, you could figure out a way to download and install dependencies using Make and Curl, and maybe you could build the same library for multiple targets using a bunch of variable expansions, possibly by invoking make multiple times, or including the same makefile multiple times. Make sucks; it sucks a lot; I'd be miserable. And then if you aren't very careful, you'll end up forgetting to declare some dependency, or some flag change will invalidate your build but Make won't do anything, or you'll make a change to your makefile and forget to clean, and then you spend another hour or day debugging some build that was made out of stale parts. I've done this before, which is why I avoid make for everything but the smallest projects. > without java dependency Bazel does not have a Java dependency. I think you might be assuming that because Bazel is written in Java, somehow that means that Java must be installed on your computer. This is not true. Bazel is a self-contained executable you can drop in /usr/local/bin. I really don't understand your perspective here at all. I've seen people defend make, but make is full of so many traps and gotchas--I wonder how someone could use make and then decide that these traps and gotchas are okay. The projects that successfully use it tend to be small projects, use generated makefiles (automake, cmake, etc), or tend to be a bit simpler. I use make myself, but as a rule of thumb, only for projects with a few files.
jen20 4y ago
> straightforward to read at any time later Any makefile doing anything remotely complex is definitely not readable later.
baby 4y ago
Honestly Makefiles that are more than 10 lines should not exist

robbintt 4y ago

Do you have anything OSS that I could try to graft?

dietrichepp 4y ago
I made a demo project that you can build with Bazel. I've tested it with Bazel 5. It should build on macOS and Linux with just devtools + Bazel 5. https://github.com/depp/bazel-gba-example https://github.com/depp/bazel-gba-example Cross-compiling toolchains are a pain to set up in Bazel. Once you get them working, it's nice.