5 ms·

Anyone got any stories about migrating big, crufty, multiplatform cmake C++ projects to Bazel? "It sucked but was worth it" or "it was surprisingly easy" or "LO

Anyone got any stories about migrating big, crufty, multiplatform cmake C++ projects to Bazel? "It sucked but was worth it" or "it was surprisingly easy" or "LOL don't even try?"

rufius 4y ago
I’ve not got specifically what you’re describing but I’ve encountered Bazel twice in my career - both in very large Java/Scala/Go code bases (distinct repo for Go but a lot of code). Bazel is extremely underwhelming. I’ve worked with crusty ancient systems that built huge systems and Bazel is just the most clown shoes build tool in comparison. Something didn’t work? Try typing the same command repeatedly and hope that this time it sticks. Multiple commands to achieve a seemingly straightforward task? Why isn’t there a single one that will get us there. FWIW - I suspect like most tools that Google open sources, the tool makes far more sense in the context of Google’s systems and architecture. If you’re adhering to that, it’s probably coherent. I’d never choose it willingly though.
- rfoo 4y ago
  > Multiple commands to achieve a seemingly straightforward task? Why isn’t there a single one that will get us there. Could you elaborate on this? I seldomly need commands beside `bazel build`, `bazel test` and `bazel run` and am curious to your story.
  - rufius 4y ago
    Full admission - I don’t own or have responsibility for how the build system I work with is setup. I’m on parental leave so details are fuzzy but roughly: - run Bazel build - expect dependencies to be built - they weren’t built - run Bazel build again - something else decides to be built - repeat ad nauseum This is within the same project directory, etc. It’s entirely possible that the project is setup in some pathological way but I’ve encountered this enough times in two companies that it’s stuck in my head.
    - rfoo 4y ago
      Whoa, that's tough. Sounds like you have indeed hit a case where > the tool makes far more sense in the context of Google’s systems and architecture Bazel should not behave like what you described (at least in my experience) for in-repo sources and build rules. Except that the world does not work in this way, so they added a duct tape called "using workspace rules to fetch and potentially build external dependencies", which is as fragile as a ./build.sh pulling in all your dependencies. And Google? They did vendor everything they use at //third_party in their monorepo, so (╯‵□′)╯︵┻━┻
    - mattnewton 4y ago
      This sounds highly broken somehow. Bazel’s raison d'etre is rebuilding only the necessary dependencies. Maybe this is a case of someone trying to force-fit a process that doesn’t look like bazel’s opinions with some custom rules that break it at the core? Pretty much all my bazel/blaze experience has been at google or in other side projects built from the start with bazel, but I have never encountered anything like that. The only complaints I’ve had is that building python slows the scripting loop, and that for side projects without google’s build infrastructure the rebuild-the-world-at-head-from-source method can get very expensive. But this sounds really unfortunate and nothing like the tool I’ve used :/
    - Difwif 4y ago
      That sounds like some one hacked up an existing project into Bazel without resolving the opinionated differences. I've worked on Bazel projects at multiple companies and I've seen things go off the rails like that a couple times before someone that actually understands the tool rewrites the problematic build process. It's usually some nasty stuff where someone tried to work outside of Bazel because they didn't understand it and created a bunch of impedance mismatches. New people doing this instead of the "right way" is the most valid criticism of Bazel IMO. Sounds like you're working in the worst of both worlds right now.
rfoo 4y ago
Having gone through this twice, I'd say it is not that difficult, but could take reasonably huge effort. On par with turning your crufty CMakeLists to be so called "Modern CMake" (whatever that means). But, why? Bazel is very opinionated on how you layout your C++ source code in the repository, and it's something which could not be retrofitted easily.
elteto 4y ago
It delivers on its premise of always correct, incremental builds but it is extremely opinionated. I don’t blame it for that, maybe having truly hermetic and reproducible builds requires that level of structure. It is almost magical changing a linker flag in some bazel config and see it _only_ relink affected targets. If you need to do cross-compilation then I feel like it is extremely overengineered with the whole platform/toolchain concepts, and after _years_ the docs are still incredibly lacking on this aspect. I almost prefer the previous approach with the semi-documented protobuf as JSON crosstool file. If you need the safety guarantees or the reproducibility there’s no other build system out there. If you don’t then you will be inclined to hate it because you are not extracting value of it.
- colatkinson 4y ago
  Yeah the cross-compilation thing is definitely a rough spot. I have one project that's able to work around it via extensive hacks with macros, but at some point I'll need to do it "the right way." Honestly if the docs had a canonical example of e.g. using unix_cc_toolchain_config (example: [0]) + Bootlin to compile for aarch64, it'd probably go a long way to making things understandable. Because say what you will about the old CROSSTOOL approach, at least there was a nice tutorial for it. [0] https://github.com/grailbio/bazel-toolchain/blob/f14a8a5de8f7e98a011a52163d4855572c07a1a3/toolchain/cc_toolchain_config.bzl#L336 https://github.com/grailbio/bazel-toolchain/blob/f14a8a5de8f...
  - kldx 4y ago
    Have you come across https://github.com/aspect-build/gcc-toolchain https://github.com/aspect-build/gcc-toolchain? I use it as a starting point for my toolchains.
    - colatkinson 4y ago
      Woah I hadn't. Looks super useful, thanks!
  - elteto 4y ago
    My normal workflow to bootstrap cross-compilation with bazel is to create a dummy project with some dummy C/C++ file and build it. Then go into whatever bazel-X internal folder and extract the autogenerated bzl for the local system’s compiler. Then update it with my toolchain and strip it down (I hate the “features” feature) until it is somewhat understandable. This is a _terrible_ DX.
lolpython 4y ago
I migrated a mid-size polyglot project from Makefiles to Bazel and C++ was a large component of the project. Some obstacles: 1. Building with QT5 MOC & UI files. There is a great library[0] for it but it has hardcoded paths to the QT binaries and header files assuming a system-wide installation. I had to patch the rule to point to our QT location. Then it worked fine. 2. There is no rule to build a fully static library[1]. Since we were shipping a static library to clients via our Makefile system, that was somewhat annoying. 3. We were using system links like `$PROJECT_ROOT/links/GCC/vX.Y.Z/ -> /opt/gcc/...` to point to all the build tools, but these didn't work in Bazel I think because it required absolute paths for any binaries it calls. We ended up putting them in a .bazelrc but we would need a different one for Windows and Linux. 4. Not good integration with IDEs 5. (edit) The Bazel toolchain system is confusing and I couldn't understand it after reading all its docs Ultimately we did not keep using Bazel because we were building Python binaries and py_binary was too slow on Windows. And we didn't have enough time to write a PyInstaller rule. [0]: https://github.com/justbuchanan/bazel_rules_qt https://github.com/justbuchanan/bazel_rules_qt [1]: https://github.com/bazelbuild/bazel/issues/1920 https://github.com/bazelbuild/bazel/issues/1920
- klodolph 4y ago
  Regarding #3, my approach to solve problems like this is to make a custom repository rule which creates the desired symlinks. The repository rule can invoke external programs or examine the environment as necessary to figure out how this should be created. Basically, you create a repository rule that symlinks your $PROJECT_ROOT/links/GCC/vX.Y.Z/ to $repo/... somewhere, and then generates a BUILD file for the repository. Writing your own repository rule is not especially difficult and they do have a lot of power not available to ordinary rules. This is the API that you can use from within repository rules--you can see that it lets you run arbitrary programs, create files and symlinks, download files, etc. https://bazel.build/rules/lib/repository_ctx https://bazel.build/rules/lib/repository_ctx
- rfoo 4y ago
  #2: To be fair it is reasonably easy to make a cc_static_library_binary-ish rule which merges all transitive .a-s (just generate an ar script and call archiver). But I have to admit that I spent non-trivial time on maintaining our "CROSSTOOL in skylark" (forgot the term) for 20+ target platforms before and it helped a lot on understanding the (still incomplete) C++ sandwich.
klodolph 4y ago
I have some adjacent experience. Not exactly with that, but I have migrated large C++ build systems, migrated systems to Bazel, etc. I've also written a bunch of Bazel build scripts for various open-source C++ libraries so I can better integrate them into projects that use Bazel. Bazel is opinionated. The tradeoff here is that if you can make your project match Bazel's opinions, you get a very good experience--but you can have a bad experience if you disagree with Bazel. If you have Bazel experience, you can look at a project and get a quick sense of the distance between how the project is built and how Bazel "wants" to build the project. The payoff is that once you get your Bazel build system, everything seems a lot more trustworthy. No more "make clean". When you run a Bazel command, it just gives you the correct output, very fast, without worrying about what state your build tree is in. You can make any change to your build scripts and just "bazel build" and get the correct result immediately, as long as you aren't trying to bypass how Bazel works. I never have to do anything like run "make" twice. This is something I've never gotten with systems like Make or CMake, which put more of the onus on individual developers to get things correct. So I can kind of shut off my brain when using Bazel. Depending on the particulars of your project, the most straightforward migration path to Bazel will not be obvious. You may need to make certain choices about how much you modify your project to fit Bazel's expectations, versus how much you adapt Bazel to fit your existing project. One example is include paths... do you modify all of your '#include' directives to match how Bazel expects you to write them? Or do you adapt Bazel to do things your way? The difficulty and payoff is highly variable. My experience has generally been positive, but I also have a lot of familiarity with Bazel. It's easy enough to find an example of a project where I'd just never bother migrating to Bazel, or to find examples of projects (even large ones) where migrating is super easy.
eklitzke 4y ago
The canonical build system for LLVM is CMake, but you can also build LLVM with Bazel now. That might give you a good idea of how to do it. The Bazel support code is in https://github.com/llvm/llvm-project/tree/main/utils/bazel https://github.com/llvm/llvm-project/tree/main/utils/bazel

Anyone got any stories about migrating big, crufty, multiplatform cmake C++ projects to Bazel? "It sucked but was worth it" or "it was surprisingly easy" or "LOL don't even try?"

rufius 4y ago

I’ve not got specifically what you’re describing but I’ve encountered Bazel twice in my career - both in very large Java/Scala/Go code bases (distinct repo for Go but a lot of code). Bazel is extremely underwhelming. I’ve worked with crusty ancient systems that built huge systems and Bazel is just the most clown shoes build tool in comparison. Something didn’t work? Try typing the same command repeatedly and hope that this time it sticks. Multiple commands to achieve a seemingly straightforward task? Why isn’t there a single one that will get us there. FWIW - I suspect like most tools that Google open sources, the tool makes far more sense in the context of Google’s systems and architecture. If you’re adhering to that, it’s probably coherent. I’d never choose it willingly though.

rfoo 4y ago
> Multiple commands to achieve a seemingly straightforward task? Why isn’t there a single one that will get us there. Could you elaborate on this? I seldomly need commands beside `bazel build`, `bazel test` and `bazel run` and am curious to your story.
- rufius 4y ago
  Full admission - I don’t own or have responsibility for how the build system I work with is setup. I’m on parental leave so details are fuzzy but roughly: - run Bazel build - expect dependencies to be built - they weren’t built - run Bazel build again - something else decides to be built - repeat ad nauseum This is within the same project directory, etc. It’s entirely possible that the project is setup in some pathological way but I’ve encountered this enough times in two companies that it’s stuck in my head.
  - rfoo 4y ago
    Whoa, that's tough. Sounds like you have indeed hit a case where > the tool makes far more sense in the context of Google’s systems and architecture Bazel should not behave like what you described (at least in my experience) for in-repo sources and build rules. Except that the world does not work in this way, so they added a duct tape called "using workspace rules to fetch and potentially build external dependencies", which is as fragile as a ./build.sh pulling in all your dependencies. And Google? They did vendor everything they use at //third_party in their monorepo, so (╯‵□′)╯︵┻━┻
  - mattnewton 4y ago
    This sounds highly broken somehow. Bazel’s raison d'etre is rebuilding only the necessary dependencies. Maybe this is a case of someone trying to force-fit a process that doesn’t look like bazel’s opinions with some custom rules that break it at the core? Pretty much all my bazel/blaze experience has been at google or in other side projects built from the start with bazel, but I have never encountered anything like that. The only complaints I’ve had is that building python slows the scripting loop, and that for side projects without google’s build infrastructure the rebuild-the-world-at-head-from-source method can get very expensive. But this sounds really unfortunate and nothing like the tool I’ve used :/
  - Difwif 4y ago
    That sounds like some one hacked up an existing project into Bazel without resolving the opinionated differences. I've worked on Bazel projects at multiple companies and I've seen things go off the rails like that a couple times before someone that actually understands the tool rewrites the problematic build process. It's usually some nasty stuff where someone tried to work outside of Bazel because they didn't understand it and created a bunch of impedance mismatches. New people doing this instead of the "right way" is the most valid criticism of Bazel IMO. Sounds like you're working in the worst of both worlds right now.

rfoo 4y ago

Having gone through this twice, I'd say it is not that difficult, but could take reasonably huge effort. On par with turning your crufty CMakeLists to be so called "Modern CMake" (whatever that means). But, why? Bazel is very opinionated on how you layout your C++ source code in the repository, and it's something which could not be retrofitted easily.

elteto 4y ago

It delivers on its premise of always correct, incremental builds but it is extremely opinionated. I don’t blame it for that, maybe having truly hermetic and reproducible builds requires that level of structure. It is almost magical changing a linker flag in some bazel config and see it _only_ relink affected targets. If you need to do cross-compilation then I feel like it is extremely overengineered with the whole platform/toolchain concepts, and after _years_ the docs are still incredibly lacking on this aspect. I almost prefer the previous approach with the semi-documented protobuf as JSON crosstool file. If you need the safety guarantees or the reproducibility there’s no other build system out there. If you don’t then you will be inclined to hate it because you are not extracting value of it.

colatkinson 4y ago
Yeah the cross-compilation thing is definitely a rough spot. I have one project that's able to work around it via extensive hacks with macros, but at some point I'll need to do it "the right way." Honestly if the docs had a canonical example of e.g. using unix_cc_toolchain_config (example: [0]) + Bootlin to compile for aarch64, it'd probably go a long way to making things understandable. Because say what you will about the old CROSSTOOL approach, at least there was a nice tutorial for it. [0] https://github.com/grailbio/bazel-toolchain/blob/f14a8a5de8f7e98a011a52163d4855572c07a1a3/toolchain/cc_toolchain_config.bzl#L336 https://github.com/grailbio/bazel-toolchain/blob/f14a8a5de8f...
- kldx 4y ago
  Have you come across https://github.com/aspect-build/gcc-toolchain https://github.com/aspect-build/gcc-toolchain? I use it as a starting point for my toolchains.
  - colatkinson 4y ago
    Woah I hadn't. Looks super useful, thanks!
- elteto 4y ago
  My normal workflow to bootstrap cross-compilation with bazel is to create a dummy project with some dummy C/C++ file and build it. Then go into whatever bazel-X internal folder and extract the autogenerated bzl for the local system’s compiler. Then update it with my toolchain and strip it down (I hate the “features” feature) until it is somewhat understandable. This is a _terrible_ DX.

lolpython 4y ago

I migrated a mid-size polyglot project from Makefiles to Bazel and C++ was a large component of the project. Some obstacles: 1. Building with QT5 MOC & UI files. There is a great library[0] for it but it has hardcoded paths to the QT binaries and header files assuming a system-wide installation. I had to patch the rule to point to our QT location. Then it worked fine. 2. There is no rule to build a fully static library[1]. Since we were shipping a static library to clients via our Makefile system, that was somewhat annoying. 3. We were using system links like `$PROJECT_ROOT/links/GCC/vX.Y.Z/ -> /opt/gcc/...` to point to all the build tools, but these didn't work in Bazel I think because it required absolute paths for any binaries it calls. We ended up putting them in a .bazelrc but we would need a different one for Windows and Linux. 4. Not good integration with IDEs 5. (edit) The Bazel toolchain system is confusing and I couldn't understand it after reading all its docs Ultimately we did not keep using Bazel because we were building Python binaries and py_binary was too slow on Windows. And we didn't have enough time to write a PyInstaller rule. [0]: https://github.com/justbuchanan/bazel_rules_qt https://github.com/justbuchanan/bazel_rules_qt [1]: https://github.com/bazelbuild/bazel/issues/1920 https://github.com/bazelbuild/bazel/issues/1920

klodolph 4y ago
Regarding #3, my approach to solve problems like this is to make a custom repository rule which creates the desired symlinks. The repository rule can invoke external programs or examine the environment as necessary to figure out how this should be created. Basically, you create a repository rule that symlinks your $PROJECT_ROOT/links/GCC/vX.Y.Z/ to $repo/... somewhere, and then generates a BUILD file for the repository. Writing your own repository rule is not especially difficult and they do have a lot of power not available to ordinary rules. This is the API that you can use from within repository rules--you can see that it lets you run arbitrary programs, create files and symlinks, download files, etc. https://bazel.build/rules/lib/repository_ctx https://bazel.build/rules/lib/repository_ctx
rfoo 4y ago
#2: To be fair it is reasonably easy to make a cc_static_library_binary-ish rule which merges all transitive .a-s (just generate an ar script and call archiver). But I have to admit that I spent non-trivial time on maintaining our "CROSSTOOL in skylark" (forgot the term) for 20+ target platforms before and it helped a lot on understanding the (still incomplete) C++ sandwich.

klodolph 4y ago

I have some adjacent experience. Not exactly with that, but I have migrated large C++ build systems, migrated systems to Bazel, etc. I've also written a bunch of Bazel build scripts for various open-source C++ libraries so I can better integrate them into projects that use Bazel. Bazel is opinionated. The tradeoff here is that if you can make your project match Bazel's opinions, you get a very good experience--but you can have a bad experience if you disagree with Bazel. If you have Bazel experience, you can look at a project and get a quick sense of the distance between how the project is built and how Bazel "wants" to build the project. The payoff is that once you get your Bazel build system, everything seems a lot more trustworthy. No more "make clean". When you run a Bazel command, it just gives you the correct output, very fast, without worrying about what state your build tree is in. You can make any change to your build scripts and just "bazel build" and get the correct result immediately, as long as you aren't trying to bypass how Bazel works. I never have to do anything like run "make" twice. This is something I've never gotten with systems like Make or CMake, which put more of the onus on individual developers to get things correct. So I can kind of shut off my brain when using Bazel. Depending on the particulars of your project, the most straightforward migration path to Bazel will not be obvious. You may need to make certain choices about how much you modify your project to fit Bazel's expectations, versus how much you adapt Bazel to fit your existing project. One example is include paths... do you modify all of your '#include' directives to match how Bazel expects you to write them? Or do you adapt Bazel to do things your way? The difficulty and payoff is highly variable. My experience has generally been positive, but I also have a lot of familiarity with Bazel. It's easy enough to find an example of a project where I'd just never bother migrating to Bazel, or to find examples of projects (even large ones) where migrating is super easy.

eklitzke 4y ago

The canonical build system for LLVM is CMake, but you can also build LLVM with Bazel now. That might give you a good idea of how to do it. The Bazel support code is in https://github.com/llvm/llvm-project/tree/main/utils/bazel https://github.com/llvm/llvm-project/tree/main/utils/bazel