9 ms·

Scaling Rust Builds with Bazel

107 points by zdw 4y ago

shashashasha___ 4y ago
https://github.com/sagiegurari/cargo-make https://github.com/sagiegurari/cargo-make is also a good (and simple) solution for the requirements in the post
IceWreck 4y ago
If you want something similar thats written in Rust instead of Java, look at Buck2 https://github.com/facebook/buck2 https://github.com/facebook/buck2
- Alifatisk 4y ago
  The readme states the following > This project is not yet polished. We are continuing to develop it in the open, but don't expect it to be suitable for most people… So it’s not ready yet to be used for anything serious I assume.
- sluongng 4y ago
  Similar to EdenFS/Sapling VCS, buck2 is being developed as such a fast pace that it's hard to imagine there would be any OSS support for it. Together with buck2 are 3-5 other repos that are supposed to be built and used together with it. If you want stability, Bazel is still a much better choice with their LTS release program.
- hocuspocus 4y ago
  I would rather have a look at Pants since it's now developed for the needs of Toolchain's customers, and not following the roadmap of a big company with a pretty unique infra. Its engine was also rewritten from Java to Rust.
  - twunde 4y ago
    I cannot wait until Pants supports Rust as a target language (they were looking for rust consumers as of 3-6 months ago if I recall). For our python needs it was a real breath of fresh air. I'm now at a shop without it and need to deal with slow dependency installs and dependency conflicts again :'(
- baby 4y ago
  I was wondering the same. I’ve heard of Buck2 the other day and it seems much better. Anyone has tried both?
Animats 4y ago
[flagged]
mrkeen 4y ago
The joy of tooling. If there's one tool it's not a mature ecosystem, if there's two, there's no standard way to do things. "Cargo is not a build system" reminded me of "Cabal is not a package manager" https://ivanmiljenovic.wordpress.com/2010/03/15/repeat-after-me-cabal-is-not-a-package-manager/ https://ivanmiljenovic.wordpress.com/2010/03/15/repeat-after...
- Smaug123 4y ago
  Wait til you see the eight ways Nix has of packaging Rust applications! https://nixos.wiki/wiki/Rust#Packaging_Rust_projects_with_nix https://nixos.wiki/wiki/Rust#Packaging_Rust_projects_with_ni...
  - baby 4y ago
    And they’re all horrible
    - 0x457 4y ago
      Why? As long as it can take `Cargo.lock` as input instead of making you calculate hash yourself - it's all good. I'm using `crane` and `buildRustPackage` to build wasm front-end that is embedded into server binary (also rust). It works like a charm.
      - baby 4y ago
        So one problem we have at the moment in our project is that outdated Cargo.lock will basically break everything. Using `nix develop` will just fail to build the derivation, instead of regenerating the correct Cargo.lock (because it depends on someone not using `nix develop` to build the Rust project). It seems like the ideal way to mix nix and cargo is to simply trust and delegate to cargo once you reach a rust dependency, not try to nixify it. I think there was a rustup library for nix that probably would allow doing that but I haven't spent enough time trying (it's such a pain to debug flake.nix files)
        4y ago
illiarian 4y ago
One question I have after all these articles and internal company presentations is: how exactly does Bazel track dependencies? Module A importing module B importing module C is a language-level feature, not a build system-level feature.
- iampims 4y ago
  You have to declare them in each BUILD file.
- teaearlgraycold 4y ago
  You need to duplicate all dependency declarations of the language into the bazel build system.
- keddad 4y ago
  You basically have to declare all those dependencies in Bazel. For instance, if you look at rules for Go, they declare the whole dependency graph (Modules, their relations) in Bazel. There is tooling to generate that for you, so it's not really that bad. Where it gets harder is when this language feature is integrated with an existing build system. C# is normally build with it's own build system, and to build it with Bazel you have to essentially rewrite some of it's logic in Bazel, which can be hard, but doable.
  - DC-3 4y ago
    Have you got any more information on how to use Bazel with C#?
    - keddad 4y ago
      There are open-source rules for that, but at my place of work, we found those lacking. We had to write our own set of rules which would basically take all the source files for a given target, generate MSbuild files depending on the target type and call it from Bazel to build the code in this isolated environment. Another problem are nugets: you have to wrap each in a Bazel target in order to make it available during build, and analyze them beforehand to understand what kind of outputs (like .dlls) will they generate during build. It's a bit of a mess, sadly.
habitue 4y ago
Hey, what a surprise, ex-google engineers dragging everyone into bazel... Maybe you have some really complicated use case that makes cargo insufficient, but really, it's pretty damn nice and it's integrated with everything. There are languages that need bazel, rust isn't one of them
- rtpg 4y ago
  Even just for running test suites incrementally, Bazel is pretty nice. If you have CI that takes "too long", Bazel is basically the only serious tool that can get you to incremental CI in a pretty rigorous way. Though I seriously wish there were a better option, because there are a _lot_ of usability things with Bazel that make me very frustrated when working with a repo that is "merely" quite large.
- bilekas 4y ago
  I'm inclined to agree, but mainly because I can't see an issue where cargo is insufficient, it seems to me that would be an issue somewhere else. Not cargo specific.
  - charcircuit 4y ago
    One category where it helps are *-sys packages which are lacking a proper build system. Cargo's build.rs is a messy solution and is pretty limited.
  - ajconway 4y ago
    Cargo is insufficient if a project has cross-language dependencies.
  - duped 4y ago
    Cargo is sufficient if what you're building is an executable or a Rust library. It still doesn't support post build steps, and integrates poorly with other tools that are used for more general builds.
- bjackman 4y ago
  Lol, Googler here, I can't deny that I have he urge to force Bazel into the world. It's just, when I joined Google I found a build system I didn't hate. This was the first time. I don't actually love Bazel (Blaze internally), but I love that I don't hate it. Every time I have to learn a new build system in open source work I groan. Why are there so many systems for essentially the same thing?
sbt 4y ago
We use a similar setup where I work. One challenge we have has been generating bazel targets for different sets of crate features for dependencies. We have solved this by writing Bazel tooling, but learning how to do this has cost a fair bit of time. We also rely heavily on cargo-raze, which has rough edges. As with all things Bazel, it's great when it works, but it has a steep learning curve that could be hard to justify in many situations. You need at least one person on the team who is willing to really spend time on it.
- ttiurani 4y ago
  Have you tried to migrate from raze to crate universe? It's close to feature parity and raze seems to be close to deprecation. I successfully migrated recently and it's such a gigantic improvement.
  - sbt 4y ago
    I didn't know about this but will check it out, thanks.
SuperSandro2000 4y ago
NixOS maintainer here and I hate bazel through and through. I don't know any other build system that is more anti distro them bazel.
- Mesopropithecus 4y ago
  In what way?
  - fancl20 4y ago
    I guess because package managers (apt, nix, ..) build around the idea of shared libraries, while bazel assume statically built. TBH it’s hard to say which operate model is better anyway. They are simply incompatible :) Related topic: https://youtu.be/Pzl1B7nB9Kc https://youtu.be/Pzl1B7nB9Kc
    - unicornmama 4y ago
      Bazel does not assume statically built. Where do you get this idea?
      - saidinesh5 4y ago
        From their open issues and ugly hacks needed to build anything else... https://github.com/bazelbuild/bazel/issues/1920 https://github.com/bazelbuild/bazel/issues/1920 https://stackoverflow.com/questions/32845940/symbols-from-static-cc-library-dependency-in-so-missing https://stackoverflow.com/questions/32845940/symbols-from-st...
        unicornmama 4y ago
        Those look like the problems with rules cc for building C++. But it’s not inherent to Bazel, and would not expect problems producing static binaries in other languages.
        4y ago
ghuntley 4y ago
hmm. this migration would have been possible by staying with nix and developing a rules_rust equivalent (maybe it already exists) - see https://github.com/ghuntley/depot/tree/trunk/nix/buildGo https://github.com/ghuntley/depot/tree/trunk/nix/buildGo for example but for golang. provides a simple dsl over nix and hides nix.
tragomaskhalos 4y ago
There is a broad tendency in software to jump onto whatever the big FAANG guys are doing. The trouble with this is often that they are working at a vastly bigger scale than you are, hence their solutions are vastly overfitted to your problem. So it's definitely interesting to read about this stuff but I'm sure Cargo is fine for most folks.
- saidinesh5 4y ago
  Bigger scale.. or sometimes they just have different requirements.
- malkia 4y ago
  There is also big tendendcy to ignore them "because we are not at that scale" without realizing that you've reached the scale where you need it, even though your scale is magnitudes still less than what FAANG is. The fallacy here is that just because you are not at the scale of the FAANG does not mean that their tools/libraries/products are not suitable for you - e.g. it's not the case that only if you reach their scale, then maybe you should switch there. This is bs.
  - habitue 4y ago
    This is true, and so you have to look at it case by case. I think if most developers look at the switching costs of bazel vs. just using a decent build tool like cargo, they wont make that tradeoff. Similarly, they might choose some other tool out of google because it doesnt extract such a high price
wongarsu 4y ago
I can get behind a story of starting with cargo, trying to improve by adding sccache, and when that's not enough switching to Bazel. I believe 99% of projects/organisations are best served with the first two of these steps, but it's nice to have the option of going to Bazel for the other 1% of cases (like when you have a lot of inhouse C++ in the mix).
renewiltord 4y ago
I think this would have been useful with some code. My experience with Bazel is that you need a BUILD file in each subdirectory or something like that and every time I add a file I have to add it to that BUILD file. Exceedingly tedious to use and has poor support in Clion. It is likely that I don't have the constraints that this company has. The last time I tried this in Java I had to replicate my package dependency graph in Bazel. It was exceedingly pointless. I found an example repo for rust and Bazel and it is just as tedious as I recall https://github.com/laramiel/bazel-example-rust https://github.com/laramiel/bazel-example-rust
- Game_Ender 4y ago
  It is super tedious but for some languages a tool called Gazelle can auto-generate these for you. I know Go works well and there is a Java plugin too [0]. There is definitely some level rust support because that is how external crates are handled. 0 - https://github.com/bazel-contrib/rules_jvm/tree/main/java/gazelle https://github.com/bazel-contrib/rules_jvm/tree/main/java/ga...
- malkia 4y ago
  You don't necesseraily need to, but ideally. a BUILD file declares a "package" and the demarcation line happens when it sees another BUILD file in a sub-directory. Then that's another "package".
cybrexalpha 4y ago
I've never seen a project that uses Bazel that didn't also have a backlog task to remove it.
- dilyevsky 4y ago
  You’re welcome - https://github.com/envoyproxy/envoy https://github.com/envoyproxy/envoy
- sam_bristow 4y ago
  The [Magma project](https://github.com/magma/magma https://github.com/magma/magma) uses Bazel. The transition seems to have gone well enough they're looking at ripping out the legacy Makefile based build system they maintained in parallel.
19h 4y ago
I used bazel for a few internal repositories written in Rust and it's amazing .... until you hit an issue. Edge cases in Bazel are about as documented as any random fish you might find in a crack at the bottom of the ocean. And the most hilarious thing to deal with is "yeah this was written for a Google project, that other use case didn't apply to our project". When eventually after the Xth issue I was told "why don't you just fork this standard package" (it was for typescript support inside the Rust repo) I was like "f*** bazel" and never touched it again. It's amazing when it works. If it doesn't you're f******.
- klodolph 4y ago
  The experience of using Bazel inside Google with support is different from the experience of using Bazel outside Google. Some ex-Googler may want to use Bazel and may have very good reasons for it, but they’re often unprepared to take over the support duties that, at Google, are provided by the developer tool teams.
  - dboreham 4y ago
    "We did X at Google" for me has become an antipattern and a red flag.
    - klodolph 4y ago
      In general, yes. Bazel has a lot of potential. It can improve developer productivity, with better correctness guarantees and improved build times. Or it can be a big sink of developer time, with people spending endless hours managing Bazel. It has potential to go both ways.
  - dilyevsky 4y ago
    I found when i started using bazel outside of google that my mental model for how its apis really work was really lacking. Once i learned the apis (and the docs are outstanding at helping with that), and with the help of a few examples easily searchable via github, it became quite manageable
- 4y ago
dpc_01234 4y ago
I'm using Nix Flakes to support a active and medium-sized Open Source development team working on a decently-sized Rust project which includes cross-compilation for mobile devices, CI with larger integration tests, and so on. I had to find solutions to some of the same aspects as described here. > This nix-alienation bifurcated our build environment: the CI servers built the code with nix-build, and developers built the code by entering the nix-shell and invoking cargo. I don't anything wrong with this. Why would you throw the ability to simply use the Rust toolchain and all other tools directly? It might be the fault of `nix-shell` which was a confusing and worked in some contrived ways. Nix Flakes have `nix develop` with an explicit "nix dev shell" construct which starts a shell that simply puts all the tools neccessary to build the project in the PATH (and possibly other relevant envs). In our project developers simply need to install Nix and run `nix develop` (or use direnv) and from there on they mostly don't need to remember about Nix during their local work. Only when they need to touch something build/CI related they need to touch nix files, which is maybe 5% of the work, and then they can just ask for help. The higher level tests do require a bit of thought so they reuse Nix building system, instead of fighting against it. > Our security team chose Ubuntu as the deployment target and insisted that production binaries link against the regularly updated system libraries (libc, libc++, openssl, etc.) the deployment platform provides. > Furthermore, the infrastructure team got a few new members unfamiliar with nix and decided to switch to a more familiar technology, Docker containers. The team implemented a new build system that runs cargo builds inside a docker container with the versions of dynamic libraries identical to those in the production environment. I think at this point it was game over for using Nix. Nix has a great support for building OCI (aka Docker) containers. It's great to be able to write 10 lines of Nix and just import all the required derivations from existing Nix build system and get a quick to build, nicely cacheable container that has only 35M. But if you're not going to use it, then you turn a benefit into an extra work. Using Nix to build inside Dockerfile is a terrible combination, eliminating lots of benefits of each done properly in separation. > The idea of migrating the build system came from a few engineers (read Xooglers) who were tired of fighting with long build times and poor tooling. It seems to me that it comes down to this. People didn't want to learn Nix, so they constructed bunch of incompatible ways of doing things. Eventually there were already bunch of people familiar with Bazel so that's what they used. Had you have few engineers familiar with Nix, they would restructure things and make them work with Nix. Some final tactical advice for people that would want to solve these problems with Rust. Use Nix Flakes, they are soo much better than old style Nix. Use `crane` for Rust - IMO the best compromise between "not rebuilding everything" and "simple wrapper over cargo". Use cachix or some shared Nix caching. Let developers work in `nix develop` shell and do their thing without thinking too much about Nix. Whenever they need part of codebase running reproducibly (fixtures for tests etc.) tactically invoke `nix build` and `nix run` . For normal Linux distros, provide OCI containers built natively with Nix. If you need raw binaries in order of preference: cross-compile to Musl (can't be done for C++), patchelf (a bit risky with C++ as OP mentioned), maintain a parallel build procedure via Dockerfile, but don't use Nix in it, and use it only to build the artifacts and nothing else.

107 points by zdw 4y ago

https://github.com/sagiegurari/cargo-make https://github.com/sagiegurari/cargo-make is also a good (and simple) solution for the requirements in the post

IceWreck 4y ago

If you want something similar thats written in Rust instead of Java, look at Buck2 https://github.com/facebook/buck2 https://github.com/facebook/buck2

Alifatisk 4y ago
The readme states the following > This project is not yet polished. We are continuing to develop it in the open, but don't expect it to be suitable for most people… So it’s not ready yet to be used for anything serious I assume.
sluongng 4y ago
Similar to EdenFS/Sapling VCS, buck2 is being developed as such a fast pace that it's hard to imagine there would be any OSS support for it. Together with buck2 are 3-5 other repos that are supposed to be built and used together with it. If you want stability, Bazel is still a much better choice with their LTS release program.
hocuspocus 4y ago
I would rather have a look at Pants since it's now developed for the needs of Toolchain's customers, and not following the roadmap of a big company with a pretty unique infra. Its engine was also rewritten from Java to Rust.
- twunde 4y ago
  I cannot wait until Pants supports Rust as a target language (they were looking for rust consumers as of 3-6 months ago if I recall). For our python needs it was a real breath of fresh air. I'm now at a shop without it and need to deal with slow dependency installs and dependency conflicts again :'(
baby 4y ago
I was wondering the same. I’ve heard of Buck2 the other day and it seems much better. Anyone has tried both?

Animats 4y ago

[flagged]

mrkeen 4y ago

The joy of tooling. If there's one tool it's not a mature ecosystem, if there's two, there's no standard way to do things. "Cargo is not a build system" reminded me of "Cabal is not a package manager" https://ivanmiljenovic.wordpress.com/2010/03/15/repeat-after-me-cabal-is-not-a-package-manager/ https://ivanmiljenovic.wordpress.com/2010/03/15/repeat-after...

Smaug123 4y ago
Wait til you see the eight ways Nix has of packaging Rust applications! https://nixos.wiki/wiki/Rust#Packaging_Rust_projects_with_nix https://nixos.wiki/wiki/Rust#Packaging_Rust_projects_with_ni...
- baby 4y ago
  And they’re all horrible
  - 0x457 4y ago
    Why? As long as it can take `Cargo.lock` as input instead of making you calculate hash yourself - it's all good. I'm using `crane` and `buildRustPackage` to build wasm front-end that is embedded into server binary (also rust). It works like a charm.
    - baby 4y ago
      So one problem we have at the moment in our project is that outdated Cargo.lock will basically break everything. Using `nix develop` will just fail to build the derivation, instead of regenerating the correct Cargo.lock (because it depends on someone not using `nix develop` to build the Rust project). It seems like the ideal way to mix nix and cargo is to simply trust and delegate to cargo once you reach a rust dependency, not try to nixify it. I think there was a rustup library for nix that probably would allow doing that but I haven't spent enough time trying (it's such a pain to debug flake.nix files)
      - 4y ago

illiarian 4y ago

One question I have after all these articles and internal company presentations is: how exactly does Bazel track dependencies? Module A importing module B importing module C is a language-level feature, not a build system-level feature.

iampims 4y ago
You have to declare them in each BUILD file.
teaearlgraycold 4y ago
You need to duplicate all dependency declarations of the language into the bazel build system.
keddad 4y ago
You basically have to declare all those dependencies in Bazel. For instance, if you look at rules for Go, they declare the whole dependency graph (Modules, their relations) in Bazel. There is tooling to generate that for you, so it's not really that bad. Where it gets harder is when this language feature is integrated with an existing build system. C# is normally build with it's own build system, and to build it with Bazel you have to essentially rewrite some of it's logic in Bazel, which can be hard, but doable.
- DC-3 4y ago
  Have you got any more information on how to use Bazel with C#?
  - keddad 4y ago
    There are open-source rules for that, but at my place of work, we found those lacking. We had to write our own set of rules which would basically take all the source files for a given target, generate MSbuild files depending on the target type and call it from Bazel to build the code in this isolated environment. Another problem are nugets: you have to wrap each in a Bazel target in order to make it available during build, and analyze them beforehand to understand what kind of outputs (like .dlls) will they generate during build. It's a bit of a mess, sadly.

habitue 4y ago

Hey, what a surprise, ex-google engineers dragging everyone into bazel... Maybe you have some really complicated use case that makes cargo insufficient, but really, it's pretty damn nice and it's integrated with everything. There are languages that need bazel, rust isn't one of them

rtpg 4y ago
Even just for running test suites incrementally, Bazel is pretty nice. If you have CI that takes "too long", Bazel is basically the only serious tool that can get you to incremental CI in a pretty rigorous way. Though I seriously wish there were a better option, because there are a _lot_ of usability things with Bazel that make me very frustrated when working with a repo that is "merely" quite large.
bilekas 4y ago
I'm inclined to agree, but mainly because I can't see an issue where cargo is insufficient, it seems to me that would be an issue somewhere else. Not cargo specific.
- charcircuit 4y ago
  One category where it helps are *-sys packages which are lacking a proper build system. Cargo's build.rs is a messy solution and is pretty limited.
- ajconway 4y ago
  Cargo is insufficient if a project has cross-language dependencies.
- duped 4y ago
  Cargo is sufficient if what you're building is an executable or a Rust library. It still doesn't support post build steps, and integrates poorly with other tools that are used for more general builds.
bjackman 4y ago
Lol, Googler here, I can't deny that I have he urge to force Bazel into the world. It's just, when I joined Google I found a build system I didn't hate. This was the first time. I don't actually love Bazel (Blaze internally), but I love that I don't hate it. Every time I have to learn a new build system in open source work I groan. Why are there so many systems for essentially the same thing?

sbt 4y ago

We use a similar setup where I work. One challenge we have has been generating bazel targets for different sets of crate features for dependencies. We have solved this by writing Bazel tooling, but learning how to do this has cost a fair bit of time. We also rely heavily on cargo-raze, which has rough edges. As with all things Bazel, it's great when it works, but it has a steep learning curve that could be hard to justify in many situations. You need at least one person on the team who is willing to really spend time on it.

ttiurani 4y ago
Have you tried to migrate from raze to crate universe? It's close to feature parity and raze seems to be close to deprecation. I successfully migrated recently and it's such a gigantic improvement.
- sbt 4y ago
  I didn't know about this but will check it out, thanks.

SuperSandro2000 4y ago

NixOS maintainer here and I hate bazel through and through. I don't know any other build system that is more anti distro them bazel.

Mesopropithecus 4y ago
In what way?
- fancl20 4y ago
  I guess because package managers (apt, nix, ..) build around the idea of shared libraries, while bazel assume statically built. TBH it’s hard to say which operate model is better anyway. They are simply incompatible :) Related topic: https://youtu.be/Pzl1B7nB9Kc https://youtu.be/Pzl1B7nB9Kc
  - unicornmama 4y ago
    Bazel does not assume statically built. Where do you get this idea?
    - saidinesh5 4y ago
      From their open issues and ugly hacks needed to build anything else... https://github.com/bazelbuild/bazel/issues/1920 https://github.com/bazelbuild/bazel/issues/1920 https://stackoverflow.com/questions/32845940/symbols-from-static-cc-library-dependency-in-so-missing https://stackoverflow.com/questions/32845940/symbols-from-st...
      - unicornmama 4y ago
        Those look like the problems with rules cc for building C++. But it’s not inherent to Bazel, and would not expect problems producing static binaries in other languages.
        4y ago

ghuntley 4y ago

hmm. this migration would have been possible by staying with nix and developing a rules_rust equivalent (maybe it already exists) - see https://github.com/ghuntley/depot/tree/trunk/nix/buildGo https://github.com/ghuntley/depot/tree/trunk/nix/buildGo for example but for golang. provides a simple dsl over nix and hides nix.

tragomaskhalos 4y ago

There is a broad tendency in software to jump onto whatever the big FAANG guys are doing. The trouble with this is often that they are working at a vastly bigger scale than you are, hence their solutions are vastly overfitted to your problem. So it's definitely interesting to read about this stuff but I'm sure Cargo is fine for most folks.

saidinesh5 4y ago
Bigger scale.. or sometimes they just have different requirements.
malkia 4y ago
There is also big tendendcy to ignore them "because we are not at that scale" without realizing that you've reached the scale where you need it, even though your scale is magnitudes still less than what FAANG is. The fallacy here is that just because you are not at the scale of the FAANG does not mean that their tools/libraries/products are not suitable for you - e.g. it's not the case that only if you reach their scale, then maybe you should switch there. This is bs.
- habitue 4y ago
  This is true, and so you have to look at it case by case. I think if most developers look at the switching costs of bazel vs. just using a decent build tool like cargo, they wont make that tradeoff. Similarly, they might choose some other tool out of google because it doesnt extract such a high price

wongarsu 4y ago

I can get behind a story of starting with cargo, trying to improve by adding sccache, and when that's not enough switching to Bazel. I believe 99% of projects/organisations are best served with the first two of these steps, but it's nice to have the option of going to Bazel for the other 1% of cases (like when you have a lot of inhouse C++ in the mix).

renewiltord 4y ago

I think this would have been useful with some code. My experience with Bazel is that you need a BUILD file in each subdirectory or something like that and every time I add a file I have to add it to that BUILD file. Exceedingly tedious to use and has poor support in Clion. It is likely that I don't have the constraints that this company has. The last time I tried this in Java I had to replicate my package dependency graph in Bazel. It was exceedingly pointless. I found an example repo for rust and Bazel and it is just as tedious as I recall https://github.com/laramiel/bazel-example-rust https://github.com/laramiel/bazel-example-rust

Game_Ender 4y ago
It is super tedious but for some languages a tool called Gazelle can auto-generate these for you. I know Go works well and there is a Java plugin too [0]. There is definitely some level rust support because that is how external crates are handled. 0 - https://github.com/bazel-contrib/rules_jvm/tree/main/java/gazelle https://github.com/bazel-contrib/rules_jvm/tree/main/java/ga...
malkia 4y ago
You don't necesseraily need to, but ideally. a BUILD file declares a "package" and the demarcation line happens when it sees another BUILD file in a sub-directory. Then that's another "package".

cybrexalpha 4y ago

I've never seen a project that uses Bazel that didn't also have a backlog task to remove it.

dilyevsky 4y ago
You’re welcome - https://github.com/envoyproxy/envoy https://github.com/envoyproxy/envoy
sam_bristow 4y ago
The [Magma project](https://github.com/magma/magma https://github.com/magma/magma) uses Bazel. The transition seems to have gone well enough they're looking at ripping out the legacy Makefile based build system they maintained in parallel.

19h 4y ago

I used bazel for a few internal repositories written in Rust and it's amazing .... until you hit an issue. Edge cases in Bazel are about as documented as any random fish you might find in a crack at the bottom of the ocean. And the most hilarious thing to deal with is "yeah this was written for a Google project, that other use case didn't apply to our project". When eventually after the Xth issue I was told "why don't you just fork this standard package" (it was for typescript support inside the Rust repo) I was like "f*** bazel" and never touched it again. It's amazing when it works. If it doesn't you're f******.

klodolph 4y ago
The experience of using Bazel inside Google with support is different from the experience of using Bazel outside Google. Some ex-Googler may want to use Bazel and may have very good reasons for it, but they’re often unprepared to take over the support duties that, at Google, are provided by the developer tool teams.
- dboreham 4y ago
  "We did X at Google" for me has become an antipattern and a red flag.
  - klodolph 4y ago
    In general, yes. Bazel has a lot of potential. It can improve developer productivity, with better correctness guarantees and improved build times. Or it can be a big sink of developer time, with people spending endless hours managing Bazel. It has potential to go both ways.
- dilyevsky 4y ago
  I found when i started using bazel outside of google that my mental model for how its apis really work was really lacking. Once i learned the apis (and the docs are outstanding at helping with that), and with the help of a few examples easily searchable via github, it became quite manageable
4y ago

dpc_01234 4y ago

I'm using Nix Flakes to support a active and medium-sized Open Source development team working on a decently-sized Rust project which includes cross-compilation for mobile devices, CI with larger integration tests, and so on. I had to find solutions to some of the same aspects as described here. > This nix-alienation bifurcated our build environment: the CI servers built the code with nix-build, and developers built the code by entering the nix-shell and invoking cargo. I don't anything wrong with this. Why would you throw the ability to simply use the Rust toolchain and all other tools directly? It might be the fault of `nix-shell` which was a confusing and worked in some contrived ways. Nix Flakes have `nix develop` with an explicit "nix dev shell" construct which starts a shell that simply puts all the tools neccessary to build the project in the PATH (and possibly other relevant envs). In our project developers simply need to install Nix and run `nix develop` (or use direnv) and from there on they mostly don't need to remember about Nix during their local work. Only when they need to touch something build/CI related they need to touch nix files, which is maybe 5% of the work, and then they can just ask for help. The higher level tests do require a bit of thought so they reuse Nix building system, instead of fighting against it. > Our security team chose Ubuntu as the deployment target and insisted that production binaries link against the regularly updated system libraries (libc, libc++, openssl, etc.) the deployment platform provides. > Furthermore, the infrastructure team got a few new members unfamiliar with nix and decided to switch to a more familiar technology, Docker containers. The team implemented a new build system that runs cargo builds inside a docker container with the versions of dynamic libraries identical to those in the production environment. I think at this point it was game over for using Nix. Nix has a great support for building OCI (aka Docker) containers. It's great to be able to write 10 lines of Nix and just import all the required derivations from existing Nix build system and get a quick to build, nicely cacheable container that has only 35M. But if you're not going to use it, then you turn a benefit into an extra work. Using Nix to build inside Dockerfile is a terrible combination, eliminating lots of benefits of each done properly in separation. > The idea of migrating the build system came from a few engineers (read Xooglers) who were tired of fighting with long build times and poor tooling. It seems to me that it comes down to this. People didn't want to learn Nix, so they constructed bunch of incompatible ways of doing things. Eventually there were already bunch of people familiar with Bazel so that's what they used. Had you have few engineers familiar with Nix, they would restructure things and make them work with Nix. Some final tactical advice for people that would want to solve these problems with Rust. Use Nix Flakes, they are soo much better than old style Nix. Use `crane` for Rust - IMO the best compromise between "not rebuilding everything" and "simple wrapper over cargo". Use cachix or some shared Nix caching. Let developers work in `nix develop` shell and do their thing without thinking too much about Nix. Whenever they need part of codebase running reproducibly (fixtures for tests etc.) tactically invoke `nix build` and `nix run` . For normal Linux distros, provide OCI containers built natively with Nix. If you need raw binaries in order of preference: cross-compile to Musl (can't be done for C++), patchelf (a bit risky with C++ as OP mentioned), maintain a parallel build procedure via Dockerfile, but don't use Nix in it, and use it only to build the artifacts and nothing else.