4 ms·

Woah!! You say "If you never hit these needs, you don't need Bazel", but even for a small-ish project, I would like to have Automatic caching of everything, Dep

by j1elo 2y ago

Woah!! You say "If you never hit these needs, you don't need Bazel", but even for a small-ish project, I would like to have Automatic caching of everything, Dependency based test selection, Automatic sandboxing of build steps, or seamless shared caches! I guess for a small team (say less than 10 people) the complexity of Bazel is too much to make it worth it, but those features you mention seem like universally useful for any project. I bet there are lots and lots of reinvented wheels out there trying to do just a subset of all these.

lihaoyi 2y ago
Empirically speaking, unless you are already a Bazel expert or are lucky enough to hire one, most orgs I've seen need at least 3-5 full-time engineers to become experts in Bazel and about ~1 person decade of engineering to roll it out (i.e. it takes those 3-5 guys 2-3 years). Adopting Bazel is not something you do over a weekend hackathon; the cost starts becoming reasonable once you have 100-200 engineers who can benefit from the 3-5 guys maintaining their build tool For your small-ish project I'm guessing you have wants, but if the project grows at some point those will transition into needs and then you may have to make hard choices. That's my experience rolling it out Bazel myself and maintaining it for a growing organization and codebase over 7 years I'm hoping the Mill build tool I'm working on can become an easier/friendlier alternative that provides many of the same benefits, even if it can't support the extreme Google scale that Bazel does. So if you think these things sound cool, but you don't want to spend 1 person decade on your build tool, you should check out Mill!
- j1elo 2y ago
  That sounds awesome, great work trying to simplify the access to those features! Also thanks for the insights
- dilyevsky 2y ago
  You make this sound considerably more involved than it actually is from my experience observing/supporting Bazel infra at three companies ranging from 10 to 1K engineers. Adopting Bazel for C, C++, Java and Go has been pretty straight-forward for years now unless you want power features like remote execution, custom rules/macros, etc
  - lihaoyi 2y ago
    I don't doubt your experience, I can only provide mine. I've seen one rollout from the inside (~1000 engineers), performed one rollout myself (100-1000 engineers), and talked to ~10 other companies trying to do their own rollouts (100-1000 engineers). Everyone has their own unique circumstances, and I can only speak for what I have seen myself (first or second hand)
  - jrockway 2y ago
    I agree. We moved to Bazel at my last job and it probably took about 6 person months. I was most of that person and it includes other tooling not related to Bazel. Some extremely competent engineers also moved over our frontend code (including stuff like Cypress) and Python code (which needs to run against 3 different versions of Python). They had no Bazel experience beforehand and asked me maybe a couple hours worth of questions and just got it done. So I don't think you need to be a Bazel genius to get this done, but it helps to have someone with a Vision, which was me in this case. All in all, I'd do it again. I'm in the process of moving all my open source code to a monorepo (jrockway/monorepo, which really should have been called jrockway/jrockway) because the development experience is so much better. The biggest goal for me in doing the project at work was because new employees couldn't run the code they were working on. We hired people. They tried. They weren't very productive. That's my fault, and I wanted to fix that while supporting our policy of "you can use any Linux distribution you want, or you can use an arm64 Mac". Many people suggested things like "force everyone to use NixOS" which I would be in favor of, but it wasn't the solution that won. (I honestly prefer Debian myself and didn't think that my preference should dictate how the team works. The fact that I disagreed with the proposed solution is a good indicator that people would be unhappy with anything I declared by fiat.) Rather, using Bazel to provide a framework for retrieving third-party tooling and also building our code was a comfortable compromise. A secondary goal was test caching. If you edit README.md, CI doesn't need to rerun the Cypress tests. (As a corollary, if you edit "metadata_server.go", the "pfs_server.go" tests don't need to run, as the PFS server does not depend on the Metadata server.) The biggest piece of slowness and complexity in the workflow would be building our code into a container to run in k8s. We used goreleaser and that involved building the code twice, one for each architecture, to build into an OCI image index which was our main release artifact. The usual shell scripts for local development just reused this and it was terribly slow. Throw in docker to do the builds, which deletes the Go build cache after every build, and you have a recipe for not getting anything done. Bazel is a much better way to build containers. Containers are just some JSON files and tar files. Bazel (rules_oci) just assembles your build artifacts into the necessary JSON files and tar files. To build a multi-architecture image index, you build twice and add a JSON file. Bazel handles this with platform transitions; you make a rule to build for the host architecture (technically transitioned to linux on a macos host), and then the image index rule just builds that target with two configurations (cross-compiled, not emulated with binfmt_misc like "docker buildx") and assembles the two artifacts into the desired multi-arch image. When running locally, you skip 2 of the 3 steps, just build for the host machine. Combined with proper build caching (thanks BuildBuddy!), this means that making an image to run in k8s takes about 10 seconds instead of 6 minutes. With the previous system you could try your code 80 times a day. With the new system, you could try it 2880 times ;) This increased productivity. I also wrote a bunch of tools to make setting up k8s easier, which would have been perfectly possible without Bazel, but it helped. (Before, everyone pushed their built image to DockerHub and then reconfigured k8s to pull that. Now we have a local registry, and if two people do a build at the same time, you always get yours and not theirs. I did not design this previous system, I merely set out to fix it because it's Wrong.) Bazel makes vendoring tools pretty easy. For our product we needed things like kubectl, kind, skopeo, postgres, etc. These are all in //tools/whatever and can be run for your host machine with `bazel run //tools/whatever`. So once you ran my program to create and update your environment, you automatically had the right version of the tool to interact with it. We upgraded k8s regularly, nobody noticed. They would just get the new tool the next time they tried to run it. (A centrally managed linux distribution would do the same thing, but it couldn't revert you to an old tool when you checked out an old version to debug. A README with versions would work, but I learned that nobody really reads the READMEs until you ask them to. "How do I do X" "See this section of the README" "Oh damn I wish I thought of that" "Me too." ;) The biggest problem I had with Bazel in the past was dealing with generated files. Editor support, "go get <our thing>", etc. I got by when I used Blaze at Google, but realistically, there was no editor support for Go at that time, so I didn't notice how badly it worked. There is now GOPACKAGESDRIVER, which technically helps, but it didn't work well for me and I wasn't going to inflict it upon my team. I punted this time and continued to check in generated files. We have a target //:make_proto that rebuilds the proto files, and a test that checks that you did it. You check in the generated protos when you change the protos. It works and I have a general rule for all generations like this. (We also generate a bunch of JSON from Jsonnet files; this mechanism helps with that.) All in all, you can get a fresh machine, install git and Bazelisk, check out our repo, and get "bazel test ..." to succeed. That, to me, is the minimum dev experience that your employer owes you, and if you joined my team, you'd get it. That made me happy and it wouldn't be as good without Bazel. I'd do it again! Just as an aside, after the Bazel conversion, I did a really complicated change, and Bazel didn't make it any harder. We made our main product depend on pg_dump, and adjusting the container building rules from "distroless plus a go binary" to "debian plus postgres plus a go binary" was pretty easy. rules_debian is very nifty, and it gives me a sense of reproducibility that I never got from "FROM debian:stable@sha256...; RUN apt-get update && apt-get install ....". Indeed, the reproducibility is there. You can take any release tag, "bazel build //oci:whatever" and see that the resulting multi-arch image has the same sha256 of what's on DockerHub. I couldn't have done that without Bazel, or at least not without writing a lot of code. I don't work there anymore but I'm really happy about the project. I don't even do Build & Release stuff. I just add features to the product. But this needed to be done and I was happy to wear the hat.
dieortin 2y ago
I migrated my team of ~20 developers to Bazel, and I have to say I don’t think it is that complex. Not only don’t we have 3 full time engineers devoted to Bazel, as I do everything myself, but in fact it’s not even my main role to maintain the build system. We do not use some of the more complex features like remote execution, but we do enjoy all the other features including remote caching. We reduced build times by 92% after the migration. So my advice would be, try it and see for yourself if you think it’s worth the hassle. For my team, it definitely has been.
- m1keil 2y ago
  So what happens if you leave tomorrow? Will your team be able to pick up where you left off?
  - binary132 2y ago
    It is really not that hard to grok and use. I think cmake is actually a bit harder. It doesn’t hurt that bazel builds are implemented in starlark instead of a zany state manipulation language-that-isn’t-a-language.
- liontwist 2y ago
  I would hope that a build system for 20 people takes zero time to maintain.

by j1elo 2y ago

lihaoyi 2y ago

Empirically speaking, unless you are already a Bazel expert or are lucky enough to hire one, most orgs I've seen need at least 3-5 full-time engineers to become experts in Bazel and about ~1 person decade of engineering to roll it out (i.e. it takes those 3-5 guys 2-3 years). Adopting Bazel is not something you do over a weekend hackathon; the cost starts becoming reasonable once you have 100-200 engineers who can benefit from the 3-5 guys maintaining their build tool For your small-ish project I'm guessing you have wants, but if the project grows at some point those will transition into needs and then you may have to make hard choices. That's my experience rolling it out Bazel myself and maintaining it for a growing organization and codebase over 7 years I'm hoping the Mill build tool I'm working on can become an easier/friendlier alternative that provides many of the same benefits, even if it can't support the extreme Google scale that Bazel does. So if you think these things sound cool, but you don't want to spend 1 person decade on your build tool, you should check out Mill!

j1elo 2y ago
That sounds awesome, great work trying to simplify the access to those features! Also thanks for the insights
dilyevsky 2y ago
You make this sound considerably more involved than it actually is from my experience observing/supporting Bazel infra at three companies ranging from 10 to 1K engineers. Adopting Bazel for C, C++, Java and Go has been pretty straight-forward for years now unless you want power features like remote execution, custom rules/macros, etc
- lihaoyi 2y ago
  I don't doubt your experience, I can only provide mine. I've seen one rollout from the inside (~1000 engineers), performed one rollout myself (100-1000 engineers), and talked to ~10 other companies trying to do their own rollouts (100-1000 engineers). Everyone has their own unique circumstances, and I can only speak for what I have seen myself (first or second hand)
- jrockway 2y ago
  I agree. We moved to Bazel at my last job and it probably took about 6 person months. I was most of that person and it includes other tooling not related to Bazel. Some extremely competent engineers also moved over our frontend code (including stuff like Cypress) and Python code (which needs to run against 3 different versions of Python). They had no Bazel experience beforehand and asked me maybe a couple hours worth of questions and just got it done. So I don't think you need to be a Bazel genius to get this done, but it helps to have someone with a Vision, which was me in this case. All in all, I'd do it again. I'm in the process of moving all my open source code to a monorepo (jrockway/monorepo, which really should have been called jrockway/jrockway) because the development experience is so much better. The biggest goal for me in doing the project at work was because new employees couldn't run the code they were working on. We hired people. They tried. They weren't very productive. That's my fault, and I wanted to fix that while supporting our policy of "you can use any Linux distribution you want, or you can use an arm64 Mac". Many people suggested things like "force everyone to use NixOS" which I would be in favor of, but it wasn't the solution that won. (I honestly prefer Debian myself and didn't think that my preference should dictate how the team works. The fact that I disagreed with the proposed solution is a good indicator that people would be unhappy with anything I declared by fiat.) Rather, using Bazel to provide a framework for retrieving third-party tooling and also building our code was a comfortable compromise. A secondary goal was test caching. If you edit README.md, CI doesn't need to rerun the Cypress tests. (As a corollary, if you edit "metadata_server.go", the "pfs_server.go" tests don't need to run, as the PFS server does not depend on the Metadata server.) The biggest piece of slowness and complexity in the workflow would be building our code into a container to run in k8s. We used goreleaser and that involved building the code twice, one for each architecture, to build into an OCI image index which was our main release artifact. The usual shell scripts for local development just reused this and it was terribly slow. Throw in docker to do the builds, which deletes the Go build cache after every build, and you have a recipe for not getting anything done. Bazel is a much better way to build containers. Containers are just some JSON files and tar files. Bazel (rules_oci) just assembles your build artifacts into the necessary JSON files and tar files. To build a multi-architecture image index, you build twice and add a JSON file. Bazel handles this with platform transitions; you make a rule to build for the host architecture (technically transitioned to linux on a macos host), and then the image index rule just builds that target with two configurations (cross-compiled, not emulated with binfmt_misc like "docker buildx") and assembles the two artifacts into the desired multi-arch image. When running locally, you skip 2 of the 3 steps, just build for the host machine. Combined with proper build caching (thanks BuildBuddy!), this means that making an image to run in k8s takes about 10 seconds instead of 6 minutes. With the previous system you could try your code 80 times a day. With the new system, you could try it 2880 times ;) This increased productivity. I also wrote a bunch of tools to make setting up k8s easier, which would have been perfectly possible without Bazel, but it helped. (Before, everyone pushed their built image to DockerHub and then reconfigured k8s to pull that. Now we have a local registry, and if two people do a build at the same time, you always get yours and not theirs. I did not design this previous system, I merely set out to fix it because it's Wrong.) Bazel makes vendoring tools pretty easy. For our product we needed things like kubectl, kind, skopeo, postgres, etc. These are all in //tools/whatever and can be run for your host machine with `bazel run //tools/whatever`. So once you ran my program to create and update your environment, you automatically had the right version of the tool to interact with it. We upgraded k8s regularly, nobody noticed. They would just get the new tool the next time they tried to run it. (A centrally managed linux distribution would do the same thing, but it couldn't revert you to an old tool when you checked out an old version to debug. A README with versions would work, but I learned that nobody really reads the READMEs until you ask them to. "How do I do X" "See this section of the README" "Oh damn I wish I thought of that" "Me too." ;) The biggest problem I had with Bazel in the past was dealing with generated files. Editor support, "go get <our thing>", etc. I got by when I used Blaze at Google, but realistically, there was no editor support for Go at that time, so I didn't notice how badly it worked. There is now GOPACKAGESDRIVER, which technically helps, but it didn't work well for me and I wasn't going to inflict it upon my team. I punted this time and continued to check in generated files. We have a target //:make_proto that rebuilds the proto files, and a test that checks that you did it. You check in the generated protos when you change the protos. It works and I have a general rule for all generations like this. (We also generate a bunch of JSON from Jsonnet files; this mechanism helps with that.) All in all, you can get a fresh machine, install git and Bazelisk, check out our repo, and get "bazel test ..." to succeed. That, to me, is the minimum dev experience that your employer owes you, and if you joined my team, you'd get it. That made me happy and it wouldn't be as good without Bazel. I'd do it again! Just as an aside, after the Bazel conversion, I did a really complicated change, and Bazel didn't make it any harder. We made our main product depend on pg_dump, and adjusting the container building rules from "distroless plus a go binary" to "debian plus postgres plus a go binary" was pretty easy. rules_debian is very nifty, and it gives me a sense of reproducibility that I never got from "FROM debian:stable@sha256...; RUN apt-get update && apt-get install ....". Indeed, the reproducibility is there. You can take any release tag, "bazel build //oci:whatever" and see that the resulting multi-arch image has the same sha256 of what's on DockerHub. I couldn't have done that without Bazel, or at least not without writing a lot of code. I don't work there anymore but I'm really happy about the project. I don't even do Build & Release stuff. I just add features to the product. But this needed to be done and I was happy to wear the hat.

dieortin 2y ago

I migrated my team of ~20 developers to Bazel, and I have to say I don’t think it is that complex. Not only don’t we have 3 full time engineers devoted to Bazel, as I do everything myself, but in fact it’s not even my main role to maintain the build system. We do not use some of the more complex features like remote execution, but we do enjoy all the other features including remote caching. We reduced build times by 92% after the migration. So my advice would be, try it and see for yourself if you think it’s worth the hassle. For my team, it definitely has been.

m1keil 2y ago
So what happens if you leave tomorrow? Will your team be able to pick up where you left off?
- binary132 2y ago
  It is really not that hard to grok and use. I think cmake is actually a bit harder. It doesn’t hurt that bazel builds are implemented in starlark instead of a zany state manipulation language-that-isn’t-a-language.
liontwist 2y ago
I would hope that a build system for 20 people takes zero time to maintain.