13 ms·

Building Uber’s Go Monorepo with Bazel

158 points by dayanruben 6y ago

yannoninator 6y ago
I wish my company was at the scale of Uber to work on all these kinds of tech. Makes me want to leave my current job just to go to Uber.
- msoad 6y ago
  Coordinating this sort of change in a large org like Uber is not entirely "hacking on cool tech". It's a lot of communication and writing documents and going to meetings
  - save_ferris 6y ago
    This. It's very easy to forget about the political processes involved in making projects like this happen. Also, getting a job at a company as big as Uber in no way guarantees you'll be able to work on tech like this. You'd need to make sure you're joining a team close to this project, and if the culture of team turns out to be toxic, the fun of working on "cool" tech fades really, really fast.
  - chimeracoder 6y ago
    > Coordinating this sort of change in a large org like Uber is not entirely "hacking on cool tech". It's a lot of communication and writing documents and going to meetings And a lot of tearing your hair out when trying to use tools in the ways that they weren't intended to be used. Of all languages, Go is not the language where you want to be doing things against the grain (that is, against the way the language designers intend it to be used). Some people like that about Go, and some people don't. Either way, Go is a very opinionated language with a very opinionated ecosystem, and ignoring those is a recipe for frustration. Last time I checked, Bazel was not recommended by the Go developers - and with good reason: there are a lot of gaps in rules_go/Gazelle, which this blog post alludes to but glosses over. While Google uses Blaze internally, Blaze is not Bazel, and the differences are very apparent to anyone who has used both to build Go specifically. Furthermore, rules_go was developed entirely independently of the Blaze ruleset that Google uses internally, so it's a tool that's not actually used internally at Google, but also not used by the majority of the non-Google Go developer community either, in addition to not being recommended by the Go team at Google[0]. [0] There is exactly one mention of Bazel on the entire golang.org domain - in a changelog from over two years ago, in an /x/ package, where Bazel is mentioned as one of two build systems in a "such as" clause that the new package could potentially enable support for (/x/ packages are considered experimental and not subject to the same backwards compatibility or maintenance guarantees as the rest of the project).
    - parsnips 6y ago
      From the issues I've read on github, anecdotally, is that the approach of the gazelle and rules_go teams is to leverage the standard go tooling and not make changes to go itself. Also anecdotally, bazel and golang work really well together IME. The community seems pretty active, and the upsides of using gazelle/bazel with golang seem to outweigh any downsides (though I'd be hard pressed to name a downside, that isn't inherit to golang itself).
      - chimeracoder 6y ago
        > Also anecdotally, bazel and golang work really well together IME... and the upsides of using gazelle/bazel with golang seem to outweigh any downsides This is really not my experience from having used Bazel with Go for the last four years. But I'm happy you are are apparently not running into issues.
        parsnips 6y ago
        I've been using about the same amount of time. Have you tried using python and bazel? Now there's some real weeping and gnashing of teeth ;) Lot's of bad code depending on python == python2 type of non-sense.
        lhorie 6y ago
        Javascript is another target that is just about as gnarly. The node module resolution algorithm is reimplemented a million similar-but-not-quite-the-same-ways by popular packages, most of which struggle dealing w/ symlink-heavy codebases (like bazel sandboxes).
    - jeffbee 6y ago
      Having used both for years I honestly have not perceived the difference between blaze and bazel go_binary/go_library/go_test. What's the big difference?
    - malandrew 6y ago
      > Either way, Go is a very opinionated language with a very opinionated ecosystem except when it comes to dependency management
  - 18nleung 6y ago
    We're not at the scale of Uber yet, but at Jupiter (YC S19) we're using tech like Bazel, protos, and Kotlin to prepare us for when we are: https://starship.jupiter.co/jupiter-stack/ https://starship.jupiter.co/jupiter-stack/ We're actively hiring for a senior SWE right now, so feel free to shoot me a note if you're looking.
    - yuribro 6y ago
      I'm not familiar with the Kotlin build chain, but what role does Bazel play? Is it needed for integration with the protocol buffers?
      - 18nleung 6y ago
        Yes - Bazel has excellent protobuf integration compared to other build tools (we were originally using Gradle). We're compiling Kotlin to the JVM, so the build chain is similar to Java's.
        eleventigers 6y ago
        Gradle has a great protobuf plugin, what is so not excellent compared to Bazel?
    - novok 6y ago
      Protos is more of a faster & smaller network serialization tradeoff vs. ease of development thing than a company scale thing. And a thing to do if you want to integrate with gRPC. I worked at a company that used protos from day one for example. Many large companies are still using JSON w/ schemas as their network serialization layer just fine.
    - deleted 6y ago
      [deleted]
- msrmthehomeless 6y ago
  Don't wish, start leetcoding.
- herrkanin 6y ago
  I wish they weren't morally bankrupt as I would have loved to work on their tech stack.
  - flak48 6y ago
    > morally bankrupt How so, in 2020?
    - morelisp 6y ago
      I think once you're stealing critics' medical records, your morally bankruptcy looks more like a chapter 7 than an 11. There's no expiry date; raze the whole thing.
- lima 6y ago
  Not just large companies - plenty of small startups building cool developer tooling. And much more fun when you get to start on a green field, rather than slowly migrating the existing code over the course of a decade.
- kerng 6y ago
  More politics, then tech.
- klodolph 6y ago
  Not personal experience, but I’ve had several coworkers who worked at Uber, and they all gave me the same story. This is the story: Uber reinvented a ton of technology because of the idea that off-the-shelf solutions wouldn’t work at “Uber scale”. However, there was very little accountability for whether the in-house solutions were necessary, and working on these tools would get you promoted. So for every “Uber scale” problem that a team actually solved, there were a couple other projects that were just half-baked alternatives to the off-the-shelf software that they should be using. It turns out that “Uber scale” is not really that large, despite the name. But engineers kept repeating “Uber scale” and building infrastructure. The same problem occurs at the larger tech companies like Google, Facebook, Microsoft, Amazon, and Apple, but in different ways and to different degrees. And to a large extent, engineers are copying what other companies do, and bringing ideas from one company to another when they hang out after work or switch jobs. For example, you can bet that these companies mostly have their own containerization and scheduling systems, many of which are undoubtedly not competitive with Docker or K8s in 2020, but K8s only goes back to 2014 and all these companies are older. I’m sure Borg and Tupperware are great if you work at Google or Facebook but I’m also sure that they’re missing a bunch of tooling that you’re used to. Same thing with build systems. Bazel, Buck, Pants, Please, and that Frankensteined system that Chrome uses are all copies of each other but Bazel is the only one with a decent size community and ecosystem, as far as I can tell. Uber is absolutely not on the scale of those companies, and most of the time they should probably be using off-the-shelf solutions when they become available.
  - schoolornot 6y ago
    - nothing wrong with NIH syndrome if you have the means and resources to refine existing ideas and rebuild them with scale in mind - uber is a 50 billion dollar company and they have to de-risk themselves by owning entire stacks, top to bottom. if it means re-creating something from scratch... who cares, they have billions.
    - nickv 6y ago
      Or, at that point, you use industry standard solutions and hire the best subject matter experts on that solution. Like, for example, using Cockroach DB and hire one of the core committers cause you're big enough. It's what MSFT did with things like the Python extension for VSCode and DBOX did by hiring Guido since they're a Python shop. I still shake my head whenever I hear about how Uber built an entire Slack clone for "Uber-scale".
- rvz 6y ago
  Uber Engineering sounds like they are collecting Google's open source tools and name-dropping them in their blogs to market themselves as "cool tech". One companies boring tool is another companies favourite hyped third-party tech tool. (Until it doesn't work for them)
- deleted 6y ago
  [deleted]
- wincent 6y ago
  My experience working at Facebook was that the "working at scale" is nowhere near as glamorous as it sounds, especially when it comes to scaling build systems and monorepos. As a company, you have to invest immense resources in evolving the infrastructure and it is a never-ending task: the more capacity you build, the more your organization will grow in ways that tax that capacity. The goal posts are ever-receding. As an individual developer, all that fancy distributed infrastructure becomes a barrier that must be laboriously overcome at every step. Your expectations about what constitutes a "fast" operation get gradually distorted. Working on "small" side-projects (a few thousand files or less) feels effortless in comparison. Not to say you shouldn't seek out the "big tech" experience; just that it is not all roses.
- xbeta 6y ago
  They just had a laid-off recently, and still not profitable given how much $$$ they have burnt over the years. I am not sure if you wanna work there. FANG such as FB and Google has some big monorepos, and you can try that if you are interested.
- q3k 6y ago
  Scale is not a prerequisite, but you do have to sell it to your company. If not a greenfield, porting a codebase over is not easy, and you have to do your organizational excellence homework too. Monorepo, well reviewed code, team not in permanent crunch, high-quality tooling culture. Just like Kubernetes, Bazel is not a silver bullet. But if you get all your ducks in a row, switching over is a gamechanger and will help you out even at the smallest scale.
- typon 6y ago
  You want to join Uber because you think Bazel is cool tech? Having spent the last few weeks of my life converting our build system to Bazel I guarantee you it's far from what I would consider cool. It's more of a necessary evil because the alternatives are even worse.
WoahNoun 6y ago
This will surely bring them to profitability. Exactly the innovation the food delivery space was looking for.
- tequila_shot 6y ago
  Not sure if this is sarcasm. If not, how?
- pm90 6y ago
  I understand the snark but this is hacker news after all. Any kind of technical innovation is awesome to me as an HN reader even if it may not make business sense. Also, I’m totally fine with them using all that investment money for R&D rather than lobbying or returning to shareholders among other things.
  - dmoy 6y ago
    I agree with you. When there was a SpaceX article about bazel, it'd be a bit unfortunate to see a top comment be something about an exploding rocket. (In reality, there were zero comments, so w/e)
  - quest88 6y ago
    What about profitability? R&D is fine if they're profitable, but they're not.
    - bhupy 6y ago
      Uber doesn't need to be profitable yet, they have $10B in cash. While this doesn't mean that they can continue to be unprofitable forever, it means that they have latitude to engage in R&D (like this post) while also moving towards profitability, at the same time.
      - lhorie 6y ago
        Adopting an open source build system doesn't exactly strike me as R&D
chetanbhasin 6y ago
This is excellent! The only reason we haven't moved some of our Go codebase to a Bazel monorepo is the IDE integration. I have been tinkering with a few ideas to make the existing tooling work with Bazel, but the effort is larger than I had originally expected.
DavyJone 6y ago
That seems like a ridiculously complex pipeline to maintain and onboard into. I personally dislike this "do it all" monorepos in most cases, there are cases were they work, but they also break a lot of other things. I have not seen metris or blogs that prove this "uptick in build efficiency" or an increase productivity. While I do like the idea behind bazel, I hate repeating deps in things like "go_repository" with gazelle.
- parsnips 6y ago
  go mod vendor && bazel run gazelle No go_repository duplication.
agounaris 6y ago
The article says that monorepos are more efficient but also that "As the monorepo grew, the build target list increased to a point where it became too long to pass it through Bazel’s command line interface.". Monorepos are not efficient. They are easier to manage when a team is small but as the team grows and you have more and more deliverables with separate versioning you are introducing control structures in your automation. Complexity explodes! Anyway, all this does matter if you don't make any profit :)
- luckydata 6y ago
  Google is a monorepo as far as I know, and they are doing fine.
  - jeffbee 6y ago
    Indeed, in their paper from 5 years ago Google claimed 300000 commits per day across 9 million source files, compared to this article claiming 10000 commits per month on 70000 source files at Uber. Whatever the differences are between blaze and bazel, it must be the case that the former can easily scale to this size of repo. I like to look at Google's GitHub commit messages to get an idea of the pace of their revision history. Yesterday they committed something with a Piper revision of 311324901. A month ago it was 306514102, and a year ago it was 248381230. That's about 160k revision numbers per day.
  - tehlike 6y ago
    Google monorepo is a productivity boon. Disclaimer: ex-googler.
  - rantwasp 6y ago
    huh. last i checked with the people that work there the tooling google has around managing the repo, build, test etc is insanely good. if you’re thinking you can do monorepos because google think again. you don’t have the tooling and you don’t even know what the tooling is
- elteto 6y ago
  But the number of targets is a function of _their_ code, not of using a monorepo, no? If you need to build N libraries and M executables then you are going to have N+M targets (assuming building for one arch only) whether you use a monorepo or not... Either way you are going to issue N+M build commands. Also, in bazel you can do bazel build //... to do a full build of all targets in a workspace. If they are not doing this but instead passing each target name individually then that probably means that they are only building a subset of everything, and even that subset is too much to pass in a single command line invocation. I'll grant you that it seems excessive to have that many targets, but again I don't see having so many targets as an explicit issue of the monorepo.
wgyn 6y ago
I wish they had spent more time describing how/why Go's built-in build tool stopped working. Or perhaps that's part of the basics of Bazel. Anyone able to share more on that?
- cflewis 6y ago
  The Go tool builds Go on a package by package basis. It won't help you with anything else. As soon as you need to build anything that isn't Go, or you need to do some pre/post-flight work, or you need to push artifacts etc etc etc the Go tool can't help. As the blog post notes, Makefiles are the usual weapon of choice for Go projects of any sufficient complexity, but these are quickly outgrown.
- jchw 6y ago
  Honestly the main reason Bazel is useful in a monorepo configuration is simply because it’s designed for it. It supports cross-language dependencies and generated files, and is designed for large repositories with many nested targets. Go’s build system is fine, but things like Go generate vs genrule probably come into play. Bazel also offers target visibility, to protect targets from being depended on in unintended ways, and build isolation, to allow builds to be more predictable (and help enforce target visibility.) It also has a few convenience features for vendoring 3rd party libraries and handling their licenses. Of course Bazel is far from perfect, in fact sometimes you may be better off running your own rules instead of the official ones in some cases, but IMO it makes a pretty decent build system for putting all of your code in one place. On the other hand, it’s one of those Google things where if you haven’t seen it in action in a functional configuration it’s hard to explain why it’s actually nice. And sadly I feel unsatisfied with the Node.JS rules for example, which is probably how a lot of people will first experience Bazel (since I believe Angular supports Bazel this way.)
- sleepydog 6y ago
  As bazel is not limited to Go it can be nicer when the Go programs or libraries are part of a more diverse system. For example, you can build a Go library, along with the rules to generate a protocol buffer package, along with the C library that the Go library interfaces with, along with some static assets that you want to compile into your program all from the same BUILD file, and are built from a single command. I think to do at least some of these things, if they are possible with the go tool at all, you would have to run 'go generate' first. Bazel also come with a query tool that makes it easy to tell what other targets your package depends on. Go also has such a tool, but again it's limited to Go dependencies. I believe Uber has been using bazel for awhile now, and probably were invested in it well before the Go module cache was a thing, so Bazel's build cache is an important optimization.
giacaglia 6y ago
Super interesting! There is also a video explaining their system that might be worth watching: https://vimeo.com/358691692 https://vimeo.com/358691692
filipn 6y ago
I've made a similar post about building a Go monorepo using Bazel two years ago, check it out - https://filipnikolovski.com/posts/managing-go-monorepo-with-bazel https://filipnikolovski.com/posts/managing-go-monorepo-with-.... We've been using Bazel not just to build and test our Go apps, but to build docker images, compile proto files, even deploy to k8s. It's really versatile and the developers only need to know one tool to build and test the whole environment.
- zahreeley 6y ago
  Your article is lot better than Uber’s blah blah type
blauditore 6y ago
> Out-of-the-box software solutions rarely work for a codebase as large and complex as Uber’s Go monorepo. That's a weird way to put it. Bazel is the open source variant of what Google uses for its monorepo, which is several orders of magnitude larger and arguably more complex.
- __s 6y ago
  Point is that take two sufficiently large monorepos & they'll have enough differences between them to require custom work That Google had to come up with Bazel for themselves only supports that point. It isn't obvious whether it'd then be applicable to other existing cases without customization Lots of knobs that vary here tho, such as where one draws the line between configuration & customization
- antoinealb 6y ago
  The version of Bazel we (Google) uses internally has a lot of features that are not out of the box in open source Bazel. Mostly around distributed compilation or object caching.
  - laurentlb 6y ago
    The public version also offers remote execution and remote caching. https://docs.bazel.build/versions/3.1.0/remote-execution.html https://docs.bazel.build/versions/3.1.0/remote-execution.htm... https://docs.bazel.build/versions/3.1.0/remote-caching.html https://docs.bazel.build/versions/3.1.0/remote-caching.html What's not open-sourced is mostly the interaction with Google internal infrastructure.
steeve 6y ago
We have been Go with Bazel for the last 2 1/2 years to build most of our backends, but also on mobile (gomobile) as well. It's not an easy tool, but it delivers amazingly and it's very, very reliable. Haven't run a bazel clean in... 6 months maybe? The multi-language really is a killer feature when the project inevitably becomes polyglot, such as when doing protobuf/gRPC or using CGo.
simfoo 6y ago
Might make sense if you use Go or any other language with a straightforward tooling landscape (compilers, package management etc.). That changes drastically if your codebase contains a lot of C++ and your SW model doesn't quite match the way Bazel tries to push you into. For instance doing any of the following things will quickly turn into a nightmare when using the Bazel C++ rules: * dynamically linking libraries * using a containerized approach for library includes (so transitive relative include paths) * using different toolchains and cross compiling * interfacing with thirdparty libraries We are talking seasoned build engineers ending up frustrated after literally months of trying to achieve something that is easy as pie in CMake. In addition there is still no real IDE support. The CLion plugin is permanently broken and lags behind versions. No real VSCode support. Using custom rules makes this even worse, to the point where CLion will refuse to sync and not having a way to produce a compilation database json. There are so many bugs open on those projects and no progress or answers. I can not recommend Bazel if C++ or C is what you care about.
- maybeiambatman 6y ago
  FWIW, Uber ATG uses the same monorepo culture with Bazel. It has for some time now. And most of the stuff is written in C++.
- q3k 6y ago
  > We are talking seasoned build engineers ending up frustrated after literally months of trying to achieve something that is easy as pie in CMake. But it's not the same as CMake. A properly set up Bazel project gives you so much more: organization-wide incremental builds, build cache and build farm. Also, actual full hermeticity of builds (no, taking ambient deps from a Docker container doesn't replace that). Comparing CMake to Bazel is like comparing some barely-working bash scripts on a single box to a kubernetes deployment. Maybe you're okay with just bash scripts, but some of us aren't, and that's where Bazel comes in.
  - simfoo 6y ago
    Completely agree that you can't compare Bazel and CMake directly, they are not on the same level. But for compiling C++ Bazel will have to compare itself to CMake, which is plain and simple the defacto standard. All the complex things that Bazel brings into the game (distributed builds and caches primarily) simply don't have any value when the main task that you're trying to make faster is not done well. I would much rather like to see more tooling that does these things but builds upon layers of battle-tested existing software.
quicklime 6y ago
Does anyone have experience using Bazel (or a similar build system) together with create-react-app? Specifically, is there a way do it without ejecting? I work on a project with a Go backend and a React frontend, and having to update all those React dependencies myself is what keeps me from moving to a system like Bazel.
quux 6y ago
Wait, didn't Uber decide that Bazel didn't scale enough (speaking as a xoogler, lol) and made their own build tool? Or am I misremembering?
vladaionescu 6y ago
If you also want to migrate off of Makefile and also want reproductibile builds, try out Earthly. Normal companies can't do Bazel because it's too alien and requires deep investment. https://github.com/earthly/earthly https://github.com/earthly/earthly Disclaimer: I am Earthly's creator.

158 points by dayanruben 6y ago

I wish my company was at the scale of Uber to work on all these kinds of tech. Makes me want to leave my current job just to go to Uber.

msoad 6y ago
Coordinating this sort of change in a large org like Uber is not entirely "hacking on cool tech". It's a lot of communication and writing documents and going to meetings
- save_ferris 6y ago
  This. It's very easy to forget about the political processes involved in making projects like this happen. Also, getting a job at a company as big as Uber in no way guarantees you'll be able to work on tech like this. You'd need to make sure you're joining a team close to this project, and if the culture of team turns out to be toxic, the fun of working on "cool" tech fades really, really fast.
- chimeracoder 6y ago
  > Coordinating this sort of change in a large org like Uber is not entirely "hacking on cool tech". It's a lot of communication and writing documents and going to meetings And a lot of tearing your hair out when trying to use tools in the ways that they weren't intended to be used. Of all languages, Go is not the language where you want to be doing things against the grain (that is, against the way the language designers intend it to be used). Some people like that about Go, and some people don't. Either way, Go is a very opinionated language with a very opinionated ecosystem, and ignoring those is a recipe for frustration. Last time I checked, Bazel was not recommended by the Go developers - and with good reason: there are a lot of gaps in rules_go/Gazelle, which this blog post alludes to but glosses over. While Google uses Blaze internally, Blaze is not Bazel, and the differences are very apparent to anyone who has used both to build Go specifically. Furthermore, rules_go was developed entirely independently of the Blaze ruleset that Google uses internally, so it's a tool that's not actually used internally at Google, but also not used by the majority of the non-Google Go developer community either, in addition to not being recommended by the Go team at Google[0]. [0] There is exactly one mention of Bazel on the entire golang.org domain - in a changelog from over two years ago, in an /x/ package, where Bazel is mentioned as one of two build systems in a "such as" clause that the new package could potentially enable support for (/x/ packages are considered experimental and not subject to the same backwards compatibility or maintenance guarantees as the rest of the project).
  - parsnips 6y ago
    From the issues I've read on github, anecdotally, is that the approach of the gazelle and rules_go teams is to leverage the standard go tooling and not make changes to go itself. Also anecdotally, bazel and golang work really well together IME. The community seems pretty active, and the upsides of using gazelle/bazel with golang seem to outweigh any downsides (though I'd be hard pressed to name a downside, that isn't inherit to golang itself).
    - chimeracoder 6y ago
      > Also anecdotally, bazel and golang work really well together IME... and the upsides of using gazelle/bazel with golang seem to outweigh any downsides This is really not my experience from having used Bazel with Go for the last four years. But I'm happy you are are apparently not running into issues.
      - parsnips 6y ago
        I've been using about the same amount of time. Have you tried using python and bazel? Now there's some real weeping and gnashing of teeth ;) Lot's of bad code depending on python == python2 type of non-sense.
        lhorie 6y ago
        Javascript is another target that is just about as gnarly. The node module resolution algorithm is reimplemented a million similar-but-not-quite-the-same-ways by popular packages, most of which struggle dealing w/ symlink-heavy codebases (like bazel sandboxes).
  - jeffbee 6y ago
    Having used both for years I honestly have not perceived the difference between blaze and bazel go_binary/go_library/go_test. What's the big difference?
  - malandrew 6y ago
    > Either way, Go is a very opinionated language with a very opinionated ecosystem except when it comes to dependency management
- 18nleung 6y ago
  We're not at the scale of Uber yet, but at Jupiter (YC S19) we're using tech like Bazel, protos, and Kotlin to prepare us for when we are: https://starship.jupiter.co/jupiter-stack/ https://starship.jupiter.co/jupiter-stack/ We're actively hiring for a senior SWE right now, so feel free to shoot me a note if you're looking.
  - yuribro 6y ago
    I'm not familiar with the Kotlin build chain, but what role does Bazel play? Is it needed for integration with the protocol buffers?
    - 18nleung 6y ago
      Yes - Bazel has excellent protobuf integration compared to other build tools (we were originally using Gradle). We're compiling Kotlin to the JVM, so the build chain is similar to Java's.
      - eleventigers 6y ago
        Gradle has a great protobuf plugin, what is so not excellent compared to Bazel?
  - novok 6y ago
    Protos is more of a faster & smaller network serialization tradeoff vs. ease of development thing than a company scale thing. And a thing to do if you want to integrate with gRPC. I worked at a company that used protos from day one for example. Many large companies are still using JSON w/ schemas as their network serialization layer just fine.
  - deleted 6y ago
    [deleted]
msrmthehomeless 6y ago
Don't wish, start leetcoding.
herrkanin 6y ago
I wish they weren't morally bankrupt as I would have loved to work on their tech stack.
- flak48 6y ago
  > morally bankrupt How so, in 2020?
  - morelisp 6y ago
    I think once you're stealing critics' medical records, your morally bankruptcy looks more like a chapter 7 than an 11. There's no expiry date; raze the whole thing.
lima 6y ago
Not just large companies - plenty of small startups building cool developer tooling. And much more fun when you get to start on a green field, rather than slowly migrating the existing code over the course of a decade.
kerng 6y ago
More politics, then tech.
klodolph 6y ago
Not personal experience, but I’ve had several coworkers who worked at Uber, and they all gave me the same story. This is the story: Uber reinvented a ton of technology because of the idea that off-the-shelf solutions wouldn’t work at “Uber scale”. However, there was very little accountability for whether the in-house solutions were necessary, and working on these tools would get you promoted. So for every “Uber scale” problem that a team actually solved, there were a couple other projects that were just half-baked alternatives to the off-the-shelf software that they should be using. It turns out that “Uber scale” is not really that large, despite the name. But engineers kept repeating “Uber scale” and building infrastructure. The same problem occurs at the larger tech companies like Google, Facebook, Microsoft, Amazon, and Apple, but in different ways and to different degrees. And to a large extent, engineers are copying what other companies do, and bringing ideas from one company to another when they hang out after work or switch jobs. For example, you can bet that these companies mostly have their own containerization and scheduling systems, many of which are undoubtedly not competitive with Docker or K8s in 2020, but K8s only goes back to 2014 and all these companies are older. I’m sure Borg and Tupperware are great if you work at Google or Facebook but I’m also sure that they’re missing a bunch of tooling that you’re used to. Same thing with build systems. Bazel, Buck, Pants, Please, and that Frankensteined system that Chrome uses are all copies of each other but Bazel is the only one with a decent size community and ecosystem, as far as I can tell. Uber is absolutely not on the scale of those companies, and most of the time they should probably be using off-the-shelf solutions when they become available.
- schoolornot 6y ago
  - nothing wrong with NIH syndrome if you have the means and resources to refine existing ideas and rebuild them with scale in mind - uber is a 50 billion dollar company and they have to de-risk themselves by owning entire stacks, top to bottom. if it means re-creating something from scratch... who cares, they have billions.
  - nickv 6y ago
    Or, at that point, you use industry standard solutions and hire the best subject matter experts on that solution. Like, for example, using Cockroach DB and hire one of the core committers cause you're big enough. It's what MSFT did with things like the Python extension for VSCode and DBOX did by hiring Guido since they're a Python shop. I still shake my head whenever I hear about how Uber built an entire Slack clone for "Uber-scale".
rvz 6y ago
Uber Engineering sounds like they are collecting Google's open source tools and name-dropping them in their blogs to market themselves as "cool tech". One companies boring tool is another companies favourite hyped third-party tech tool. (Until it doesn't work for them)
deleted 6y ago
[deleted]
wincent 6y ago
My experience working at Facebook was that the "working at scale" is nowhere near as glamorous as it sounds, especially when it comes to scaling build systems and monorepos. As a company, you have to invest immense resources in evolving the infrastructure and it is a never-ending task: the more capacity you build, the more your organization will grow in ways that tax that capacity. The goal posts are ever-receding. As an individual developer, all that fancy distributed infrastructure becomes a barrier that must be laboriously overcome at every step. Your expectations about what constitutes a "fast" operation get gradually distorted. Working on "small" side-projects (a few thousand files or less) feels effortless in comparison. Not to say you shouldn't seek out the "big tech" experience; just that it is not all roses.
xbeta 6y ago
They just had a laid-off recently, and still not profitable given how much $$$ they have burnt over the years. I am not sure if you wanna work there. FANG such as FB and Google has some big monorepos, and you can try that if you are interested.
q3k 6y ago
Scale is not a prerequisite, but you do have to sell it to your company. If not a greenfield, porting a codebase over is not easy, and you have to do your organizational excellence homework too. Monorepo, well reviewed code, team not in permanent crunch, high-quality tooling culture. Just like Kubernetes, Bazel is not a silver bullet. But if you get all your ducks in a row, switching over is a gamechanger and will help you out even at the smallest scale.
typon 6y ago
You want to join Uber because you think Bazel is cool tech? Having spent the last few weeks of my life converting our build system to Bazel I guarantee you it's far from what I would consider cool. It's more of a necessary evil because the alternatives are even worse.

WoahNoun 6y ago

This will surely bring them to profitability. Exactly the innovation the food delivery space was looking for.

tequila_shot 6y ago
Not sure if this is sarcasm. If not, how?
pm90 6y ago
I understand the snark but this is hacker news after all. Any kind of technical innovation is awesome to me as an HN reader even if it may not make business sense. Also, I’m totally fine with them using all that investment money for R&D rather than lobbying or returning to shareholders among other things.
- dmoy 6y ago
  I agree with you. When there was a SpaceX article about bazel, it'd be a bit unfortunate to see a top comment be something about an exploding rocket. (In reality, there were zero comments, so w/e)
- quest88 6y ago
  What about profitability? R&D is fine if they're profitable, but they're not.
  - bhupy 6y ago
    Uber doesn't need to be profitable yet, they have $10B in cash. While this doesn't mean that they can continue to be unprofitable forever, it means that they have latitude to engage in R&D (like this post) while also moving towards profitability, at the same time.
    - lhorie 6y ago
      Adopting an open source build system doesn't exactly strike me as R&D

chetanbhasin 6y ago

This is excellent! The only reason we haven't moved some of our Go codebase to a Bazel monorepo is the IDE integration. I have been tinkering with a few ideas to make the existing tooling work with Bazel, but the effort is larger than I had originally expected.

DavyJone 6y ago

That seems like a ridiculously complex pipeline to maintain and onboard into. I personally dislike this "do it all" monorepos in most cases, there are cases were they work, but they also break a lot of other things. I have not seen metris or blogs that prove this "uptick in build efficiency" or an increase productivity. While I do like the idea behind bazel, I hate repeating deps in things like "go_repository" with gazelle.

parsnips 6y ago
go mod vendor && bazel run gazelle No go_repository duplication.

agounaris 6y ago

The article says that monorepos are more efficient but also that "As the monorepo grew, the build target list increased to a point where it became too long to pass it through Bazel’s command line interface.". Monorepos are not efficient. They are easier to manage when a team is small but as the team grows and you have more and more deliverables with separate versioning you are introducing control structures in your automation. Complexity explodes! Anyway, all this does matter if you don't make any profit :)

luckydata 6y ago
Google is a monorepo as far as I know, and they are doing fine.
- jeffbee 6y ago
  Indeed, in their paper from 5 years ago Google claimed 300000 commits per day across 9 million source files, compared to this article claiming 10000 commits per month on 70000 source files at Uber. Whatever the differences are between blaze and bazel, it must be the case that the former can easily scale to this size of repo. I like to look at Google's GitHub commit messages to get an idea of the pace of their revision history. Yesterday they committed something with a Piper revision of 311324901. A month ago it was 306514102, and a year ago it was 248381230. That's about 160k revision numbers per day.
- tehlike 6y ago
  Google monorepo is a productivity boon. Disclaimer: ex-googler.
- rantwasp 6y ago
  huh. last i checked with the people that work there the tooling google has around managing the repo, build, test etc is insanely good. if you’re thinking you can do monorepos because google think again. you don’t have the tooling and you don’t even know what the tooling is
elteto 6y ago
But the number of targets is a function of _their_ code, not of using a monorepo, no? If you need to build N libraries and M executables then you are going to have N+M targets (assuming building for one arch only) whether you use a monorepo or not... Either way you are going to issue N+M build commands. Also, in bazel you can do bazel build //... to do a full build of all targets in a workspace. If they are not doing this but instead passing each target name individually then that probably means that they are only building a subset of everything, and even that subset is too much to pass in a single command line invocation. I'll grant you that it seems excessive to have that many targets, but again I don't see having so many targets as an explicit issue of the monorepo.

wgyn 6y ago

I wish they had spent more time describing how/why Go's built-in build tool stopped working. Or perhaps that's part of the basics of Bazel. Anyone able to share more on that?

cflewis 6y ago
The Go tool builds Go on a package by package basis. It won't help you with anything else. As soon as you need to build anything that isn't Go, or you need to do some pre/post-flight work, or you need to push artifacts etc etc etc the Go tool can't help. As the blog post notes, Makefiles are the usual weapon of choice for Go projects of any sufficient complexity, but these are quickly outgrown.
jchw 6y ago
Honestly the main reason Bazel is useful in a monorepo configuration is simply because it’s designed for it. It supports cross-language dependencies and generated files, and is designed for large repositories with many nested targets. Go’s build system is fine, but things like Go generate vs genrule probably come into play. Bazel also offers target visibility, to protect targets from being depended on in unintended ways, and build isolation, to allow builds to be more predictable (and help enforce target visibility.) It also has a few convenience features for vendoring 3rd party libraries and handling their licenses. Of course Bazel is far from perfect, in fact sometimes you may be better off running your own rules instead of the official ones in some cases, but IMO it makes a pretty decent build system for putting all of your code in one place. On the other hand, it’s one of those Google things where if you haven’t seen it in action in a functional configuration it’s hard to explain why it’s actually nice. And sadly I feel unsatisfied with the Node.JS rules for example, which is probably how a lot of people will first experience Bazel (since I believe Angular supports Bazel this way.)
sleepydog 6y ago
As bazel is not limited to Go it can be nicer when the Go programs or libraries are part of a more diverse system. For example, you can build a Go library, along with the rules to generate a protocol buffer package, along with the C library that the Go library interfaces with, along with some static assets that you want to compile into your program all from the same BUILD file, and are built from a single command. I think to do at least some of these things, if they are possible with the go tool at all, you would have to run 'go generate' first. Bazel also come with a query tool that makes it easy to tell what other targets your package depends on. Go also has such a tool, but again it's limited to Go dependencies. I believe Uber has been using bazel for awhile now, and probably were invested in it well before the Go module cache was a thing, so Bazel's build cache is an important optimization.

giacaglia 6y ago

Super interesting! There is also a video explaining their system that might be worth watching: https://vimeo.com/358691692 https://vimeo.com/358691692

filipn 6y ago

I've made a similar post about building a Go monorepo using Bazel two years ago, check it out - https://filipnikolovski.com/posts/managing-go-monorepo-with-bazel https://filipnikolovski.com/posts/managing-go-monorepo-with-.... We've been using Bazel not just to build and test our Go apps, but to build docker images, compile proto files, even deploy to k8s. It's really versatile and the developers only need to know one tool to build and test the whole environment.

zahreeley 6y ago
Your article is lot better than Uber’s blah blah type

blauditore 6y ago

> Out-of-the-box software solutions rarely work for a codebase as large and complex as Uber’s Go monorepo. That's a weird way to put it. Bazel is the open source variant of what Google uses for its monorepo, which is several orders of magnitude larger and arguably more complex.

__s 6y ago
Point is that take two sufficiently large monorepos & they'll have enough differences between them to require custom work That Google had to come up with Bazel for themselves only supports that point. It isn't obvious whether it'd then be applicable to other existing cases without customization Lots of knobs that vary here tho, such as where one draws the line between configuration & customization
antoinealb 6y ago
The version of Bazel we (Google) uses internally has a lot of features that are not out of the box in open source Bazel. Mostly around distributed compilation or object caching.
- laurentlb 6y ago
  The public version also offers remote execution and remote caching. https://docs.bazel.build/versions/3.1.0/remote-execution.html https://docs.bazel.build/versions/3.1.0/remote-execution.htm... https://docs.bazel.build/versions/3.1.0/remote-caching.html https://docs.bazel.build/versions/3.1.0/remote-caching.html What's not open-sourced is mostly the interaction with Google internal infrastructure.

steeve 6y ago

We have been Go with Bazel for the last 2 1/2 years to build most of our backends, but also on mobile (gomobile) as well. It's not an easy tool, but it delivers amazingly and it's very, very reliable. Haven't run a bazel clean in... 6 months maybe? The multi-language really is a killer feature when the project inevitably becomes polyglot, such as when doing protobuf/gRPC or using CGo.

simfoo 6y ago

Might make sense if you use Go or any other language with a straightforward tooling landscape (compilers, package management etc.). That changes drastically if your codebase contains a lot of C++ and your SW model doesn't quite match the way Bazel tries to push you into. For instance doing any of the following things will quickly turn into a nightmare when using the Bazel C++ rules: * dynamically linking libraries * using a containerized approach for library includes (so transitive relative include paths) * using different toolchains and cross compiling * interfacing with thirdparty libraries We are talking seasoned build engineers ending up frustrated after literally months of trying to achieve something that is easy as pie in CMake. In addition there is still no real IDE support. The CLion plugin is permanently broken and lags behind versions. No real VSCode support. Using custom rules makes this even worse, to the point where CLion will refuse to sync and not having a way to produce a compilation database json. There are so many bugs open on those projects and no progress or answers. I can not recommend Bazel if C++ or C is what you care about.

maybeiambatman 6y ago
FWIW, Uber ATG uses the same monorepo culture with Bazel. It has for some time now. And most of the stuff is written in C++.
q3k 6y ago
> We are talking seasoned build engineers ending up frustrated after literally months of trying to achieve something that is easy as pie in CMake. But it's not the same as CMake. A properly set up Bazel project gives you so much more: organization-wide incremental builds, build cache and build farm. Also, actual full hermeticity of builds (no, taking ambient deps from a Docker container doesn't replace that). Comparing CMake to Bazel is like comparing some barely-working bash scripts on a single box to a kubernetes deployment. Maybe you're okay with just bash scripts, but some of us aren't, and that's where Bazel comes in.
- simfoo 6y ago
  Completely agree that you can't compare Bazel and CMake directly, they are not on the same level. But for compiling C++ Bazel will have to compare itself to CMake, which is plain and simple the defacto standard. All the complex things that Bazel brings into the game (distributed builds and caches primarily) simply don't have any value when the main task that you're trying to make faster is not done well. I would much rather like to see more tooling that does these things but builds upon layers of battle-tested existing software.

quicklime 6y ago

Does anyone have experience using Bazel (or a similar build system) together with create-react-app? Specifically, is there a way do it without ejecting? I work on a project with a Go backend and a React frontend, and having to update all those React dependencies myself is what keeps me from moving to a system like Bazel.

quux 6y ago

Wait, didn't Uber decide that Bazel didn't scale enough (speaking as a xoogler, lol) and made their own build tool? Or am I misremembering?

vladaionescu 6y ago

If you also want to migrate off of Makefile and also want reproductibile builds, try out Earthly. Normal companies can't do Bazel because it's too alien and requires deep investment. https://github.com/earthly/earthly https://github.com/earthly/earthly Disclaimer: I am Earthly's creator.