3 ms·

> company with a monorepo that has to build and test everything in CI for every change That smells like covering up an architecture problem (tight coupling) wi

> company with a monorepo that has to build and test everything in CI for every change That smells like covering up an architecture problem (tight coupling) with a build tool. Not a recipe for success.

deathanatos 4y ago
I don't think that's actually the case. Bazel, AIUI, understands the entire dependency tree and can cache steps effectively, which a Makefile cannot (alone) do. (You could certainly build it into one.) CI tooling is orthogonal; your CI tooling can invoke Bazel or Makefile+custom caching, it wouldn't really matter. But the problem in CI is that you lack state; whereas a developer's machine might e.g., have a bunch of .o files from a previous build, CI will be starting from a clean slate¹. So builds take longer. My understanding of Bazel is that it has decent support for caching and can thus makes those rebuilds a lot faster, which it has b/c it understands exactly the inputs to the steps at hand. (Which, if you write your own caching layer, you'd have to figure out.) … but … I've also seen the same thing the top commenter has, with Bazel: endless suggestions to use Bazel … but migrating a large existing codebase is pain, and if you're not willing to put your work where your mouth is … then the status quo prevails. ¹While here CI lacking state is a sort of negative b/c of the time we need to rebuild it, normally, I think this is a good thing: it means you're constantly verifying you can build from scratch. Or, at least, something close to scratch, modulo things like vendoring. (I.e., if you do an "apt-get" from Ubuntu's servers in your CI pipeline … well … your build obviously isn't quite from scratch.)
- JustLurking2022 4y ago
  Having experience FAANG monorepos, the other commenter was entirely correct about them papering over tight coupling with tools. It makes more sense when you realize it was really built to deal with statically linked C++ - there's no guarantee of ABI, so you really need to ensure that all your dependencies are built the same way as your project to ensure compatibility (and the dependency graphs in monorepos get DEEP). For any language with a stable ABI that utilizes dynamic linking, the complexity is far more then the benefit. The whole monorepo concept and all the tooling that goes with it really makes more sense when you think of it in that context of statically linked code that can't easily be packaged into something as portable as a JAR, so it's critical to build it all the same way.
hot_gril 4y ago
Google uses a monorepo intentionally, and for good reasons. They're also a massive company. For them, a complex build system makes sense... and even that system isn't exactly Bazel.
- Conan_Kudo 4y ago
  Google uses a monorepo because they were popular in 1999, not because they had a specific reason to do so[1]. As the codebase grew, they never changed and instead built tools around it. That's a perfectly fine choice if you have bodies to throw at the problem and can deal with the consequences of that choice, but most people can't. Google can and so it does. If it wasn't as successful as it is, it probably would have collapsed under the strain of the people and systems required to manage that code. [1]: https://qeunit.com/blog/how-google-does-monorepo/ https://qeunit.com/blog/how-google-does-monorepo/
  - hot_gril 4y ago
    Ok, could've been chosen for 1999 reasons back then, but there are good reasons today. It's easier to build things collaboratively, esp across teams, when there's just one repo and one branch (I don't count release branches). Yes it takes effort to support due to its size, but that's contained within a few teams, and everyone else benefits from it. Other tooling decisions place a burden on every SWE. Their config language is a weird trash that everyone has to sorta learn. Their deployment and integration testing tooling have similar qualities, and it's not cause of the monorepo. One consequence is that many teams cannot feasibly deploy microservices, even though it'd often make sense to, which is even worse if they're using C++.

deathanatos 4y ago

I don't think that's actually the case. Bazel, AIUI, understands the entire dependency tree and can cache steps effectively, which a Makefile cannot (alone) do. (You could certainly build it into one.) CI tooling is orthogonal; your CI tooling can invoke Bazel or Makefile+custom caching, it wouldn't really matter. But the problem in CI is that you lack state; whereas a developer's machine might e.g., have a bunch of .o files from a previous build, CI will be starting from a clean slate¹. So builds take longer. My understanding of Bazel is that it has decent support for caching and can thus makes those rebuilds a lot faster, which it has b/c it understands exactly the inputs to the steps at hand. (Which, if you write your own caching layer, you'd have to figure out.) … but … I've also seen the same thing the top commenter has, with Bazel: endless suggestions to use Bazel … but migrating a large existing codebase is pain, and if you're not willing to put your work where your mouth is … then the status quo prevails. ¹While here CI lacking state is a sort of negative b/c of the time we need to rebuild it, normally, I think this is a good thing: it means you're constantly verifying you can build from scratch. Or, at least, something close to scratch, modulo things like vendoring. (I.e., if you do an "apt-get" from Ubuntu's servers in your CI pipeline … well … your build obviously isn't quite from scratch.)

JustLurking2022 4y ago
Having experience FAANG monorepos, the other commenter was entirely correct about them papering over tight coupling with tools. It makes more sense when you realize it was really built to deal with statically linked C++ - there's no guarantee of ABI, so you really need to ensure that all your dependencies are built the same way as your project to ensure compatibility (and the dependency graphs in monorepos get DEEP). For any language with a stable ABI that utilizes dynamic linking, the complexity is far more then the benefit. The whole monorepo concept and all the tooling that goes with it really makes more sense when you think of it in that context of statically linked code that can't easily be packaged into something as portable as a JAR, so it's critical to build it all the same way.

hot_gril 4y ago

Google uses a monorepo intentionally, and for good reasons. They're also a massive company. For them, a complex build system makes sense... and even that system isn't exactly Bazel.

Conan_Kudo 4y ago
Google uses a monorepo because they were popular in 1999, not because they had a specific reason to do so[1]. As the codebase grew, they never changed and instead built tools around it. That's a perfectly fine choice if you have bodies to throw at the problem and can deal with the consequences of that choice, but most people can't. Google can and so it does. If it wasn't as successful as it is, it probably would have collapsed under the strain of the people and systems required to manage that code. [1]: https://qeunit.com/blog/how-google-does-monorepo/ https://qeunit.com/blog/how-google-does-monorepo/
- hot_gril 4y ago
  Ok, could've been chosen for 1999 reasons back then, but there are good reasons today. It's easier to build things collaboratively, esp across teams, when there's just one repo and one branch (I don't count release branches). Yes it takes effort to support due to its size, but that's contained within a few teams, and everyone else benefits from it. Other tooling decisions place a burden on every SWE. Their config language is a weird trash that everyone has to sorta learn. Their deployment and integration testing tooling have similar qualities, and it's not cause of the monorepo. One consequence is that many teams cannot feasibly deploy microservices, even though it'd often make sense to, which is even worse if they're using C++.