3 ms·

As a few others have mentioned this is something that build systems handle since they understand the dependency graph. For example, Bazel is often used to this

As a few others have mentioned this is something that build systems handle since they understand the dependency graph. For example, Bazel is often used to this end. However... I would strongly advise not going for a monorepo. No, I don't mean something like tensorflow where you have a bunch of related tools and projects in a single repo. I mean one repo for the entire org where totally unrelated projects live. Every company I've been at that used a monorepo found themselves struggling to make it work since you need a ton of full time engineers just to keep things working and scaling. Many of the problems that monorepos try to solve (simplifying dependency and version management) are traded for 10x as many problems and many of them are hard (incremental builds, dependency resolution). Google has a huge team in charge of helping their monorepo scale and work efficiently. You are not google... don't be tempted.

benreesman 6y ago
I'll try to tread at least a little lightly here because this topic does tend to be a bit flammable, but caveat emptor. My contrasting anecdotal experience is that whether at BigCo or on a small team monorepo is almost always the right answer until your requirements get exotic enough that you're in special-case land anyways (like a separate repo for machine-initiated commits, or something that's security-sensitive enough to wall off some contributors). Both `git` and `hg` scale easily to to really big projects if you're storing text in them (at FB our C++ code was in a `git` monorepo on stock software until like 2014 or something before it started bogging down, I'll gloss over the numbers but, big): the monorepo-scaling argument is brought out a lot but rarely quantified. The multi-repo problem that gets you is dependency management, which in the general case requires a SAT-solver (https://research.swtch.com/version-sat https://research.swtch.com/version-sat), but of course you don't have a SAT-solver in your build script for your small-to-medium organization, so you get some half-assed thing like what `pip` and `npm` do. Again purely anecdotal, but in my personal experience multi-repo too often gets pushed by folks who want to make their own rules for part of the codebase ("the braces go here"), push an agenda around unnecessary RPCs, or both. That's not true of all cases of course, but it's a common enough antipattern to be memorable.
adsfoiu1 6y ago
I personally have seen the opposite problem - the friction of making small changes to "utility" libraries becomes a huge pain point for developers when you have to make changes, test locally, push to package manager, update all consumers to use the new version... It's much easier, in my experience, to just consume a class that's already in the same project / repo.
- agency 6y ago
  I have also experienced this pain where a company I worked for went too hard on splitting every thing into separate repos, such that updating something deep in the dependency tree becomes very painful and involves a protracted "version bump dance" on dependent repos. There's no silver bullet here.
TechBro8615 6y ago
> Google has a huge team in charge of helping their monorepo scale and work efficiently. You are not google... don't be tempted. It’s funny, I’ve heard this exact same argument for why you should not use micro services.
jayd16 6y ago
"You are not google" is also an argument for why you don't have to worry about scaling a monorepo.
brown9-2 6y ago
The other half of needing to use a build system that understands the dependency graph like Bazel is that Bazel _keeps state_, so that it knows which part of the graph does not need to be re-built when you push commit B because it was already built in commit A.
jschwartzi 6y ago
Well, sure. if you have a pile of totally unrelated things that never need to change in lock-step, then you don't need a "monorepo." But on the other hand if you're building an entire software system such as a collection of API services, a database schema, embedded device firmware, and a website, and all of these things are interdependent and incompatible across versions then please for the love of god use a monorepo. At my job our cloud team uses multiple separate repositories which makes sense, but it also moves the burden of versioning to run-time. This is because they have to interface with multiple different versions of the device firmware. So they deploy different run-time versions of the APIs to support legacy and current production firmware versions. But our firmware repository is a monorepo in that the sources and build system builds the artifacts for multiple devices from the same source tree. So it's not so cut and dried as "never use a monorepo" or "always use a monorepo." It involves engineering tradeoffs and decisions that are made in a context, and you can't extract your advice from the context in which it exists. What works for our cloud team would be a terrible mess on the embedded side simply because of how the software is deployed and managed.
benjyw 6y ago
There are definitely pros and cons to either approach. My personal preference/bias is a monorepo architecture, even for smaller teams and orgs. Working effectively in a monorepo does require suitable tooling (full disclosure: I am a core contributor to Pants, a monorepo build system with, currently, a Python focus). But with that tooling in place a monorepo can make it a lot easier to collaborate, to manage changes and dependencies, and to avoid balkanization and fiefs. Granted there are no silver bullets, but we are working to make tooling good enough to avoid needing the large in-house support team you allude to. Having a large number of small repos can work if they don't have a lot of interdependencies, and maybe this is what you mean by "totally unrelated projects". But if there are significant interdependencies (and repos usually end up this way, in my experience, which may or may not be representative) things become very brittle, and it's very hard to make changes and reason about how they affect downstream code, not to mention the challenge of version resolution, aka "dependency hell". In the end, managing a large codebase, whether it follows a monorepo or multi-repo architecture, requires suitable tooling. On balance I think that monorepo is often a better choice, but YMMV for sure.

benreesman 6y ago

I'll try to tread at least a little lightly here because this topic does tend to be a bit flammable, but caveat emptor. My contrasting anecdotal experience is that whether at BigCo or on a small team monorepo is almost always the right answer until your requirements get exotic enough that you're in special-case land anyways (like a separate repo for machine-initiated commits, or something that's security-sensitive enough to wall off some contributors). Both `git` and `hg` scale easily to to really big projects if you're storing text in them (at FB our C++ code was in a `git` monorepo on stock software until like 2014 or something before it started bogging down, I'll gloss over the numbers but, big): the monorepo-scaling argument is brought out a lot but rarely quantified. The multi-repo problem that gets you is dependency management, which in the general case requires a SAT-solver (https://research.swtch.com/version-sat https://research.swtch.com/version-sat), but of course you don't have a SAT-solver in your build script for your small-to-medium organization, so you get some half-assed thing like what `pip` and `npm` do. Again purely anecdotal, but in my personal experience multi-repo too often gets pushed by folks who want to make their own rules for part of the codebase ("the braces go here"), push an agenda around unnecessary RPCs, or both. That's not true of all cases of course, but it's a common enough antipattern to be memorable.

adsfoiu1 6y ago

I personally have seen the opposite problem - the friction of making small changes to "utility" libraries becomes a huge pain point for developers when you have to make changes, test locally, push to package manager, update all consumers to use the new version... It's much easier, in my experience, to just consume a class that's already in the same project / repo.

agency 6y ago
I have also experienced this pain where a company I worked for went too hard on splitting every thing into separate repos, such that updating something deep in the dependency tree becomes very painful and involves a protracted "version bump dance" on dependent repos. There's no silver bullet here.

TechBro8615 6y ago

> Google has a huge team in charge of helping their monorepo scale and work efficiently. You are not google... don't be tempted. It’s funny, I’ve heard this exact same argument for why you should not use micro services.

jayd16 6y ago

"You are not google" is also an argument for why you don't have to worry about scaling a monorepo.

brown9-2 6y ago

The other half of needing to use a build system that understands the dependency graph like Bazel is that Bazel _keeps state_, so that it knows which part of the graph does not need to be re-built when you push commit B because it was already built in commit A.

jschwartzi 6y ago

Well, sure. if you have a pile of totally unrelated things that never need to change in lock-step, then you don't need a "monorepo." But on the other hand if you're building an entire software system such as a collection of API services, a database schema, embedded device firmware, and a website, and all of these things are interdependent and incompatible across versions then please for the love of god use a monorepo. At my job our cloud team uses multiple separate repositories which makes sense, but it also moves the burden of versioning to run-time. This is because they have to interface with multiple different versions of the device firmware. So they deploy different run-time versions of the APIs to support legacy and current production firmware versions. But our firmware repository is a monorepo in that the sources and build system builds the artifacts for multiple devices from the same source tree. So it's not so cut and dried as "never use a monorepo" or "always use a monorepo." It involves engineering tradeoffs and decisions that are made in a context, and you can't extract your advice from the context in which it exists. What works for our cloud team would be a terrible mess on the embedded side simply because of how the software is deployed and managed.

benjyw 6y ago

There are definitely pros and cons to either approach. My personal preference/bias is a monorepo architecture, even for smaller teams and orgs. Working effectively in a monorepo does require suitable tooling (full disclosure: I am a core contributor to Pants, a monorepo build system with, currently, a Python focus). But with that tooling in place a monorepo can make it a lot easier to collaborate, to manage changes and dependencies, and to avoid balkanization and fiefs. Granted there are no silver bullets, but we are working to make tooling good enough to avoid needing the large in-house support team you allude to. Having a large number of small repos can work if they don't have a lot of interdependencies, and maybe this is what you mean by "totally unrelated projects". But if there are significant interdependencies (and repos usually end up this way, in my experience, which may or may not be representative) things become very brittle, and it's very hard to make changes and reason about how they affect downstream code, not to mention the challenge of version resolution, aka "dependency hell". In the end, managing a large codebase, whether it follows a monorepo or multi-repo architecture, requires suitable tooling. On balance I think that monorepo is often a better choice, but YMMV for sure.