4 ms·

What problems did you encounter with just a few services? Monorepos should be straightforward unless you are managing the code of >1k engineers.

gen220 7y ago
We’ve run into some nontrivial but totally solvable issues at about 100-200 engineers. IME, most consternation comes from people adopting a mono repo without adopting a build/dependency graph tool (like Bazel, buck or pants). An additional source of strain is from people abusing the repo (checking in large binaries, third party dependencies, etc). A third is when people try to do branch-based feature development, instead of the “correct” practice of only deploying master (or weekly cuts of master). I think even a simple list of these sort of “gotchas” would be valuable for the aspirational mono repo company. My impression is that a lot of teams hit these early and painful roadblocks, and imagine that they’ll never go away (they do!!).
- Hackbraten 7y ago
  Checking in third-party dependencies is not always abuse. It can be a useful habit for certain kinds of reproducible builds. The Buck documentation even endorses keeping your dependencies in your monorepo along with your own sources.
  - gen220 7y ago
    I understand the reasoning, and agree that it’s not always abuse. At first blush it’s a good idea, but I’d maintain that it’s one of the things that balloons your repo size quite quickly. Plus, one have to draw a line somewhere on what to include (a Python interpreter? A Go version? awk and grep?), and third party vs in-house is a fairly robust one imo. We host a private mirror for third party dependencies, so that “pip install”/“go get” fail on our CI system if the dependency isn’t hosted by us. This gives us reproducible builds, while allowing us to hold 3rd party libraries to a higher standard of entry than source code. For certain libraries we pin version numbers in our build system, but in general it allows us to update dependencies transparently. It also keeps our source repo size small, for developers, and allows for conflicting versions (example Kafka X.Y and X.Z) without cluttering the repo with duplicates. It’s definitely a smaller gotcha than the others I listed, maybe to the point where it’s not a gotcha, but I stand by it :)
    - peterwwillis 7y ago
      If you can do that with 3rd party dependencies, can't you do that with all the code? This is what confuses me about monorepos. Their design requires an array of confusing processes and complex software to make the process of merging, testing, and releasing code manageable at scale (and "scale" can even be 6 developers working on 2 separate features each across 10 services, in one repo). But it turns out that you can also develop individual components, version their releases, link their dependencies, and still have a usable system. That's literally how all Linux distros have worked for decades, and how most other language-specific packaging systems work. None of which requires a monorepo. So what I'd like to know is, of the 3 actual reasons I've heard companies claim are why they need a monorepo, is it impossible to do these things with multirepo? If it is indeed "hard" to do, is it "so hard" that it justifies all the complexity inherent to the monorepo? Or is it really just a meme? And are these things even necessary at all, if other systems seem to get away without it?
      - gen220 7y ago
        These are great questions!! :) > Can you treat all code like 3rd party dependencies? Yes, but there are trade-offs. Discoverability, enforcing hard deadlines on global changes, style consistency, etc. > Is it impossible to do these things with multi-repo? No, but there are trade-offs to consider. > If it's hard, is it "so hard" that it justifies the complexity? Hitting the nail on the head; there are trade-offs :) > Are these things necessary, if other systems get away without it? There are many stable equilibria; open source ecosystem evolved one solution and large companies evolved another, because they have been subject to very different constraints. The organization of the open source projects is extremely different from the organization of 100+ engineer companies, even if the contributor headcounts are similar. For me, the the semantic distinction between monorepos and multirepos is the same as the distinction between internal and 3rd party dependencies. Does your team want to treat other teams as a 3rd party dependency? The correct answer depends on company culture, etc. It's a set of tradeoffs, including transparency over privacy, consistency over freedom, collaboration over compartmentalization. With monorepos, you can gain a little privacy, freedom, and compartmentalization by being clever, but get the rest for cheap; vice versa for multirepos. It's trading one set of problems for another. I'd challenge the base assumption that multirepos are "simpler", they're just more tolerant of chaos, in a way that's very valuable for the open source community. I hope we've not been talking past each other, I really like the ideas your raising! :)
- gdxhyrd 7y ago
  > IME, most consternation comes from people adopting a mono repo without adopting a build/dependency graph tool (like Bazel, buck or pants). That seems like a build problem, not a Git problem. > An additional source of strain is from people abusing the repo (checking in large binaries, third party dependencies, etc). That is not necessarily abuse. In fact, it is a good practice in many cases! > A third is when people try to do branch-based feature development, instead of the “correct” practice of only deploying master (or weekly cuts of master). I am not sure what you mean by branch-based development, but I don't see why that would be a specific problem of monorepos.
peterwwillis 7y ago
How are they straightforward? Like rebuilding a car's engine is straightforward? If you know how they're built, it's easy...
- gdxhyrd 7y ago
  What? I don't understand what that means. A monorepo is just 1 repo. There is nothing more straightforward than that.

gen220 7y ago

We’ve run into some nontrivial but totally solvable issues at about 100-200 engineers. IME, most consternation comes from people adopting a mono repo without adopting a build/dependency graph tool (like Bazel, buck or pants). An additional source of strain is from people abusing the repo (checking in large binaries, third party dependencies, etc). A third is when people try to do branch-based feature development, instead of the “correct” practice of only deploying master (or weekly cuts of master). I think even a simple list of these sort of “gotchas” would be valuable for the aspirational mono repo company. My impression is that a lot of teams hit these early and painful roadblocks, and imagine that they’ll never go away (they do!!).

Hackbraten 7y ago
Checking in third-party dependencies is not always abuse. It can be a useful habit for certain kinds of reproducible builds. The Buck documentation even endorses keeping your dependencies in your monorepo along with your own sources.
- gen220 7y ago
  I understand the reasoning, and agree that it’s not always abuse. At first blush it’s a good idea, but I’d maintain that it’s one of the things that balloons your repo size quite quickly. Plus, one have to draw a line somewhere on what to include (a Python interpreter? A Go version? awk and grep?), and third party vs in-house is a fairly robust one imo. We host a private mirror for third party dependencies, so that “pip install”/“go get” fail on our CI system if the dependency isn’t hosted by us. This gives us reproducible builds, while allowing us to hold 3rd party libraries to a higher standard of entry than source code. For certain libraries we pin version numbers in our build system, but in general it allows us to update dependencies transparently. It also keeps our source repo size small, for developers, and allows for conflicting versions (example Kafka X.Y and X.Z) without cluttering the repo with duplicates. It’s definitely a smaller gotcha than the others I listed, maybe to the point where it’s not a gotcha, but I stand by it :)
  - peterwwillis 7y ago
    If you can do that with 3rd party dependencies, can't you do that with all the code? This is what confuses me about monorepos. Their design requires an array of confusing processes and complex software to make the process of merging, testing, and releasing code manageable at scale (and "scale" can even be 6 developers working on 2 separate features each across 10 services, in one repo). But it turns out that you can also develop individual components, version their releases, link their dependencies, and still have a usable system. That's literally how all Linux distros have worked for decades, and how most other language-specific packaging systems work. None of which requires a monorepo. So what I'd like to know is, of the 3 actual reasons I've heard companies claim are why they need a monorepo, is it impossible to do these things with multirepo? If it is indeed "hard" to do, is it "so hard" that it justifies all the complexity inherent to the monorepo? Or is it really just a meme? And are these things even necessary at all, if other systems seem to get away without it?
    - gen220 7y ago
      These are great questions!! :) > Can you treat all code like 3rd party dependencies? Yes, but there are trade-offs. Discoverability, enforcing hard deadlines on global changes, style consistency, etc. > Is it impossible to do these things with multi-repo? No, but there are trade-offs to consider. > If it's hard, is it "so hard" that it justifies the complexity? Hitting the nail on the head; there are trade-offs :) > Are these things necessary, if other systems get away without it? There are many stable equilibria; open source ecosystem evolved one solution and large companies evolved another, because they have been subject to very different constraints. The organization of the open source projects is extremely different from the organization of 100+ engineer companies, even if the contributor headcounts are similar. For me, the the semantic distinction between monorepos and multirepos is the same as the distinction between internal and 3rd party dependencies. Does your team want to treat other teams as a 3rd party dependency? The correct answer depends on company culture, etc. It's a set of tradeoffs, including transparency over privacy, consistency over freedom, collaboration over compartmentalization. With monorepos, you can gain a little privacy, freedom, and compartmentalization by being clever, but get the rest for cheap; vice versa for multirepos. It's trading one set of problems for another. I'd challenge the base assumption that multirepos are "simpler", they're just more tolerant of chaos, in a way that's very valuable for the open source community. I hope we've not been talking past each other, I really like the ideas your raising! :)
gdxhyrd 7y ago
> IME, most consternation comes from people adopting a mono repo without adopting a build/dependency graph tool (like Bazel, buck or pants). That seems like a build problem, not a Git problem. > An additional source of strain is from people abusing the repo (checking in large binaries, third party dependencies, etc). That is not necessarily abuse. In fact, it is a good practice in many cases! > A third is when people try to do branch-based feature development, instead of the “correct” practice of only deploying master (or weekly cuts of master). I am not sure what you mean by branch-based development, but I don't see why that would be a specific problem of monorepos.

peterwwillis 7y ago

How are they straightforward? Like rebuilding a car's engine is straightforward? If you know how they're built, it's easy...

gdxhyrd 7y ago
What? I don't understand what that means. A monorepo is just 1 repo. There is nothing more straightforward than that.