12 ms·

I've worked with both a distributed repo model and a monorepo model and vastly prefer the distributed approach (given the right tooling). The trade-offs are com

I've worked with both a distributed repo model and a monorepo model and vastly prefer the distributed approach (given the right tooling). The trade-offs are complementary and no doubt with proper discipline you can try to maximize the benefits, while minimizing the downside. But here's what I don't like about working in a large monorepo: 1) Difficult to track changes to the code I'm interested in. Every day there are hundreds of changes in the repo and almost all of them have nothing to do with what I'm working on. 2) all sorts of operations take longer (pulling, grepping source, etc.) to support code I couldn't care less about. 3) Frequently have to update the world at once. Unless the repo can store multiple versions of the same module, then all the consumers have to be updated at once, even if it's inconvenient. Sometimes migrations are better done gradually. 4) Encourages sloppy dependency management. There are frequently unclear boundaries between software layers. I'm sure people will say "if you're having those problems, you're doing it wrong" but the same thing could be said to people who find the distributed model problematic.

Sacho 9y ago
> Every day there are hundreds of changes in the repo and almost all of them have nothing to do with what I'm working on. Wouldn't the right tooling be able to show you changes to the slice of code you're interested in? I remember SVN would allow you to checkout just a single subdirectory, for example. > all sorts of operations take longer (pulling, grepping source, etc.) to support code I couldn't care less about. Makes sense about the pulling, but again, wouldn't grepping be configurable to only search where you need to? > Frequently have to update the world at once. Unless the repo can store multiple versions of the same module, then all the consumers have to be updated at once, even if it's inconvenient. Sometimes migrations are better done gradually. I'm not sure I understood you here, can you expand on that? Do you mean all the devs have to update their module? Why not use tagged/branched version of libs instead of working off trunk? What kind of tooling do you use for the distributed approach?
- gmueckl 9y ago
  Using different branches for different modules becomes hard in a monorepo when branching is a global operation. You can only have a single branch checked out in your working copy then. The only solution I can think of is to create a copy within the monorepo to create de facto branches without regular VCS support. This would be kind of terrible.
- nickm12 9y ago
  Yeah, it's all about the tooling. The distributed system I used was when I worked at Amazon. Each "package" there has its own git repo, which can have multiple branches and the dependencies of each branch are versioned along with the branch. I wonder if at the end of the day it really matters whether something is a "monorepo" or not if the tooling provides the necessary abstractions to version things the way you need.
  - gmueckl 9y ago
    Did you use standard software for the versioning? We are looking for a solution for this at work and have come up blank so far.
geocar 9y ago
Software A version 1 consumes format F1 and produces format G1 data, and software B version 1 consumes format G1 and produces H1. To upgrade format G2 we must change both software A and B. First, software B version 2 must accept both G1 and G2. To do this we may need to build software A version 2 and try them in a sandbox environment to gain confidence that ∀F1 we produce the correct G2. If F1 is complete, we may be able to do this exhaustively, but if F1 is sufficiently diverse, monte carlo simulation might be used. Then, if there's a 1:1 relationship between A/B we can upgrade pairs. If there's a N:M relationship, we need to upgrade all of the instances of software version B1 to B2 (at least within a shard). If you're running in a non-stop environment, this might have it's own challenges. Only then, can we begin the upgrade from A1 to A2. Now: Something, somewhere needs to record what and where we are in this journey. It is relatively straightforward how to do this with a monorepo, but it is very unclear how to do it with a distributed repository: Almost everyone I know punts and uses some other golden record (like a continuous integration server, or a ticketing system, or an admin/staging system), and like it or not: that's your monorepo.
- newfoundglory 9y ago
  Not by any definition I can think of. How are you defining "monorepo"?
- qquark 9y ago
  > but it is very unclear how to do it with a distributed repository Versioning through branching and tagging, while having some drawbacks - at least the fact that you have to DO an operation and that this is not automatic - seem to solve this problem, and are not, in my eyes, a form of monorepo. You globally get more flexibility at the cost of a bit more repo management work. If the problem is retrieving the right version automatically, externals or submodules should be able to solve this problem. If A and B have no clear dependency direction, a top level repo might help.
  - geocar 9y ago
    > a top level repo might help. This is the way I generally do it: A repository that represents my system/environment that has submodules for A1, A2, B1, and B2, and scripts for updating the environment.
vertex-four 9y ago
The trick is that Google have their own VCS, build tooling, automated refactoring tools, etc etc, specifically designed to deal with their monorepo. Nobody else has that - we're stuck with git and a complex landscape of tools for managing code in ad-hoc ways. As a result, with the tools we have, many repos is better than a monorepo - but perhaps if we had those tools, for some cases, a monorepo might be better than many repos. Note that even where Google are forced to use git (e.g. Android, Chrome) they use a many-repo approach.
- w_t_payne 9y ago
  I have written my own tooling. It is doable. Just time-consuming. -- It took me about a year (2015) of my (limited) spare time.
- user5994461 9y ago
  Google has that, Facebook has that, Microsoft partially has that, some investment banks have that too. Everywhere I've seen mono repo, mono repo was better than multi repo. They all built special tooling and have dedicated teams to support it.
  - perfmode 9y ago
    I wish someone would create an open-source vcs that supports mono-repos at scale out of the box.
    - albertzeyer 9y ago
      Microsoft has released some tools to support that for Git. https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-largest-git-repo-on-the-planet/ https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-large... https://blogs.msdn.microsoft.com/devops/2017/05/30/optimizing-git-beyond-gvfs/ https://blogs.msdn.microsoft.com/devops/2017/05/30/optimizin...
    - qznc 9y ago
      Why is the problem with Subversion?
w_t_payne 9y ago
Both approaches can be made to work. For me, the overriding concern is simplicity and ease of configuration management, so I prefer something that on the surface looks like a monorepo. Somewhat paradoxically, in my attempt to solve the issues that you mention, I ended up scripting my commits and checkouts so I could place the repository in a set of git repositories -- so I have a distributed set of repositories under the hood that look like a monorepo to the people using it! Neat, huh?
user5994461 9y ago
I will get a furry of downvotes for saying it but the cause of your first three problems is git, not the mono repo approach. Git only allows to check/commit/view to the entire repo at once. Then, some git operations are superlinear with the number of files or revisions, they are slow on large repo to the point of being unusable. It's mandatory to have operation on a per file or subdirectory level in a mono repo approach. Companies that have mono repos all built tooling to support it. CVS/SVN used to do that out of the box but everyone hate them now.
- k__ 9y ago
  Why aren't submodules more wide spread? Building a monorepo out of submodules should solve these problems, or not? Also, there is subtree.
  - iveqy 9y ago
    Because submodules is hard to understand. They require that you understand git and that you can add one level of abstraktion. Once you done that, they är great!
    - Groxx 9y ago
      Also hard to deal with. Lots of operations leave submodules in unclean / out-of-date states. In general / light use, yeah, they're great. Unfortunately, they have a very large number of edge cases where they essentially require either a) everyone to be experts in the edge cases, or b) tons of new tooling (because existing tools won't take these steps for you).
    - lamlam 9y ago
      That is... until you try to remove a submodule. I like sobmodules, but the whole feature needs a lot more polish before it's widely adopted.
- svckr 9y ago
  > CSV/SVN used to do that out of the box but everyone hate them now. It's "CVS" and it lacks a concept for a repository-wide version (except, maybe, a timestamp). A repository-wide version is –I guess– the single best reason to have monorepo in the first place. SVN is okay. Also, `git log -- subdir/`.
albertzeyer 9y ago
You could argue that all your points can be solved also by the right tooling.
nerdwaller 9y ago
> 3) Frequently have to update the world at once I’ve never worked in a monorepo, so may be wrong, but this point presumes a dependency on “latest” at all times. I’d assume the components in the mono repo still release versions to the various package management systems (maven, pypi, npm, etc), allowing dependencies to be more stable. Has that not been the common experience by those that have worked in them? I see a lot of merits of having to update everything at once (less code rot, hopefully) but it does seem to have drawbacks (many have commented on these as well).
- malkia 9y ago
  If the library/component is wide spread, it either can be developed in a branch, and specific version be tagged, or always developed "trunk" mode with feature toggles (may not be always possible, but one can adjust), e.g. - certain features are disabled, and need to be enabled after others come across, or after some specific time, etc. While at google, we used that kind of development for the project I was in. Someone would push source code changes for new features, but prefferable behind a flag (normally a command-line flag, driven to a configuration, like the one ksonnet has). The confiuration file would say - enable this flag, only the binary was compiled with this CL version, and/or these cherrypicks, or some other rule. This also allows a feature to be quickly disabled by SRE, SWE, or other personnel if it's found to be not working well.
woolvalley 9y ago
#1 in git is `git log <subdirectory>`
hvidgaard 9y ago
I am a firm believer of using processes and tools that make it hard to do stuff you should not be doing in the first place. Dependencies should be solved ala how FAKE and Paket is doing it, not in a mono repo. It's the same story for many projects in a solution vs few. With few projects you avoid wrestling with cyclic dependencies between projects, on the other hand, that is just a tell tale sign that the overall structure is starting to deteriorate. I still do not see the appeal in mono repos since you're heavily dependent on discipline to not introduce spaghetti dependencies, where you fix one bug, but introduce 4 new in unrelated parts of the code. Now you solve that with an if statement, and thus we introduced a great deal of technical debt.
zeveb 9y ago
(argh, I wasn't able to post this due to HN's over-eager 'submitting too fast' filter; at the time, I'd submitted all of two comments & one story within an hour — and then I had to restart my browser, which lost the text of this comment) > 1) Difficult to track changes to the code I'm interested in. What's wrong with 'git log $PATH'? > 2) all sorts of operations take longer (pulling, grepping source, etc.) to support code I couldn't care less about. A different format could help here, as can different tools (e.g. ripgrep or ag instead of grep). The time spent on those operations has to be balanced with the time spent updating your code to deal with someone else's incompatible library changes, again, when the other person is on vacation and you have no idea what the new philosophy of his library is. And you don't have any choice about updating, because another one of your dependencies that you really must update has already been updated to rely on his changes. > 3) Frequently have to update the world at once. IMHO that's a feature, not a bug. The person or team responsible for breaking the world is responsible for fixing it, rather than getting to break the world, then pop off down to Barton-on-Sea for an extended holiday while everyone else in the company gets to update his code to use an entirely different idiom. > 4) Encourages sloppy dependency management. My experience has been that multiple repos tend to encourage sloppy dependency management, while a monorepo tends to encourage deliberative, collaborative, professional dependency management. That's just my own experience, and of course different organisations will differ. > I'm sure people will say "if you're having those problems, you're doing it wrong" but the same thing could be said to people who find the distributed model problematic. My own experience has been that multirepos tend to be like dynamic typing and monorepos tend to be like static typing: multirepos can in theory be done right, but in practice they never are, while monorepos work, but at the cost of people having to colour within the lines. Which makes sense for any particular organisation may actually be a function of its maturity: if a place is trying to move fast and break things, maybe multiple repos make sense; if it's trying to deliver quality software, maybe a single repo makes sense.
s17n 9y ago
1 and 2 are trivially fixable. 3 and 4 are pretty fundamental though, especially 3 - if you don't want to force everybody to keep up with head, you probably don't want to use a monorepo.

Sacho 9y ago

> Every day there are hundreds of changes in the repo and almost all of them have nothing to do with what I'm working on. Wouldn't the right tooling be able to show you changes to the slice of code you're interested in? I remember SVN would allow you to checkout just a single subdirectory, for example. > all sorts of operations take longer (pulling, grepping source, etc.) to support code I couldn't care less about. Makes sense about the pulling, but again, wouldn't grepping be configurable to only search where you need to? > Frequently have to update the world at once. Unless the repo can store multiple versions of the same module, then all the consumers have to be updated at once, even if it's inconvenient. Sometimes migrations are better done gradually. I'm not sure I understood you here, can you expand on that? Do you mean all the devs have to update their module? Why not use tagged/branched version of libs instead of working off trunk? What kind of tooling do you use for the distributed approach?

gmueckl 9y ago
Using different branches for different modules becomes hard in a monorepo when branching is a global operation. You can only have a single branch checked out in your working copy then. The only solution I can think of is to create a copy within the monorepo to create de facto branches without regular VCS support. This would be kind of terrible.
nickm12 9y ago
Yeah, it's all about the tooling. The distributed system I used was when I worked at Amazon. Each "package" there has its own git repo, which can have multiple branches and the dependencies of each branch are versioned along with the branch. I wonder if at the end of the day it really matters whether something is a "monorepo" or not if the tooling provides the necessary abstractions to version things the way you need.
- gmueckl 9y ago
  Did you use standard software for the versioning? We are looking for a solution for this at work and have come up blank so far.

geocar 9y ago

Software A version 1 consumes format F1 and produces format G1 data, and software B version 1 consumes format G1 and produces H1. To upgrade format G2 we must change both software A and B. First, software B version 2 must accept both G1 and G2. To do this we may need to build software A version 2 and try them in a sandbox environment to gain confidence that ∀F1 we produce the correct G2. If F1 is complete, we may be able to do this exhaustively, but if F1 is sufficiently diverse, monte carlo simulation might be used. Then, if there's a 1:1 relationship between A/B we can upgrade pairs. If there's a N:M relationship, we need to upgrade all of the instances of software version B1 to B2 (at least within a shard). If you're running in a non-stop environment, this might have it's own challenges. Only then, can we begin the upgrade from A1 to A2. Now: Something, somewhere needs to record what and where we are in this journey. It is relatively straightforward how to do this with a monorepo, but it is very unclear how to do it with a distributed repository: Almost everyone I know punts and uses some other golden record (like a continuous integration server, or a ticketing system, or an admin/staging system), and like it or not: that's your monorepo.

newfoundglory 9y ago
Not by any definition I can think of. How are you defining "monorepo"?
qquark 9y ago
> but it is very unclear how to do it with a distributed repository Versioning through branching and tagging, while having some drawbacks - at least the fact that you have to DO an operation and that this is not automatic - seem to solve this problem, and are not, in my eyes, a form of monorepo. You globally get more flexibility at the cost of a bit more repo management work. If the problem is retrieving the right version automatically, externals or submodules should be able to solve this problem. If A and B have no clear dependency direction, a top level repo might help.
- geocar 9y ago
  > a top level repo might help. This is the way I generally do it: A repository that represents my system/environment that has submodules for A1, A2, B1, and B2, and scripts for updating the environment.

vertex-four 9y ago

The trick is that Google have their own VCS, build tooling, automated refactoring tools, etc etc, specifically designed to deal with their monorepo. Nobody else has that - we're stuck with git and a complex landscape of tools for managing code in ad-hoc ways. As a result, with the tools we have, many repos is better than a monorepo - but perhaps if we had those tools, for some cases, a monorepo might be better than many repos. Note that even where Google are forced to use git (e.g. Android, Chrome) they use a many-repo approach.

w_t_payne 9y ago
I have written my own tooling. It is doable. Just time-consuming. -- It took me about a year (2015) of my (limited) spare time.
user5994461 9y ago
Google has that, Facebook has that, Microsoft partially has that, some investment banks have that too. Everywhere I've seen mono repo, mono repo was better than multi repo. They all built special tooling and have dedicated teams to support it.
- perfmode 9y ago
  I wish someone would create an open-source vcs that supports mono-repos at scale out of the box.
  - albertzeyer 9y ago
    Microsoft has released some tools to support that for Git. https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-largest-git-repo-on-the-planet/ https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-large... https://blogs.msdn.microsoft.com/devops/2017/05/30/optimizing-git-beyond-gvfs/ https://blogs.msdn.microsoft.com/devops/2017/05/30/optimizin...
  - qznc 9y ago
    Why is the problem with Subversion?

w_t_payne 9y ago

Both approaches can be made to work. For me, the overriding concern is simplicity and ease of configuration management, so I prefer something that on the surface looks like a monorepo. Somewhat paradoxically, in my attempt to solve the issues that you mention, I ended up scripting my commits and checkouts so I could place the repository in a set of git repositories -- so I have a distributed set of repositories under the hood that look like a monorepo to the people using it! Neat, huh?

user5994461 9y ago

I will get a furry of downvotes for saying it but the cause of your first three problems is git, not the mono repo approach. Git only allows to check/commit/view to the entire repo at once. Then, some git operations are superlinear with the number of files or revisions, they are slow on large repo to the point of being unusable. It's mandatory to have operation on a per file or subdirectory level in a mono repo approach. Companies that have mono repos all built tooling to support it. CVS/SVN used to do that out of the box but everyone hate them now.

k__ 9y ago
Why aren't submodules more wide spread? Building a monorepo out of submodules should solve these problems, or not? Also, there is subtree.
- iveqy 9y ago
  Because submodules is hard to understand. They require that you understand git and that you can add one level of abstraktion. Once you done that, they är great!
  - Groxx 9y ago
    Also hard to deal with. Lots of operations leave submodules in unclean / out-of-date states. In general / light use, yeah, they're great. Unfortunately, they have a very large number of edge cases where they essentially require either a) everyone to be experts in the edge cases, or b) tons of new tooling (because existing tools won't take these steps for you).
  - lamlam 9y ago
    That is... until you try to remove a submodule. I like sobmodules, but the whole feature needs a lot more polish before it's widely adopted.
svckr 9y ago
> CSV/SVN used to do that out of the box but everyone hate them now. It's "CVS" and it lacks a concept for a repository-wide version (except, maybe, a timestamp). A repository-wide version is –I guess– the single best reason to have monorepo in the first place. SVN is okay. Also, `git log -- subdir/`.

albertzeyer 9y ago

You could argue that all your points can be solved also by the right tooling.

nerdwaller 9y ago

> 3) Frequently have to update the world at once I’ve never worked in a monorepo, so may be wrong, but this point presumes a dependency on “latest” at all times. I’d assume the components in the mono repo still release versions to the various package management systems (maven, pypi, npm, etc), allowing dependencies to be more stable. Has that not been the common experience by those that have worked in them? I see a lot of merits of having to update everything at once (less code rot, hopefully) but it does seem to have drawbacks (many have commented on these as well).

malkia 9y ago
If the library/component is wide spread, it either can be developed in a branch, and specific version be tagged, or always developed "trunk" mode with feature toggles (may not be always possible, but one can adjust), e.g. - certain features are disabled, and need to be enabled after others come across, or after some specific time, etc. While at google, we used that kind of development for the project I was in. Someone would push source code changes for new features, but prefferable behind a flag (normally a command-line flag, driven to a configuration, like the one ksonnet has). The confiuration file would say - enable this flag, only the binary was compiled with this CL version, and/or these cherrypicks, or some other rule. This also allows a feature to be quickly disabled by SRE, SWE, or other personnel if it's found to be not working well.

woolvalley 9y ago

#1 in git is `git log <subdirectory>`

hvidgaard 9y ago

I am a firm believer of using processes and tools that make it hard to do stuff you should not be doing in the first place. Dependencies should be solved ala how FAKE and Paket is doing it, not in a mono repo. It's the same story for many projects in a solution vs few. With few projects you avoid wrestling with cyclic dependencies between projects, on the other hand, that is just a tell tale sign that the overall structure is starting to deteriorate. I still do not see the appeal in mono repos since you're heavily dependent on discipline to not introduce spaghetti dependencies, where you fix one bug, but introduce 4 new in unrelated parts of the code. Now you solve that with an if statement, and thus we introduced a great deal of technical debt.

zeveb 9y ago

(argh, I wasn't able to post this due to HN's over-eager 'submitting too fast' filter; at the time, I'd submitted all of two comments & one story within an hour — and then I had to restart my browser, which lost the text of this comment) > 1) Difficult to track changes to the code I'm interested in. What's wrong with 'git log $PATH'? > 2) all sorts of operations take longer (pulling, grepping source, etc.) to support code I couldn't care less about. A different format could help here, as can different tools (e.g. ripgrep or ag instead of grep). The time spent on those operations has to be balanced with the time spent updating your code to deal with someone else's incompatible library changes, again, when the other person is on vacation and you have no idea what the new philosophy of his library is. And you don't have any choice about updating, because another one of your dependencies that you really must update has already been updated to rely on his changes. > 3) Frequently have to update the world at once. IMHO that's a feature, not a bug. The person or team responsible for breaking the world is responsible for fixing it, rather than getting to break the world, then pop off down to Barton-on-Sea for an extended holiday while everyone else in the company gets to update his code to use an entirely different idiom. > 4) Encourages sloppy dependency management. My experience has been that multiple repos tend to encourage sloppy dependency management, while a monorepo tends to encourage deliberative, collaborative, professional dependency management. That's just my own experience, and of course different organisations will differ. > I'm sure people will say "if you're having those problems, you're doing it wrong" but the same thing could be said to people who find the distributed model problematic. My own experience has been that multirepos tend to be like dynamic typing and monorepos tend to be like static typing: multirepos can in theory be done right, but in practice they never are, while monorepos work, but at the cost of people having to colour within the lines. Which makes sense for any particular organisation may actually be a function of its maturity: if a place is trying to move fast and break things, maybe multiple repos make sense; if it's trying to deliver quality software, maybe a single repo makes sense.

s17n 9y ago

1 and 2 are trivially fixable. 3 and 4 are pretty fundamental though, especially 3 - if you don't want to force everybody to keep up with head, you probably don't want to use a monorepo.