3 ms·

I worked in a company where a million line project was split in about 130 repos. Features or bug fixes frequently required syncing PRs between multiple repos. W

by auxym 7y ago

I worked in a company where a million line project was split in about 130 repos. Features or bug fixes frequently required syncing PRs between multiple repos. We needed scripts to create feature branches in all repos, or bump up the version everywhere. 4 or 5 repos would probably have been sane. 130 was hell to manage.

bluGill 7y ago
If that happens often you have the wrong repo split. This isn't a condemnation of the multi repo approach, only your architecture. Note that this is about trade offs. If you have a monorepo this problem goes away but now you need to manage the problems of monorepos. If you have multiple repos you get this and other problems instead. Pick the right tradeoffs for your own needs. The only things wrong is claiming your answer is the right one for everyone else.
- auxym 7y ago
  > If that happens often you have the wrong repo split. Yes, but: you pick out a nice, clean perfectly segregated architecture. Two years later, things have evolved, and a new feature that no one initially thought of is asked and involved refactoring 25 repos. Oops. Side note: not my architecture. I only worked there for a short while. I have no skin in the game either way, I'm sure I agree with you that mono and multi- repos are the right solution for different problems. Just trying to expose problems I've seen happen in a massive multirepo approach. Everyone thought it was the correct solution after being burned by a massive monolith that over two decades became an entangled mess. They thought they had the perfect architecture, the "correct repo split". In practice it's almost impossible, which is a major tradeoff to be considered. Minimizin the number of repos and splitting only when absolutely necessary at least has a chance of reducing that complexity. The default should be "as few repos as possible", not "we're going multirepo, let us break this down into as many small parts as possible as our first step", which might be what the architects were used to in modular program architecture (small, reusable functions/classes).
  - vorpalhex 7y ago
    > The default should be "as few repos as possible" In any corporate/enterprise-y environment with multiple teams, the default should be the smallest piece likely to change ownership. When we see a pattern where a feature involves refactoring a bunch of repos, then your initial system breakdown was horrendously off-track and you lost sight of your domains. Been there, it is fixable but it takes a bit. > Everyone thought it was the correct solution There is no correct solution, there is the best solution for your problem set. Sometimes that's a monolith, and sometimes that's several small repos. Every once in a while, it's even a monorepo.
- beagle3 7y ago
  > you need to manage the problems of monorepos If have not seen a problem on monorepos that does not exist on multirepos, other than more quickly reaching tools' repo size limits (but this is becoming less of a problem all the time on e.g. git and mercurial, and supposedly was never a problem with perforce). I have seen many problems multirepos have that monorepos do not - such as that multirepos can (and almost always do) have a wrong split, and monorepos by definition cannot.
  - yokaze 7y ago
    > If have not seen a problem on monorepos that does not exist on multirepo How do you refer to a library in a monorepo from a different repo? The workflow seems to me to entail an all-or-nothing approach.
    - beagle3 7y ago
      I never tried it, but if I needed it, I would easier use sparse submodule checkout (if it’s possible - I don’t know if that’s true), or submodule the entire repo, and the symbolic link into the internal part I need. If all else fails (e.g. you need 50 files out of a 100GB repo, which makes things too slow), I would set up a git subtree extract as a checkin hook, which would mean there exists another repo/branch automatically updated on checkin - and which you could use any way you want, e.g, simple submodule. Do note that’s just what I said: the problem, “I only need a small part of the repo” is independent of it being a monorepo or multirepo. You would only avoid this problem in a multirepo if you cut it to pieces in exactly the right places. it’s just a matter of whether the parts of the repo you don’t need are too large to make this comfortable — more likely in a monorepo as it is larger - but will happen in a big enough multirepo as well.
- throwaway894345 7y ago
  Even with the best architecture, you probably have some shared libraries that are used across teams. These will often need to be updated, and they will affect several other packages (other repos, in the multirepo case). The only way to avoid this is to duplicate all code. Moreover, you can have a great architecture, but your teams are "feature-oriented" instead of service oriented (or some other vertically oriented team structure) in which case they'll need to touch many different repos (I don't imagine feature-oriented repos are a good idea) per ticket. I'm increasingly convinced that the monorepo approach is the right one, but the tooling needs to get better. Bazel is a hydra of incidental complexity and Nix has the right core idea but suffers from a variety of other problems (poor documentation, gratuitously and unnecessarily unfamiliar, too few escape hatches, etc). These issues aren't fundamental to the domain, and there's no reason these tools couldn't change course nor is there a reason that another tool couldn't come in and eat their lunch.
mumblemumble 7y ago
I worked in a company that had at least 100 repos, often with interdependencies, and was managing it well using svn. Then we migrated to Git, which doesn't have a good equivalent to svn's externals, and things kind of went to heck for a while. We eventually hacked together an in-house system that brought back some of what we were missing, but it never worked as smoothly as svn externals did. I was originally one of the advocates of moving to Git. Hindsight being 20/20, after the dust settled I realized we would have been fine staying on svn, anyway. With such a high number of individual repositories, blocking on each other and nasty merge conflicts were rarely a problem in practice because we had over 100 swim lanes clearly marked in the pool. Lots more swim lanes than we had developers. FWIW, one team dealt with it by migrating just their stuff to their own monorepo, and I wasn't too impressed by that outcome, either. They started seeing an outsize rate of technical debt accumulation as things that previously were easier to keep well modularized started naturally getting more tangled together. I realize that is a problem that can easily be managed with just a little effort. But, like every other dev problem that can easily be managed with just a little effort, it wasn't.
- pierrebai 7y ago
  git has submodules. A repo can thus host any number of sub-repos. Using modules can also be pretty much like having a monorepo if you have a master repo with all others under it. This way, you can still publish things like public API as a repo.
  - mumblemumble 7y ago
    They don't work quite the same way as svn externals, though, in a way that didn't work for us. In particular, they leave no good way to deal with merging in upstream changes in a submodule's code if you've also made some of your own (provisional) edits.
  - mceachen 7y ago
    Submodules don't allow for atomic commits that span repos, though. Atomic cross-project commits are the most compelling selling feature for monorepos IMHO. (A common case for this is changing a function's API, and fixing all downstream consumers, in one tidy, holistic PR).

by auxym 7y ago

bluGill 7y ago

If that happens often you have the wrong repo split. This isn't a condemnation of the multi repo approach, only your architecture. Note that this is about trade offs. If you have a monorepo this problem goes away but now you need to manage the problems of monorepos. If you have multiple repos you get this and other problems instead. Pick the right tradeoffs for your own needs. The only things wrong is claiming your answer is the right one for everyone else.

auxym 7y ago
> If that happens often you have the wrong repo split. Yes, but: you pick out a nice, clean perfectly segregated architecture. Two years later, things have evolved, and a new feature that no one initially thought of is asked and involved refactoring 25 repos. Oops. Side note: not my architecture. I only worked there for a short while. I have no skin in the game either way, I'm sure I agree with you that mono and multi- repos are the right solution for different problems. Just trying to expose problems I've seen happen in a massive multirepo approach. Everyone thought it was the correct solution after being burned by a massive monolith that over two decades became an entangled mess. They thought they had the perfect architecture, the "correct repo split". In practice it's almost impossible, which is a major tradeoff to be considered. Minimizin the number of repos and splitting only when absolutely necessary at least has a chance of reducing that complexity. The default should be "as few repos as possible", not "we're going multirepo, let us break this down into as many small parts as possible as our first step", which might be what the architects were used to in modular program architecture (small, reusable functions/classes).
- vorpalhex 7y ago
  > The default should be "as few repos as possible" In any corporate/enterprise-y environment with multiple teams, the default should be the smallest piece likely to change ownership. When we see a pattern where a feature involves refactoring a bunch of repos, then your initial system breakdown was horrendously off-track and you lost sight of your domains. Been there, it is fixable but it takes a bit. > Everyone thought it was the correct solution There is no correct solution, there is the best solution for your problem set. Sometimes that's a monolith, and sometimes that's several small repos. Every once in a while, it's even a monorepo.
beagle3 7y ago
> you need to manage the problems of monorepos If have not seen a problem on monorepos that does not exist on multirepos, other than more quickly reaching tools' repo size limits (but this is becoming less of a problem all the time on e.g. git and mercurial, and supposedly was never a problem with perforce). I have seen many problems multirepos have that monorepos do not - such as that multirepos can (and almost always do) have a wrong split, and monorepos by definition cannot.
- yokaze 7y ago
  > If have not seen a problem on monorepos that does not exist on multirepo How do you refer to a library in a monorepo from a different repo? The workflow seems to me to entail an all-or-nothing approach.
  - beagle3 7y ago
    I never tried it, but if I needed it, I would easier use sparse submodule checkout (if it’s possible - I don’t know if that’s true), or submodule the entire repo, and the symbolic link into the internal part I need. If all else fails (e.g. you need 50 files out of a 100GB repo, which makes things too slow), I would set up a git subtree extract as a checkin hook, which would mean there exists another repo/branch automatically updated on checkin - and which you could use any way you want, e.g, simple submodule. Do note that’s just what I said: the problem, “I only need a small part of the repo” is independent of it being a monorepo or multirepo. You would only avoid this problem in a multirepo if you cut it to pieces in exactly the right places. it’s just a matter of whether the parts of the repo you don’t need are too large to make this comfortable — more likely in a monorepo as it is larger - but will happen in a big enough multirepo as well.
throwaway894345 7y ago
Even with the best architecture, you probably have some shared libraries that are used across teams. These will often need to be updated, and they will affect several other packages (other repos, in the multirepo case). The only way to avoid this is to duplicate all code. Moreover, you can have a great architecture, but your teams are "feature-oriented" instead of service oriented (or some other vertically oriented team structure) in which case they'll need to touch many different repos (I don't imagine feature-oriented repos are a good idea) per ticket. I'm increasingly convinced that the monorepo approach is the right one, but the tooling needs to get better. Bazel is a hydra of incidental complexity and Nix has the right core idea but suffers from a variety of other problems (poor documentation, gratuitously and unnecessarily unfamiliar, too few escape hatches, etc). These issues aren't fundamental to the domain, and there's no reason these tools couldn't change course nor is there a reason that another tool couldn't come in and eat their lunch.

mumblemumble 7y ago

I worked in a company that had at least 100 repos, often with interdependencies, and was managing it well using svn. Then we migrated to Git, which doesn't have a good equivalent to svn's externals, and things kind of went to heck for a while. We eventually hacked together an in-house system that brought back some of what we were missing, but it never worked as smoothly as svn externals did. I was originally one of the advocates of moving to Git. Hindsight being 20/20, after the dust settled I realized we would have been fine staying on svn, anyway. With such a high number of individual repositories, blocking on each other and nasty merge conflicts were rarely a problem in practice because we had over 100 swim lanes clearly marked in the pool. Lots more swim lanes than we had developers. FWIW, one team dealt with it by migrating just their stuff to their own monorepo, and I wasn't too impressed by that outcome, either. They started seeing an outsize rate of technical debt accumulation as things that previously were easier to keep well modularized started naturally getting more tangled together. I realize that is a problem that can easily be managed with just a little effort. But, like every other dev problem that can easily be managed with just a little effort, it wasn't.

pierrebai 7y ago
git has submodules. A repo can thus host any number of sub-repos. Using modules can also be pretty much like having a monorepo if you have a master repo with all others under it. This way, you can still publish things like public API as a repo.
- mumblemumble 7y ago
  They don't work quite the same way as svn externals, though, in a way that didn't work for us. In particular, they leave no good way to deal with merging in upstream changes in a submodule's code if you've also made some of your own (provisional) edits.
- mceachen 7y ago
  Submodules don't allow for atomic commits that span repos, though. Atomic cross-project commits are the most compelling selling feature for monorepos IMHO. (A common case for this is changing a function's API, and fixing all downstream consumers, in one tidy, holistic PR).