8 ms·

My problem with polyrepos is that often organizations end up splitting things too finely, and now I'm unable to make a single commit to introduce a feature beca

My problem with polyrepos is that often organizations end up splitting things too finely, and now I'm unable to make a single commit to introduce a feature because my changes have to live across several repositories. Which makes code review more annoying because you have to tab back and forth to see all the context. It's doubly frustrating when I'm (or my team is) the only people working on those repositories, because now it doesn't feel like it gained any advantages. I know the author addresses this, but I can't imagine projects are typically at the scale they're describing. Certainly it's not my experience. Also I definitely miss the ability to make changes to fundamental (internal) libraries used by every project. It's too much hassle to track down all the uses of a particular function, so I end up putting that change elsewhere, which means someone else will do it a little different in their corner of the world, which utterly confuses the first person who's unlucky enough to work in both code bases (at the same time, or after moving teams).

WorldMaker 8y ago
It's an interesting social problem in how you manage those project / library / repository boundaries. On the flipside, though, it's been well documented that among many of the major monorepos those boundaries still exist, they just become far more opaque because no one has to track them. You find the weird gatekeepers in the dark that spring out only when you get late in your code review process because you touched "their" file and they got an automated notice from a hidden rules engine in your CI process you didn't even realize existed. In the polyrepo case those boundaries have to be made explicit (otherwise no one gets anything done) and those owners should be easily visible. You may not like the friction they sometimes bring to the table, but at least it won't be a surprise.
- rgoulter 8y ago
  http://wiki.c2.com/?ConwaysLaw http://wiki.c2.com/?ConwaysLaw "Conway's Law" is something like "organisation of code will match the organisation of people". It's a neat description. I think it's more common to merge or split modules and classes than repositories. I wonder if there'd be less tension if repos and teams were 1:1 though.
  - freedomben 8y ago
    > I wonder if there'd be less tension if repos and teams were 1:1 though Anecdotally, yes I think it helps a lot. I was once part of an organization for which each "team" having a repo is the only thing that prevented violence :-)
    - matwood 8y ago
      I've also seen arbitrary separations of repos because 2 people didn't get along and couldn't work together.
- Mxtetris 8y ago
  Can a monorepo support module- or subdirectory-level ownership controls? Or do teams using a monorepo just do without them? Partially answering my own question: SVN, recommended in a prior comment [0], supports path-based authorization [1]. But what about teams using another version control system? [0] https://news.ycombinator.com/item?id=18810313 https://news.ycombinator.com/item?id=18810313 [1] http://svnbook.red-bean.com/en/1.5/svn-book.html#svn.serverconfig.pathbasedauthz http://svnbook.red-bean.com/en/1.5/svn-book.html#svn.serverc...
  - monksy 8y ago
    SVN allows for you to create mutliple repos within a repo. (That's probably why the path based auth works). Git has the idea of sub-modules, but they're really just filters. (They're in the same repo). So ultimately, you don't have that kind of control.
    - icebraining 8y ago
      Git submodules are not in the same repo, they are a link from one repo to another, and you need to push to both if you make a change to the submodule. Maybe you're thinking of subtrees? I've never used those.
  - maktouch 8y ago
    With Phabricator, yes, you can setup herald rules that stops a merge from happening if a file has changed in a specific subdir. We use service owners, so when a change spans multiple services, they are all added automatically as blocking reviewers.
  - xyzzyz 8y ago
    Piper, Google’s implementation of monorepo, has that, and it is very important and widely used.
  - gusmd 8y ago
    If I remember correctly, Gitlab has introduced some sort of ownership control where you can say who owns what directories for things like approving merge requests that affect those directories.
dmix 8y ago
OctoLinker really helps when browsing a polyrepo on Github: https://github.com/OctoLinker/OctoLinker https://github.com/OctoLinker/OctoLinker You can just click the import [project] name and it will switch to the repo.
jon-wood 8y ago
It's very much possible to make changes to internal libraries used all over the place, but it does require versioning to be something that people think about, and a mechanism by which those libraries aren't just pulled from source control to depend on them. Once you've got some sort of dependency management such as an internal gem/npm/whatever source you can treat those internal dependencies the same as you'd treat external ones, instead of having to somehow coordinate a release of absolutely everything in one go.
inertiatic 8y ago
My current team managed to break a single "component" out into a separate repository. Then that repository broke into two, then those broke into other repositories, until we've eventually have around 10 or so different repositories that we work on every day. An average change touches 4 of them, and touching one of them triggers on average releases on 2 or 3 of them. Even building these locally is super tedious, because we don't have any automation in place (not formally plan to) for chain building these locally. This is a nightmare scenario for myself. A simple change can require 4 pull requests and reviews, half a day to test and a couple hours to release. Yet my team keeps identifying small pieces that can be conceptually separated from the rest of the functionality, even if they are heavily coupled, and makes new repos for these!
- rauhl 8y ago
  I’ve come to the conclusion that an organisation should ideally have no more than one primary repo, with maybe a handful of ancillary repos for stuff that really doesn’t make sense in the primary. What does ‘organisation’ mean there? Well, it could mean a company, or a team, or a division. Just as software conforms to organisational structure (Conway’s Law), so too should repo structure. Once you start having lots of peer repos being worked on within the same organisation on a daily basis, you know that you’ve partitioned far too far, and you need to roll back. Otherwise one ends up in exactly the position you’re in. The ultimate slippery-slope end-state would be hilariously bad: a repo for each ASCII character, with repos for each word or symbol constructed out of those characters, with repos for each function constructed out of those words & symbols, with repos for each module constructed out of those functions, with repos for each system constructed out of those modules, with any change requiring a massive, intricate, failure-prone dance in order to update anything, all while patting oneself on the back about how one has avoided complexity. Noöne sane would argue for that situation, and yet I’ve seen smart people argue that requiring coördinated changes to half a dozen repos is fine & dandy.
- drugme 8y ago
  even if they are heavily coupled, So don't use polyrepos for heavily coupled projects, then. Or even better... ... try to avoid heavy coupling in the first place.
bborud 8y ago
That's not really that different in a monorepo since you often need reviews from the same number of people anyway. I once had to wait for 9 months to get a complex change through in a monorepo setting because of all the people involved, the number of stuff it touched and the fact that everything was constantly in flux so I spent half my time tracking changes. I'm not saying it would have been faster in a polyrepo. I'm saying that complex changes are complex regardless of how the source is organized. I do however think that polyrepos forces you to be more disciplined and that it is easier to slip up in a polyrepo and turn a blind eye to tighter couplings.
marcinkuzminski 8y ago
The multi-repository code review is an interesting concept. Here at RhodeCode we're actually working on such solution to implement. This is in first to solve our internal problem of release code-review spanning usually two projects at once. This is a hard and complex problem. Especially how to make code-review not too messy if you target 5-8 repos at once.

WorldMaker 8y ago

It's an interesting social problem in how you manage those project / library / repository boundaries. On the flipside, though, it's been well documented that among many of the major monorepos those boundaries still exist, they just become far more opaque because no one has to track them. You find the weird gatekeepers in the dark that spring out only when you get late in your code review process because you touched "their" file and they got an automated notice from a hidden rules engine in your CI process you didn't even realize existed. In the polyrepo case those boundaries have to be made explicit (otherwise no one gets anything done) and those owners should be easily visible. You may not like the friction they sometimes bring to the table, but at least it won't be a surprise.

rgoulter 8y ago
http://wiki.c2.com/?ConwaysLaw http://wiki.c2.com/?ConwaysLaw "Conway's Law" is something like "organisation of code will match the organisation of people". It's a neat description. I think it's more common to merge or split modules and classes than repositories. I wonder if there'd be less tension if repos and teams were 1:1 though.
- freedomben 8y ago
  > I wonder if there'd be less tension if repos and teams were 1:1 though Anecdotally, yes I think it helps a lot. I was once part of an organization for which each "team" having a repo is the only thing that prevented violence :-)
  - matwood 8y ago
    I've also seen arbitrary separations of repos because 2 people didn't get along and couldn't work together.
Mxtetris 8y ago
Can a monorepo support module- or subdirectory-level ownership controls? Or do teams using a monorepo just do without them? Partially answering my own question: SVN, recommended in a prior comment [0], supports path-based authorization [1]. But what about teams using another version control system? [0] https://news.ycombinator.com/item?id=18810313 https://news.ycombinator.com/item?id=18810313 [1] http://svnbook.red-bean.com/en/1.5/svn-book.html#svn.serverconfig.pathbasedauthz http://svnbook.red-bean.com/en/1.5/svn-book.html#svn.serverc...
- monksy 8y ago
  SVN allows for you to create mutliple repos within a repo. (That's probably why the path based auth works). Git has the idea of sub-modules, but they're really just filters. (They're in the same repo). So ultimately, you don't have that kind of control.
  - icebraining 8y ago
    Git submodules are not in the same repo, they are a link from one repo to another, and you need to push to both if you make a change to the submodule. Maybe you're thinking of subtrees? I've never used those.
- maktouch 8y ago
  With Phabricator, yes, you can setup herald rules that stops a merge from happening if a file has changed in a specific subdir. We use service owners, so when a change spans multiple services, they are all added automatically as blocking reviewers.
- xyzzyz 8y ago
  Piper, Google’s implementation of monorepo, has that, and it is very important and widely used.
- gusmd 8y ago
  If I remember correctly, Gitlab has introduced some sort of ownership control where you can say who owns what directories for things like approving merge requests that affect those directories.

dmix 8y ago

OctoLinker really helps when browsing a polyrepo on Github: https://github.com/OctoLinker/OctoLinker https://github.com/OctoLinker/OctoLinker You can just click the import [project] name and it will switch to the repo.

jon-wood 8y ago

It's very much possible to make changes to internal libraries used all over the place, but it does require versioning to be something that people think about, and a mechanism by which those libraries aren't just pulled from source control to depend on them. Once you've got some sort of dependency management such as an internal gem/npm/whatever source you can treat those internal dependencies the same as you'd treat external ones, instead of having to somehow coordinate a release of absolutely everything in one go.

inertiatic 8y ago

My current team managed to break a single "component" out into a separate repository. Then that repository broke into two, then those broke into other repositories, until we've eventually have around 10 or so different repositories that we work on every day. An average change touches 4 of them, and touching one of them triggers on average releases on 2 or 3 of them. Even building these locally is super tedious, because we don't have any automation in place (not formally plan to) for chain building these locally. This is a nightmare scenario for myself. A simple change can require 4 pull requests and reviews, half a day to test and a couple hours to release. Yet my team keeps identifying small pieces that can be conceptually separated from the rest of the functionality, even if they are heavily coupled, and makes new repos for these!

rauhl 8y ago
I’ve come to the conclusion that an organisation should ideally have no more than one primary repo, with maybe a handful of ancillary repos for stuff that really doesn’t make sense in the primary. What does ‘organisation’ mean there? Well, it could mean a company, or a team, or a division. Just as software conforms to organisational structure (Conway’s Law), so too should repo structure. Once you start having lots of peer repos being worked on within the same organisation on a daily basis, you know that you’ve partitioned far too far, and you need to roll back. Otherwise one ends up in exactly the position you’re in. The ultimate slippery-slope end-state would be hilariously bad: a repo for each ASCII character, with repos for each word or symbol constructed out of those characters, with repos for each function constructed out of those words & symbols, with repos for each module constructed out of those functions, with repos for each system constructed out of those modules, with any change requiring a massive, intricate, failure-prone dance in order to update anything, all while patting oneself on the back about how one has avoided complexity. Noöne sane would argue for that situation, and yet I’ve seen smart people argue that requiring coördinated changes to half a dozen repos is fine & dandy.
drugme 8y ago
even if they are heavily coupled, So don't use polyrepos for heavily coupled projects, then. Or even better... ... try to avoid heavy coupling in the first place.

bborud 8y ago

That's not really that different in a monorepo since you often need reviews from the same number of people anyway. I once had to wait for 9 months to get a complex change through in a monorepo setting because of all the people involved, the number of stuff it touched and the fact that everything was constantly in flux so I spent half my time tracking changes. I'm not saying it would have been faster in a polyrepo. I'm saying that complex changes are complex regardless of how the source is organized. I do however think that polyrepos forces you to be more disciplined and that it is easier to slip up in a polyrepo and turn a blind eye to tighter couplings.

marcinkuzminski 8y ago

The multi-repository code review is an interesting concept. Here at RhodeCode we're actually working on such solution to implement. This is in first to solve our internal problem of release code-review spanning usually two projects at once. This is a hard and complex problem. Especially how to make code-review not too messy if you target 5-8 repos at once.