3 ms·

The issue with multiple repositories has nothing do to with the number of commands you have to run. As you say, that's the kind of thing that can easily be auto

The issue with multiple repositories has nothing do to with the number of commands you have to run. As you say, that's the kind of thing that can easily be automated. The problems arise when you have to combine code from different repositories into a single deployable product. Most of us don't take Amazon's hard-line stance of making absolutely everything a microservice, so we end up with libraries of reusable code that are referenced by multiple projects. But when you store those libraries in separate repositories, it becomes impossible to describe the state of your deployed code without listing the version of every single dependency. That makes it easy for subtle inconsistencies and bugs to creep in, especially when the dependencies are multiple levels deep and are owned by different teams. If everything lives in the same tree, then a single commit ID reproducibly describes a complete system from top to bottom. And you can atomically make changes that cross module boundaries, which is difficult to do safely with separate repositories. I don't really follow your comment about merging. Pretty much every version control system since forever has been smart enough to realize that, if I make changes only to foo/src/ and you make changes to bar/src/, our changes don't conflict and can be merged automatically without user intervention. (There might be technical difficulties; for example, if you're using Git, I would imagine that trying to view the list of commits of a small subtree of a gigantic repo might not be terribly efficient. But just like the issue of managing multiple repos, that's something that you can solve with better tool support, if you really need to and are motivated enough.)

Confusion 11y ago
About merging (I'll get back to the other bit later): if I want to push my change, I already regularly encounter the situation where git tells me: you are not up to date, even if I pulled say 15 minutes ago. So I have to fetch and rebase before I can push. Even if that doesn't require manually solving any merge conflicts (and I think I have to in about 1/4th of the cases), it takes a bit of time. In that bit of time another developer can push a commit. If that already happens on occasion with 10 developers, I imagine it becomes very problematic with 1000 developers on the same repo. Of course you effectively organise your repos in a 'tree' like e.g. happens with the Linux kernel, but, well, then you again have many repos. One per component that has someone responsible for signing off on them. It reduces the amount of people committing to each specific repo, but in the end, it's still many repos. So that's not 'one monolithic repo' in my book.
- teraflop 11y ago
  Still, that's a limitation of the specific tools and the way you're using them, not a fundamental problem with monolithic repositories. You're running into problems because you have multiple people all trying to push to the same branch, and Git errs on the side of extreme caution rather than creating a merge that you didn't specifically request. If you use merge requests instead, as supported by Github/Gitlab, you don't get blocked by other people's commits. (Actual merge conflicts are an orthogonal problem. They'll happen whenever you have multiple people editing the same code, regardless of what VCS you use or how it's organized.) The Linux kernel example is a bit of a tricky one. Yes, kernel.org hosts a lot of repos, but they're all different versions of the same codebase. They have a common commit ancestry, and commits get merged from one to another. So they're not really separate modules in the sense that we're talking about; they're more like namespaces that each contain a set of branches.
  - Confusion 11y ago
    I guess it comes down to organisational style. Using merge requests requires some benevolent dictator (poor sod?) to perform the merges. For us, if your commit is approved in Gerrit, it's your task to merge it. If some other commit was merged in between, you have to rebase, possibly solving merge conflicts. So guess that with a monolithic repo, that wouldn't scale, but it could be made to scale by appointing someone to perform the merge requests. I'm not sure I would like to be that person...
    - Mathiasdm 11y ago
      I helped set up such a system for a few hundred developers. We had an 'automated Linus Torvalds', which did the merge, and aborted whenever a file was changed on both sides of the merge. In the good case (almost every time), there were no conflicts, and the merge went fine (we had unittests, builds and regression tests as extra checks in our CI system). In the bad case, the developer request was rejected and the developer was told to rebase or merge his code on his own, so the merge issues would be handled.
hvidgaard 11y ago
> it becomes impossible to describe the state of your deployed code without listing the version of every single dependency...If everything lives in the same tree, then a single commit ID reproducibly describes a complete system from top to bottom. This is a tooling problem, that is somewhat solved with a gigantic mono repo. All of our shared components are published with NuGet, and referenced as such. There is no interrepo dependence, and there shouldn't be.
Confusion 11y ago
The problems arise when you have to combine code from different repositories into a single deployable product. OK, we only have a single deployable product, which combines code from all those repositories, so it seems we match this criterion. But when you store those libraries in separate repositories, it becomes impossible to describe the state of your deployed code without listing the version of every single dependency. Yes, and that is what we do: each released lib has a version. The deployed version only uses released libs and lists exactly what it uses. The libs cannot be overwritten or changed after release: once released, that's what that version of the lib is. [..] If everything lives in the same tree, then a single commit ID reproducibly describes a complete system from top to bottom. In our case a single commit ID also reproducibly describes what has been deployed. Each library version was built from a specific commit ID in their respective repos. And you can atomically make changes that cross module boundaries, which is difficult to do safely with separate repositories. We don't have to atomically change anything, because a new backwards incompatible release of a certain lib doesn't affect any other lib, until we explicitly opt to upgrade it to the new version and release a new version. And if it's backwards compatible, it gets automatically picked up, because version management allows it. So I just don't recognize any of: I don't want to waste time dealing with multiple commands to manage multiple repositories. I don't want to waste time or expend cognitive load dealing with submodule, subrepository, or big files management. I don't want to waste time trying to find and reuse code, data, or documentation. I want everything at my fingertips, where it can be easily discovered, inspected, and used. I have a directory with a tree of repos. They could be in one big repo, but aren't. Otherwise, on disk, everything is the same. If I change things in two related libs, I have to commit twice, but I'm also changing two independent libs that I have to release separately anyway, just like in a monolithic repo. Finding and reusing code is independent of how many repos. I just don't see any concrete problem in the article that I can relate to that explains how what we do is more work, causes problems, can be optmimized.

Confusion 11y ago

About merging (I'll get back to the other bit later): if I want to push my change, I already regularly encounter the situation where git tells me: you are not up to date, even if I pulled say 15 minutes ago. So I have to fetch and rebase before I can push. Even if that doesn't require manually solving any merge conflicts (and I think I have to in about 1/4th of the cases), it takes a bit of time. In that bit of time another developer can push a commit. If that already happens on occasion with 10 developers, I imagine it becomes very problematic with 1000 developers on the same repo. Of course you effectively organise your repos in a 'tree' like e.g. happens with the Linux kernel, but, well, then you again have many repos. One per component that has someone responsible for signing off on them. It reduces the amount of people committing to each specific repo, but in the end, it's still many repos. So that's not 'one monolithic repo' in my book.

teraflop 11y ago
Still, that's a limitation of the specific tools and the way you're using them, not a fundamental problem with monolithic repositories. You're running into problems because you have multiple people all trying to push to the same branch, and Git errs on the side of extreme caution rather than creating a merge that you didn't specifically request. If you use merge requests instead, as supported by Github/Gitlab, you don't get blocked by other people's commits. (Actual merge conflicts are an orthogonal problem. They'll happen whenever you have multiple people editing the same code, regardless of what VCS you use or how it's organized.) The Linux kernel example is a bit of a tricky one. Yes, kernel.org hosts a lot of repos, but they're all different versions of the same codebase. They have a common commit ancestry, and commits get merged from one to another. So they're not really separate modules in the sense that we're talking about; they're more like namespaces that each contain a set of branches.
- Confusion 11y ago
  I guess it comes down to organisational style. Using merge requests requires some benevolent dictator (poor sod?) to perform the merges. For us, if your commit is approved in Gerrit, it's your task to merge it. If some other commit was merged in between, you have to rebase, possibly solving merge conflicts. So guess that with a monolithic repo, that wouldn't scale, but it could be made to scale by appointing someone to perform the merge requests. I'm not sure I would like to be that person...
  - Mathiasdm 11y ago
    I helped set up such a system for a few hundred developers. We had an 'automated Linus Torvalds', which did the merge, and aborted whenever a file was changed on both sides of the merge. In the good case (almost every time), there were no conflicts, and the merge went fine (we had unittests, builds and regression tests as extra checks in our CI system). In the bad case, the developer request was rejected and the developer was told to rebase or merge his code on his own, so the merge issues would be handled.

hvidgaard 11y ago

> it becomes impossible to describe the state of your deployed code without listing the version of every single dependency...If everything lives in the same tree, then a single commit ID reproducibly describes a complete system from top to bottom. This is a tooling problem, that is somewhat solved with a gigantic mono repo. All of our shared components are published with NuGet, and referenced as such. There is no interrepo dependence, and there shouldn't be.

Confusion 11y ago

The problems arise when you have to combine code from different repositories into a single deployable product. OK, we only have a single deployable product, which combines code from all those repositories, so it seems we match this criterion. But when you store those libraries in separate repositories, it becomes impossible to describe the state of your deployed code without listing the version of every single dependency. Yes, and that is what we do: each released lib has a version. The deployed version only uses released libs and lists exactly what it uses. The libs cannot be overwritten or changed after release: once released, that's what that version of the lib is. [..] If everything lives in the same tree, then a single commit ID reproducibly describes a complete system from top to bottom. In our case a single commit ID also reproducibly describes what has been deployed. Each library version was built from a specific commit ID in their respective repos. And you can atomically make changes that cross module boundaries, which is difficult to do safely with separate repositories. We don't have to atomically change anything, because a new backwards incompatible release of a certain lib doesn't affect any other lib, until we explicitly opt to upgrade it to the new version and release a new version. And if it's backwards compatible, it gets automatically picked up, because version management allows it. So I just don't recognize any of: I don't want to waste time dealing with multiple commands to manage multiple repositories. I don't want to waste time or expend cognitive load dealing with submodule, subrepository, or big files management. I don't want to waste time trying to find and reuse code, data, or documentation. I want everything at my fingertips, where it can be easily discovered, inspected, and used. I have a directory with a tree of repos. They could be in one big repo, but aren't. Otherwise, on disk, everything is the same. If I change things in two related libs, I have to commit twice, but I'm also changing two independent libs that I have to release separately anyway, just like in a monolithic repo. Finding and reusing code is independent of how many repos. I just don't see any concrete problem in the article that I can relate to that explains how what we do is more work, causes problems, can be optmimized.