4 ms·

I've worked with monolithic and project-based repositories, and in practice I've found the problems with monolithic repos are less than the problems with projec

by wtbob 11y ago

I've worked with monolithic and project-based repositories, and in practice I've found the problems with monolithic repos are less than the problems with project-based repos, and the benefits of monolithic repos are more than the benefits of project-based repos. Certainly there can be issues at a very large organisation with very many extremely large projects—but most of us don't work at those organisations with that many projects that large. I think that having one large repo helps identify cross-project dependency breakage faster, e.g. on a small team without fully-automated integration tests by increasing the likelihood that the person or team who broke the integration notices rather than the person or team who maintains the affected components. There's also the issue, as jacques_chester notes, of shared components, some of which are far too small too small the be their own projects and which don't necessarily make sense thrown into a pile with other projects. Project-specific repos make a lot of sense from an organised, a-place-for-everything-and-everything-in-its-place perspective, but real life is often quite messy and mutable, and the proper organisation for a project can change frequently (as the article notes); there's no sense chiselling it into stone.

viraptor 11y ago
I think there's a missing distinction here: is it a monolithic repo for products available separately, or monolithic final deployable thing(s) (google has multiple products, but it's still "google services"). This may be better for them, but it wouldn't work for example for OpenStack, which has many projects available and released separately. Putting nova, stevedore, anchor, bandit, etc. in one repository just wouldn't make sense - they have their own versioning and live their own lives, even if they will be frozen/released in one go as a single working deployment in October. So when the author writes "When I am interacting with version control, I just want to get stuff done. I don't want to waste time dealing with multiple commands to manage multiple repositories." they just don't have that use case. They don't have to care about releasing different bits separately.
- Nitramp 11y ago
  Google has self driving cars, YouTube, various Android apps, and web search in the same repository. That does seem a lot more varied at the first look than e.g. OpenStack, and different products certainly have extremely different release cycles and even deployment platforms (hardware in cars, Android phones, web servers). The question really is whether you can scale your version control practices, build tools, and source organization habits across many diverse projects.
  - viraptor 11y ago
    Do they really store youtube, self driving car and Android in the same repo? As far as I can tell, at least android bits live in many repositories: https://android.googlesource.com/ https://android.googlesource.com/ - I'd say over 300, with one for each utility.
    - deleted 11y ago
      [deleted]
    - Confusion 11y ago
      Exactly. I frankly have no clue what this article is about. I work at a company with < 10 developers. We started with one repo and now have over 20 repositories for various bits and pieces of our code. Each one maps to its own releasible component. I don't recognize the 'I don't want to waste time dealing with multiple commands to manage multiple repositories' at all. The only time there is a difference on the cli is when you clone a repo. If anything about this would hurt, we'd change it: optimizing our workflow is something we pay a lot of attention to. In fact, with hundreds of developers and everything in one repo, I don't see how you'd ever be able get a commit through : you'd be merging commits that others just did all the time and would have to get lucky?
      - teraflop 11y ago
        The issue with multiple repositories has nothing do to with the number of commands you have to run. As you say, that's the kind of thing that can easily be automated. The problems arise when you have to combine code from different repositories into a single deployable product. Most of us don't take Amazon's hard-line stance of making absolutely everything a microservice, so we end up with libraries of reusable code that are referenced by multiple projects. But when you store those libraries in separate repositories, it becomes impossible to describe the state of your deployed code without listing the version of every single dependency. That makes it easy for subtle inconsistencies and bugs to creep in, especially when the dependencies are multiple levels deep and are owned by different teams. If everything lives in the same tree, then a single commit ID reproducibly describes a complete system from top to bottom. And you can atomically make changes that cross module boundaries, which is difficult to do safely with separate repositories. I don't really follow your comment about merging. Pretty much every version control system since forever has been smart enough to realize that, if I make changes only to foo/src/ and you make changes to bar/src/, our changes don't conflict and can be merged automatically without user intervention. (There might be technical difficulties; for example, if you're using Git, I would imagine that trying to view the list of commits of a small subtree of a gigantic repo might not be terribly efficient. But just like the issue of managing multiple repos, that's something that you can solve with better tool support, if you really need to and are motivated enough.)

by wtbob 11y ago

viraptor 11y ago

I think there's a missing distinction here: is it a monolithic repo for products available separately, or monolithic final deployable thing(s) (google has multiple products, but it's still "google services"). This may be better for them, but it wouldn't work for example for OpenStack, which has many projects available and released separately. Putting nova, stevedore, anchor, bandit, etc. in one repository just wouldn't make sense - they have their own versioning and live their own lives, even if they will be frozen/released in one go as a single working deployment in October. So when the author writes "When I am interacting with version control, I just want to get stuff done. I don't want to waste time dealing with multiple commands to manage multiple repositories." they just don't have that use case. They don't have to care about releasing different bits separately.

Nitramp 11y ago
Google has self driving cars, YouTube, various Android apps, and web search in the same repository. That does seem a lot more varied at the first look than e.g. OpenStack, and different products certainly have extremely different release cycles and even deployment platforms (hardware in cars, Android phones, web servers). The question really is whether you can scale your version control practices, build tools, and source organization habits across many diverse projects.
- viraptor 11y ago
  Do they really store youtube, self driving car and Android in the same repo? As far as I can tell, at least android bits live in many repositories: https://android.googlesource.com/ https://android.googlesource.com/ - I'd say over 300, with one for each utility.
  - deleted 11y ago
    [deleted]
  - Confusion 11y ago
    Exactly. I frankly have no clue what this article is about. I work at a company with < 10 developers. We started with one repo and now have over 20 repositories for various bits and pieces of our code. Each one maps to its own releasible component. I don't recognize the 'I don't want to waste time dealing with multiple commands to manage multiple repositories' at all. The only time there is a difference on the cli is when you clone a repo. If anything about this would hurt, we'd change it: optimizing our workflow is something we pay a lot of attention to. In fact, with hundreds of developers and everything in one repo, I don't see how you'd ever be able get a commit through : you'd be merging commits that others just did all the time and would have to get lucky?
    - teraflop 11y ago
      The issue with multiple repositories has nothing do to with the number of commands you have to run. As you say, that's the kind of thing that can easily be automated. The problems arise when you have to combine code from different repositories into a single deployable product. Most of us don't take Amazon's hard-line stance of making absolutely everything a microservice, so we end up with libraries of reusable code that are referenced by multiple projects. But when you store those libraries in separate repositories, it becomes impossible to describe the state of your deployed code without listing the version of every single dependency. That makes it easy for subtle inconsistencies and bugs to creep in, especially when the dependencies are multiple levels deep and are owned by different teams. If everything lives in the same tree, then a single commit ID reproducibly describes a complete system from top to bottom. And you can atomically make changes that cross module boundaries, which is difficult to do safely with separate repositories. I don't really follow your comment about merging. Pretty much every version control system since forever has been smart enough to realize that, if I make changes only to foo/src/ and you make changes to bar/src/, our changes don't conflict and can be merged automatically without user intervention. (There might be technical difficulties; for example, if you're using Git, I would imagine that trying to view the list of commits of a small subtree of a gigantic repo might not be terribly efficient. But just like the issue of managing multiple repos, that's something that you can solve with better tool support, if you really need to and are motivated enough.)