7 ms·

Monorepos have the big advantage that cross-project changes can be made together in one PR.

bottled_poe 4y ago
This is the correct answer. Putting all integration changes in a single PR improves CI test coverage and is easier to verify in a code review.
- XorNot 4y ago
  This really begs the question why Github aren't working on cross-repo PR support - or anyone. Everytime mono-repos come up, cross-repo PRs are the reason given as to why they're needed - why can't we get this supported well?
  - mattnewton 4y ago
    That’s not just a GitHub issue, that would require serious UX work on the authoring side to make such “commits” that are actually commits in multiple repos at the same time. Effectively this is recreating a monorepo (maybe with something like for submodules?)
    - treis 4y ago
      I don't think it's recreating a monorepo. What you're looking for is something like a DB transaction where all the merges in different repos succeed or everything gets rolled back.
      - mattnewton 4y ago
        Maybe I am missing something: to do that you need some kind of “meta-repository” linking commits between all the different sub repos in some order, no? If that happens for a significant portion of the commits in a repo, isn’t that basically equivalent to a mono repo with git submodules or something?
        treis 4y ago
        Well a meta repo isn't a mono repo. In that situation you can still update individual repos. Analogous to running a DB operatioj without a transaction. But that said, no I don't think you need a meta repo. You need something. Call it a Change set. MVP would be say merge commits in 4 repos. If any fail then revert. If they all succeed then deploy in a specified order.
  - dalyons 4y ago
    Sourcegraph does a decent job of this with their “batch changes” feature
BenoitEssiambre 4y ago
It also helps keep people in sync with module and library versions. Without a monorepo, oftentimes what happens is that everybody ends up on different versions of libraries and modules. For every security update to a library, you'll have to campaign for everyone to pull that fix instead of everyone getting all fixes on every pull. It also forces dealing with incompatibilities between libraries early on when they happen instead of later when they are more entrenched and difficult to fix.
- spion 4y ago
  That can be handled with alerts and deprecations just like any other library dependencies
  - idontknowifican 4y ago
    no it can’t, an alert won’t make a PM allocate time to “tech debt” and these alerts are rarely time free upgrades
    - spion 4y ago
      If you have an issue handling tech debt due to PMs, you'll have that problem with other tech debt too; typically solved by having a better process to handle it
- deepsun 4y ago
  What I've seen in practice in smaller companies, is actually the other way around: everyone is afraid updating any library version because who knows what other team's work it might break. Theoretically, everyone must have extensive tests to cover everything, but in practice it's hard. But when your team owns a repo, then at least the damage is contained within your team. Google solves that problem with heavy NIH syndrome (its hard to get promoted by utilizing an external third-party lib, better develop your own), and writing tests, yes. And for those few third-party libraries that google still depends on, updating them is a big PITA.
  - 4y ago
spion 4y ago
A well factored system should not have cross-project changes. Individual projects should be able to upgrade dependencies separately.
- idontknowifican 4y ago
  can you provide more info here? i’m finding it hard to think of a system at a large company that would have no internal libraries used by multiple projects, that wouldn’t require ever making a cross project change.
  - spion 4y ago
    Changes will be handled just like external library updates.
    - idontknowifican 4y ago
      i suspect your internal model of a monorepo is fairly disjoint with an actual implementation. you’re arguing for non-monorepo based version concepts, without giving space to why having everyone locked to “latest” has large cost savings long term for maintenance and security. without those constraints, a monorepo is a bad choice.
      - spion 4y ago
        I've worked with both models, and the monorepo model has been far more demanding in terms of time and resources to maintain as you push past the limits of the available tools one by one. Standard build tools go out of the window almost immediately (lots of work to wrap everything with Bazel), then standard collaboration tools (GH) and so on. A dedicated tooling team becomes a must. What helps maintain polyrepos sane 1. A healthy methodology for dependency management Evolve APIs with deprecations. Decide on a healthy amount of time for a deprecations to live. Set up alerts when deprecations reach certain amount of time. Set up dashboards (e.g. Grafana) to track dependencies. Help other teams update, and think about how to make updates less painful. 2. Define good boundaries and API contracts to adhere to. The most important thing for a good API contract is stability. 3. Don't prematurely split into microservices. Better ideas for stable API contracts emerge the longer you can wait.
barrkel 4y ago
This is almost the only advantage. A related "advantage" is forced upgrades when dependencies change - since you can't have code depending on different versions of in-repo libraries and services, client code needs to adapt or die. The more clients a library or service has, the more expensive this is, and it's an ongoing maintenance cost for every client. Since people changing the service don't feel the full pain of this maintenance cost, changes keep on happening, and eventually clients get culled because it's too expensive to keep on maintaining them all. From the outside, this looks like the company abandoning venerable but still working product, and makes people scratch their heads, wondering why.
- ithkuil 4y ago
  You can have a monorepo without having all the modules share the dependencies. For example you can have multiple go.mod files in a single monorepo. People often conflate these two things because often teams actually want to have centralized dependencies (so that you're forced to update or die as you said). If that doesn't work for you you can choose to have modules (or groups of modules) keep their independent set of dependencies, all while keeping the code in the monorepo.
prestoooooo 4y ago
Pros * Breaking changes can be done at once. Very helpful for runtime deps. * No chance that a repo is out of date. * Upgrades are atomic (may be hard to test a system in a half state).
- erik_seaberg 4y ago
  While you’re deploying, production is going to be in a half-upgraded state (or maybe half rolled-back!), so it’s pretty important to be able to test that.
  - iambvk 4y ago
    This depends on the deployment model. I am curious, how does your release and deployment looks like?
    - erik_seaberg 4y ago
      Release is a privileged tool pushing to Git. Deployment is incremental across a small but growing number of containers running a microservice, so both old and new code might be running concurrently for up to a couple of hours. Percentage experiments are pretty common, which means both old and new paths actually have to work in the same commit.
likortera 4y ago
Then why not call it just a repo and build a monolith? I've seen people going crazy with this bullshit of monorepos to the point every single directory was a "package" when it could be just a plain module import. If you want to do microservices, then putting everything back into a single repository and enforcing everyone to use the same version and every change to require upgrades and deploys across the board is totally backwards. Either do a monolith and have that consistency, or do microservices and allow teams to follow their own rules as long as they keep APIs stable. Nonorepos + microservices is just a demonstration of everything that's wrong in the technical aspects of our industry. Just applying absolutely everything you read about without even considering if it might be better or not for your specific use case.
- oftenwrong 4y ago
  The way you store your source code doesn't have much to do with how your system is deployed. It's the same way you can have a closet in which you store clothing for many purposes; clothing for cold weather, clothing for hot weather, clothing for formal occasions, clothing for specific recreational activities, et cetera. You can store code for different purposes in the same source tree. You can deploy something that is built from just one part of the source tree.
  - skytreader 4y ago
    > You can store code for different purposes in the same source tree. You can deploy something that is built from just one part of the source tree. At the risk of sounding memetic, the question is not so much "Can you?" but "Should you?". Should you store a monolithic application across multiple repositories? Should you store a distributed application in a monolithic repository? I agree with @likortera that you should not. And that's because... > The way you store your source code doesn't have much to do with how your system is deployed. I disagree with this statement. Your architecture (monolithic vs distributed) imposes certain assumptions on other aspects of your distribution pipeline. Your workflow (in this discussion "how you store your source code") should support these assumptions, not hinder them. For example, one benefit of microservices is that cross-functional teams can develop independently of each other. And yet one cited advantage of monorepos is that everyone is on the same version of dependencies all the time. In short, your teams are not independent after all. Note that I'm keeping the example extremely generic to illustrate this inconsistency, a conflict of interest if you will, that I see people commit in this topic, because in my experience, these questions are not purely technical but involves product/business factors as well. Maybe for most of the people (operative emphasis on "MAYBE", because who am I to judge you), the discussion they need to have first is whether or not they are using the right architecture for their product in the first place. If you choose to have a microservices architecture, you have to live with the fact that your teams/services will operate at different cadences. If you feel the need to impose a One True Library Version All the Time, then go for a monolithic architecture, and store your code in the same way.
kitd 4y ago
The problem isn't per se monorepo v multi-repo, it's ensuring that your lines of communication between components, ie your APIs, are shared reliably and coherently. When teams take the stability & versioning of their APIs seriously, the need to use monorepos to share that info is greatly reduced. A multi-repo approach is perfectly feasible when all components are working to established APIs, which also alleviates the issues mentioned in the article.
eikenberry 4y ago
Large cross project changes mean you have a monolithic project. One project split over multiple repos, not multiple projects. If you're separate projects were really separate they'd maintain API compatibility and upgrade paths (eg. semantic versioning). SaaS and microservice architectures require this to maintain separation but most organizations lack the discipline and slowly revert back to a monolith as they are faster to develop (until they aren't).

This is the correct answer. Putting all integration changes in a single PR improves CI test coverage and is easier to verify in a code review.

XorNot 4y ago
This really begs the question why Github aren't working on cross-repo PR support - or anyone. Everytime mono-repos come up, cross-repo PRs are the reason given as to why they're needed - why can't we get this supported well?
- mattnewton 4y ago
  That’s not just a GitHub issue, that would require serious UX work on the authoring side to make such “commits” that are actually commits in multiple repos at the same time. Effectively this is recreating a monorepo (maybe with something like for submodules?)
  - treis 4y ago
    I don't think it's recreating a monorepo. What you're looking for is something like a DB transaction where all the merges in different repos succeed or everything gets rolled back.
    - mattnewton 4y ago
      Maybe I am missing something: to do that you need some kind of “meta-repository” linking commits between all the different sub repos in some order, no? If that happens for a significant portion of the commits in a repo, isn’t that basically equivalent to a mono repo with git submodules or something?
      - treis 4y ago
        Well a meta repo isn't a mono repo. In that situation you can still update individual repos. Analogous to running a DB operatioj without a transaction. But that said, no I don't think you need a meta repo. You need something. Call it a Change set. MVP would be say merge commits in 4 repos. If any fail then revert. If they all succeed then deploy in a specified order.
- dalyons 4y ago
  Sourcegraph does a decent job of this with their “batch changes” feature

BenoitEssiambre 4y ago

It also helps keep people in sync with module and library versions. Without a monorepo, oftentimes what happens is that everybody ends up on different versions of libraries and modules. For every security update to a library, you'll have to campaign for everyone to pull that fix instead of everyone getting all fixes on every pull. It also forces dealing with incompatibilities between libraries early on when they happen instead of later when they are more entrenched and difficult to fix.

spion 4y ago
That can be handled with alerts and deprecations just like any other library dependencies
- idontknowifican 4y ago
  no it can’t, an alert won’t make a PM allocate time to “tech debt” and these alerts are rarely time free upgrades
  - spion 4y ago
    If you have an issue handling tech debt due to PMs, you'll have that problem with other tech debt too; typically solved by having a better process to handle it
deepsun 4y ago
What I've seen in practice in smaller companies, is actually the other way around: everyone is afraid updating any library version because who knows what other team's work it might break. Theoretically, everyone must have extensive tests to cover everything, but in practice it's hard. But when your team owns a repo, then at least the damage is contained within your team. Google solves that problem with heavy NIH syndrome (its hard to get promoted by utilizing an external third-party lib, better develop your own), and writing tests, yes. And for those few third-party libraries that google still depends on, updating them is a big PITA.
- 4y ago

spion 4y ago

A well factored system should not have cross-project changes. Individual projects should be able to upgrade dependencies separately.

idontknowifican 4y ago
can you provide more info here? i’m finding it hard to think of a system at a large company that would have no internal libraries used by multiple projects, that wouldn’t require ever making a cross project change.
- spion 4y ago
  Changes will be handled just like external library updates.
  - idontknowifican 4y ago
    i suspect your internal model of a monorepo is fairly disjoint with an actual implementation. you’re arguing for non-monorepo based version concepts, without giving space to why having everyone locked to “latest” has large cost savings long term for maintenance and security. without those constraints, a monorepo is a bad choice.
    - spion 4y ago
      I've worked with both models, and the monorepo model has been far more demanding in terms of time and resources to maintain as you push past the limits of the available tools one by one. Standard build tools go out of the window almost immediately (lots of work to wrap everything with Bazel), then standard collaboration tools (GH) and so on. A dedicated tooling team becomes a must. What helps maintain polyrepos sane 1. A healthy methodology for dependency management Evolve APIs with deprecations. Decide on a healthy amount of time for a deprecations to live. Set up alerts when deprecations reach certain amount of time. Set up dashboards (e.g. Grafana) to track dependencies. Help other teams update, and think about how to make updates less painful. 2. Define good boundaries and API contracts to adhere to. The most important thing for a good API contract is stability. 3. Don't prematurely split into microservices. Better ideas for stable API contracts emerge the longer you can wait.

barrkel 4y ago

This is almost the only advantage. A related "advantage" is forced upgrades when dependencies change - since you can't have code depending on different versions of in-repo libraries and services, client code needs to adapt or die. The more clients a library or service has, the more expensive this is, and it's an ongoing maintenance cost for every client. Since people changing the service don't feel the full pain of this maintenance cost, changes keep on happening, and eventually clients get culled because it's too expensive to keep on maintaining them all. From the outside, this looks like the company abandoning venerable but still working product, and makes people scratch their heads, wondering why.

ithkuil 4y ago
You can have a monorepo without having all the modules share the dependencies. For example you can have multiple go.mod files in a single monorepo. People often conflate these two things because often teams actually want to have centralized dependencies (so that you're forced to update or die as you said). If that doesn't work for you you can choose to have modules (or groups of modules) keep their independent set of dependencies, all while keeping the code in the monorepo.

prestoooooo 4y ago

Pros * Breaking changes can be done at once. Very helpful for runtime deps. * No chance that a repo is out of date. * Upgrades are atomic (may be hard to test a system in a half state).

erik_seaberg 4y ago
While you’re deploying, production is going to be in a half-upgraded state (or maybe half rolled-back!), so it’s pretty important to be able to test that.
- iambvk 4y ago
  This depends on the deployment model. I am curious, how does your release and deployment looks like?
  - erik_seaberg 4y ago
    Release is a privileged tool pushing to Git. Deployment is incremental across a small but growing number of containers running a microservice, so both old and new code might be running concurrently for up to a couple of hours. Percentage experiments are pretty common, which means both old and new paths actually have to work in the same commit.

likortera 4y ago

Then why not call it just a repo and build a monolith? I've seen people going crazy with this bullshit of monorepos to the point every single directory was a "package" when it could be just a plain module import. If you want to do microservices, then putting everything back into a single repository and enforcing everyone to use the same version and every change to require upgrades and deploys across the board is totally backwards. Either do a monolith and have that consistency, or do microservices and allow teams to follow their own rules as long as they keep APIs stable. Nonorepos + microservices is just a demonstration of everything that's wrong in the technical aspects of our industry. Just applying absolutely everything you read about without even considering if it might be better or not for your specific use case.

oftenwrong 4y ago
The way you store your source code doesn't have much to do with how your system is deployed. It's the same way you can have a closet in which you store clothing for many purposes; clothing for cold weather, clothing for hot weather, clothing for formal occasions, clothing for specific recreational activities, et cetera. You can store code for different purposes in the same source tree. You can deploy something that is built from just one part of the source tree.
- skytreader 4y ago
  > You can store code for different purposes in the same source tree. You can deploy something that is built from just one part of the source tree. At the risk of sounding memetic, the question is not so much "Can you?" but "Should you?". Should you store a monolithic application across multiple repositories? Should you store a distributed application in a monolithic repository? I agree with @likortera that you should not. And that's because... > The way you store your source code doesn't have much to do with how your system is deployed. I disagree with this statement. Your architecture (monolithic vs distributed) imposes certain assumptions on other aspects of your distribution pipeline. Your workflow (in this discussion "how you store your source code") should support these assumptions, not hinder them. For example, one benefit of microservices is that cross-functional teams can develop independently of each other. And yet one cited advantage of monorepos is that everyone is on the same version of dependencies all the time. In short, your teams are not independent after all. Note that I'm keeping the example extremely generic to illustrate this inconsistency, a conflict of interest if you will, that I see people commit in this topic, because in my experience, these questions are not purely technical but involves product/business factors as well. Maybe for most of the people (operative emphasis on "MAYBE", because who am I to judge you), the discussion they need to have first is whether or not they are using the right architecture for their product in the first place. If you choose to have a microservices architecture, you have to live with the fact that your teams/services will operate at different cadences. If you feel the need to impose a One True Library Version All the Time, then go for a monolithic architecture, and store your code in the same way.

kitd 4y ago

The problem isn't per se monorepo v multi-repo, it's ensuring that your lines of communication between components, ie your APIs, are shared reliably and coherently. When teams take the stability & versioning of their APIs seriously, the need to use monorepos to share that info is greatly reduced. A multi-repo approach is perfectly feasible when all components are working to established APIs, which also alleviates the issues mentioned in the article.

eikenberry 4y ago

Large cross project changes mean you have a monolithic project. One project split over multiple repos, not multiple projects. If you're separate projects were really separate they'd maintain API compatibility and upgrade paths (eg. semantic versioning). SaaS and microservice architectures require this to maintain separation but most organizations lack the discipline and slowly revert back to a monolith as they are faster to develop (until they aren't).