7 ms·

Large-scale refactorings are actually pleasant. My team owns a framework and set of libraries that are widely used within the Google monorepo. We confidently

Large-scale refactorings are actually pleasant. My team owns a framework and set of libraries that are widely used within the Google monorepo. We confidently forward-update user code and prune deprecated APIs with relative ease — with benefits of doing it staged or all-at-once atomically. It's imperfect, but maintenance in distributed repositories is infinitely worse. Still, I remember the earlier days of the monorepo and keeping Perforce client file maps; that was a pain! https://www.perforce.com/perforce/r15.1/manuals/dvcs/_specify_mappings.html https://www.perforce.com/perforce/r15.1/manuals/dvcs/_specif...

InclinedPlane 9y ago
I agree, though I wouldn't say this is a fundamental aspect to distributed systems, mostly just a consequence of git being built with terrible merge and merge conflict tooling.
nickm12 9y ago
Infinitely worse? That's a bit hyperbolic.
- coderdude 9y ago
  Not just a bit hyperbolic. Rather, obviously hyperbolic. You should ask yourself why that's not obvious to you.
Roritharr 9y ago
How many people work on the repository tooling at Google? I'm asking because i wouldn't know how to setup a mono repository at my 50 people Startup even if we deemed this to be necessary.
- coderdude 9y ago
  (Not to mention the invaluable nature of this information to anyone working in that sector.)
- izacus 9y ago
  For a 50 people startup, a Git repository will be usually enough. At my previous company we managed to do the monorepo approach easily with similar amount of people and GitHub.
- rifung 9y ago
  > I'm asking because i wouldn't know how to setup a mono repository at my 50 people Startup even if we deemed this to be necessary. Sorry if this is a really dumb question. If you only have 50 people I'm assuming your codebase isn't that big, so why can't you just make a repo, make a folder for each of your existing repos, and put the code for those existing repos into the new repo? I imagine there's a way to do it so that your history remains intact as well.
  - aurelianito 9y ago
    Yes, there is. Move the entire content of each repo to a directory and then force-merge them all in a single repo. I did this a few years ago with 4 small mercurials repositories that belonged together.
  - fla 9y ago
    You can do that easly with a subtree
nradov 9y ago
I attended a talk by one of the Google Guava (Java collections library) authors and he told us how they didn't have to worry about maintaining backward compatibility at all. When they made a breaking change they could check out all of the impacted Java code across Google, refactor it, verify that the tests still passed, and then commit everything in one shot. It's easy to understand the productivity advantages.
- matttproud 9y ago
  One challenge is latency in the generation of codebase and identifier and callgraph search index (Cf., Code Search and Kythe). We can perform global tests across the entire monorepo, but that takes time. What happens if someone introduced new usage of old API immediately before our atomic refactoring, and what about pathological tests or flakes? This still necessitates doing some cleanups multi-stage: (1.) mark old API as deprecated (optionally announce), (2.) replace and delete legacy usages, and (3.) deletion of final trailing usage sometime soon thereafter once the codebase has been reindexed. Some languages and ecosystems are more tolerant of this problem than others. That said, incremental cleanup still has advantage with bisecting regressions. As I said, it is not perfect but broadbased change quickly is relatively easy. In my time maintaining open source, I never had these luxuries, which is why I said the monorepo is infinitely easier. Another consequence: if global cleanups are easy, perhaps that reduces the barrier to experimentation. Perfect is no longer the enemy of the good and the good enough. For me, I felt in open source where I had zero control over dependent code and its callgraph, the reverse was true: hesitance to publish something for fear of cost.
- user5994461 9y ago
  Facebook gave a conference not long ago about then doing the same thing. Interestingly, they only do that for java code. Java has good analysis and refactoring tools.
  - danabramov 9y ago
    >Interestingly, they only do that for java code. Not sure what you mean. I work at Facebook, and can confirm we keep all code in a monorepo (or, rather, one of two big monorepos) rather than just Java code. This lets us easily do React API changes: we can deprecate an API internally, and update all JS code that references the old APIs in a single commit.
Aissen 9y ago
You can do that with a many-repo, provided you have the right tooling. In fact, I'd argue Google's advantage is in the tooling they built around the repo, not the monorepo itself. E.g how fast you can find all your dependents in the whole repo.
- boulos 9y ago
  Not really, as there isn't the ability to atomically commit your changes. With 70000+ full-time employees code is getting checked in all the time. Atomicity is extremely valuable.
  - yoz-y 9y ago
    You can. You have many repos and one top level repository with submodules. Edit: And although this is a multi step process, it still allows you to de-couple modules and work on them separately.
  - kadenshep 9y ago
    It's like the people commenting about this forget what distributed means. You can have multiple repos, but you can still have a gate way/"source of truth" repo. You can run tests and whatever else on it just like you do in a "monorepo." The power behind Google's/Facebook's choice isn't and will never be the mono repo. It's specifically the tooling they built around their choice.
  - sah2ed 9y ago
    The 70,000+ number you cited includes engineers and non-engineers but surely, the actual number of engineers that need commit access will be much lower.
Groxx 9y ago
What changes can you do that you couldn't have done before, in a multirepo world? And I do mean could not have done - clearly the monorepo enjoys a few hundreds of thousands (millions?) of hours of effort that the multirepo did not. i.e. what stops your current tools from `for each repo, run...`, or how is monorepo fundamentally more capable than building automated library management / releases / etc with the same level of tooling?
- londons_explore 9y ago
  With a single commit you can change an API, and all it's users, and run all the tests for all the dependant projects, etc, and you're done. All in a days work, and no emails/communication necessary. In a multi-repo world, people are probably linking against old revisions of your library, and against certain tags/branches etc. There is probably no overarching code search to find all users of the API. You're gonna have to grep the code and hope to find all uses. You might miss some repos/branches. Everyone has their own continuous integration/testing procedures, so you can't easily migrate their code for them. You're gonna have to support both API's for probably months until you have persuaded every other user to upgrade to the latest 'release' of your code which supports the new API before finally turning off the old API. The work involved in the migration is spread amongst all the project owners, which is probably much less efficient. As others have said, it's the fully integrated version consistent codesearch with clickable xrefs across gigabytes of source code, cross repo code review, cross-repo testing, etc. which really makes a monorepo work well.
  - Groxx 9y ago
    (edit: shortened dramatically. apologies, earlier wasn't all that useful.) With the exception of cross-repo code review (I hadn't thought of that one - would be useful for multi-repo too, but I've honestly never seen a multi-repo tool for this, thanks!), this is all just the benefits of standardization, plus a massive injection of tooling enabled by the standards. Standardization of projects brings huge benefits when it's done right, absolutely agreed. But that's entirely orthogonal to mono vs multi.

InclinedPlane 9y ago

I agree, though I wouldn't say this is a fundamental aspect to distributed systems, mostly just a consequence of git being built with terrible merge and merge conflict tooling.

nickm12 9y ago

Infinitely worse? That's a bit hyperbolic.

coderdude 9y ago
Not just a bit hyperbolic. Rather, obviously hyperbolic. You should ask yourself why that's not obvious to you.

Roritharr 9y ago

How many people work on the repository tooling at Google? I'm asking because i wouldn't know how to setup a mono repository at my 50 people Startup even if we deemed this to be necessary.

coderdude 9y ago
(Not to mention the invaluable nature of this information to anyone working in that sector.)
izacus 9y ago
For a 50 people startup, a Git repository will be usually enough. At my previous company we managed to do the monorepo approach easily with similar amount of people and GitHub.
rifung 9y ago
> I'm asking because i wouldn't know how to setup a mono repository at my 50 people Startup even if we deemed this to be necessary. Sorry if this is a really dumb question. If you only have 50 people I'm assuming your codebase isn't that big, so why can't you just make a repo, make a folder for each of your existing repos, and put the code for those existing repos into the new repo? I imagine there's a way to do it so that your history remains intact as well.
- aurelianito 9y ago
  Yes, there is. Move the entire content of each repo to a directory and then force-merge them all in a single repo. I did this a few years ago with 4 small mercurials repositories that belonged together.
- fla 9y ago
  You can do that easly with a subtree

nradov 9y ago

I attended a talk by one of the Google Guava (Java collections library) authors and he told us how they didn't have to worry about maintaining backward compatibility at all. When they made a breaking change they could check out all of the impacted Java code across Google, refactor it, verify that the tests still passed, and then commit everything in one shot. It's easy to understand the productivity advantages.

matttproud 9y ago
One challenge is latency in the generation of codebase and identifier and callgraph search index (Cf., Code Search and Kythe). We can perform global tests across the entire monorepo, but that takes time. What happens if someone introduced new usage of old API immediately before our atomic refactoring, and what about pathological tests or flakes? This still necessitates doing some cleanups multi-stage: (1.) mark old API as deprecated (optionally announce), (2.) replace and delete legacy usages, and (3.) deletion of final trailing usage sometime soon thereafter once the codebase has been reindexed. Some languages and ecosystems are more tolerant of this problem than others. That said, incremental cleanup still has advantage with bisecting regressions. As I said, it is not perfect but broadbased change quickly is relatively easy. In my time maintaining open source, I never had these luxuries, which is why I said the monorepo is infinitely easier. Another consequence: if global cleanups are easy, perhaps that reduces the barrier to experimentation. Perfect is no longer the enemy of the good and the good enough. For me, I felt in open source where I had zero control over dependent code and its callgraph, the reverse was true: hesitance to publish something for fear of cost.
user5994461 9y ago
Facebook gave a conference not long ago about then doing the same thing. Interestingly, they only do that for java code. Java has good analysis and refactoring tools.
- danabramov 9y ago
  >Interestingly, they only do that for java code. Not sure what you mean. I work at Facebook, and can confirm we keep all code in a monorepo (or, rather, one of two big monorepos) rather than just Java code. This lets us easily do React API changes: we can deprecate an API internally, and update all JS code that references the old APIs in a single commit.

Aissen 9y ago

You can do that with a many-repo, provided you have the right tooling. In fact, I'd argue Google's advantage is in the tooling they built around the repo, not the monorepo itself. E.g how fast you can find all your dependents in the whole repo.

boulos 9y ago
Not really, as there isn't the ability to atomically commit your changes. With 70000+ full-time employees code is getting checked in all the time. Atomicity is extremely valuable.
- yoz-y 9y ago
  You can. You have many repos and one top level repository with submodules. Edit: And although this is a multi step process, it still allows you to de-couple modules and work on them separately.
- kadenshep 9y ago
  It's like the people commenting about this forget what distributed means. You can have multiple repos, but you can still have a gate way/"source of truth" repo. You can run tests and whatever else on it just like you do in a "monorepo." The power behind Google's/Facebook's choice isn't and will never be the mono repo. It's specifically the tooling they built around their choice.
- sah2ed 9y ago
  The 70,000+ number you cited includes engineers and non-engineers but surely, the actual number of engineers that need commit access will be much lower.

Groxx 9y ago

What changes can you do that you couldn't have done before, in a multirepo world? And I do mean could not have done - clearly the monorepo enjoys a few hundreds of thousands (millions?) of hours of effort that the multirepo did not. i.e. what stops your current tools from `for each repo, run...`, or how is monorepo fundamentally more capable than building automated library management / releases / etc with the same level of tooling?

londons_explore 9y ago
With a single commit you can change an API, and all it's users, and run all the tests for all the dependant projects, etc, and you're done. All in a days work, and no emails/communication necessary. In a multi-repo world, people are probably linking against old revisions of your library, and against certain tags/branches etc. There is probably no overarching code search to find all users of the API. You're gonna have to grep the code and hope to find all uses. You might miss some repos/branches. Everyone has their own continuous integration/testing procedures, so you can't easily migrate their code for them. You're gonna have to support both API's for probably months until you have persuaded every other user to upgrade to the latest 'release' of your code which supports the new API before finally turning off the old API. The work involved in the migration is spread amongst all the project owners, which is probably much less efficient. As others have said, it's the fully integrated version consistent codesearch with clickable xrefs across gigabytes of source code, cross repo code review, cross-repo testing, etc. which really makes a monorepo work well.
- Groxx 9y ago
  (edit: shortened dramatically. apologies, earlier wasn't all that useful.) With the exception of cross-repo code review (I hadn't thought of that one - would be useful for multi-repo too, but I've honestly never seen a multi-repo tool for this, thanks!), this is all just the benefits of standardization, plus a massive injection of tooling enabled by the standards. Standardization of projects brings huge benefits when it's done right, absolutely agreed. But that's entirely orthogonal to mono vs multi.