6 ms·

Monorepos

35 points by luu 11y ago

dlitz 11y ago
> Facebook, Google and Twitter Aren't these the companies all terrible at taking outside contributions, except for the projects that they don't manage in their monorepos?
- daleharvey 11y ago
  I see similiar issues at mozilla. Monorepos are far easier to work with from an engineering standpoint, for external contributors they are definitely prohibitive.
- shoo 11y ago
  Conway's Law for infrastructure / tooling, perhaps. If you have complete control over the organisation's dev process, and are willing to invest enough, you can build something better than the default free / distributed / popular tooling. This is probably partly enabled by the fact that they don't have to care about trading patches easily with the outside world. Then, by rolling their own in-house workflow/infrastructure/tooling, they make it harder for outsiders to contribute, even if this were organisationally possible and desirable.
- falcolas 11y ago
  Based on some brief experience at Oracle, accepting outside contributions is a legal nightmare - it is frequently less expensive to do a clean-room re-implementation of a feature than go through the lawyers and contract negotiations now and in the future to get code ownership rights worked out.
  - mcs_ 11y ago
    agree with this. I think monolithic repos are part of legal strategy in enterprise.
  - jsprogrammer 11y ago
    That's because it's Oracle. Everything associated with Oracle is a legal nightmare.
zokier 11y ago
> Require Trunk-Based Development This I think needs bit of expanding. Does he mean in contrast to feature-branches? Why monorepos would not be compatible with feature-branches?
- shoo 11y ago
  Not to answer your questions directly, but Paul Hammant has written a bunch about this over the years: http://paulhammant.com/2013/04/05/what-is-trunk-based-development/ http://paulhammant.com/2013/04/05/what-is-trunk-based-develo... http://paulhammant.com/2013/05/06/googles-scaled-trunk-based-development/ http://paulhammant.com/2013/05/06/googles-scaled-trunk-based... http://paulhammant.com/2014/04/03/microsofts-trunk-based-development/ http://paulhammant.com/2014/04/03/microsofts-trunk-based-dev... http://paulhammant.com/2014/01/06/googlers-subset-their-trunk/ http://paulhammant.com/2014/01/06/googlers-subset-their-trun... https://dzone.com/articles/legacy-app-rejuvenation https://dzone.com/articles/legacy-app-rejuvenation
  - zokier 11y ago
    I just read the first link (adding the others to my reading backlog), based on it it seems like trunk based development is less of a development model and more of a release model. Especially it does not seem to conflict directly with feature branch model.
    - shoo 11y ago
      Perhaps partly. But Paul also mentions "Developers do not break the build with any commit" -- that places a pretty heavy constraint upon regular development outside of releases. See also the so-called "not rocket science rule of software engineering": https://graydon.livejournal.com/186550.html https://graydon.livejournal.com/186550.html
      - speedkills 11y ago
        See also got-dmz or atlassian bamboo's gatekeeper feature. Team city has something similar too. Surprised the guy who wrote bors hadn't seen more examples, I have seen the idea come up over and over again (and strongly support it).
StavrosK 11y ago
Is it really a single big repo for the whole organization? What happens if you want to deploy your code on servers? Do you have to upload all the mobile app code and history as well? Doesn't the history get very polluted this way?
- NickNameNick 11y ago
  Are you really using a checkout on a production server to install and configure your software? This feels like a "Doctor, doctor - it hurts when I stick a spoon in my eye" moment - "Don't do that!" Build tools and build servers exist for a reason. Subversion has "svn export" if you insist on working that way. git can git clone repos --depth 1; rm -rf repos/.git if you really, really must. And what you are calling polluted history, they are calling efficient collaboration.
  - shoo 11y ago
    If there exists appropriate tooling (e.g. for filtering commit logs based on what touched the code belonging to a single project / what touched the code belonging to a single project and its transitive dependencies) then I dare say a huge monorepo history over many projects and libraries isn't polluted history, it is remarkably rich and useful history. In big places with solid engineering teams and a long-term investment in infrastructure it seems achievable to build the tools to do this properly. (famously: google, which isn't a very fair example). I think many people's default reaction to monorepos -- fear, loathing and disbelief, say -- is based upon working with open source tools / related cloud services / distributed workflows - that scale poorly when dealing with large repositories, at least with default configuration and without a lot of mucking about. That said, this default reaction is arguably a healthy reaction if the reality is that most teams / organisations do not have the resources to invest in setting up tooling / infrastructure beyond what they can get for free / rent for a few hundred dollars a month.
  - StavrosK 11y ago
    > Build tools and build servers exist for a reason. What's the build server going to do? Copy the Python code from the repo to the other server? It doesn't seem like setting up a build server just to do "git checkout; rm .git" is worth the effort.
fsloth 11y ago
The key is enabling maximum developer leverage - and one aspect of this is to get rid of as many systems errors and inefficiencies as possible. I think build systems and internal code management schemes are large interconnected systems. It feels fairly pointless just to expose one facet of the entire systems since the total system encompasses all of the individual entities dealing with the codebase - including developers, platform - the whole works. For example, I see no overhead in my daily work from the fact that our codebase is split to several libraries with dependencies through explicit built release versions. On the contrary,there monorepository would be just horrible. So, before saying X is great one should firsr define what type of software is built, what is the specific toolchain, what are the specific configurations, how many developers modify the same codebase frequently, and so on.
- beberlei 11y ago
  hi, author of those slides here, I actually mention this during the talk, that the talk is over-emphesizing the awesome point to take a point across. It definately is not a technique to use everywhere, so I agree with you. It works for some kind of projects and i was motiviated for this talk, because I saw so many projects using manyrepos in cases where it may not fit very well.
  - fsloth 11y ago
    Hi, yeah, sure, effective communication requires simplification. A few points got me concerned in the slides - namely, concerning emphasizing how easy this makes to change things. I would consider that it is not a positive thing that adding a library is cheap and easy, and I find that component versioning can be bypassed even more so. It all sounds like grievous local greedy optimization without a view of the entire lifecycle of the codebase. The slides say change is the only constant. I would claim that unless the change is backed by good and strong architecture - which throwing all architectural firewalls away does not support - then the end result will be a mess of spaghetti the older the codebase gets. But, I may just be too much set in my ways of coding and I respect the fact that these may be non-issues in the particular domain these slides address.
bsimpson 11y ago
Everyone I've ever met who's worked at Twitter hates the monorepo. They have a single git repo with way more files than that tool was ever designed to maintain. As such, normally instantaneous commands like status, log, ls, etc. take an inordinately long amount of time to execute. Apparently their onboarding process is "have someone copy his working copy to a flash drive, then copy it from there to your laptop" because git pull would take too long.
- beberlei 11y ago
  interesting, i can incorporate this into future versions of the talk about pain points developers might have. The list of users is value free, it is just a mention of companies I know using it. Both the Google and FB talks about this mention the technical challenges, so i guess this is really a downside at very big scales.
- indygreg2 11y ago
  Twitter is maintaining an internal fork/distribution of Git with patches to attempt to make it scale to their needs. It is described a bit at http://git-merge.com/videos/scaling-git-at-twitter-wilhelm-bierbaum.html http://git-merge.com/videos/scaling-git-at-twitter-wilhelm-b.... But even then, as you mention, Git doesn't scale well for them. This is why Facebook is using Mercurial. This is why the Chromium and Android projects are spread out over hundreds of Git repos. This is why Google's internal monorepo is using a Perforce-like homegrown tool backed by Google's distributed wizardry. Git will start to see significant scaling problems beyond say 1 GB total data, 100,000 files, and X number of refs (there has been significant upstream work around scaling refs - a bunch from Twitter I believe - and I haven't measured how well newer versions handle thousands of refs). The good news is that unless you are a software project with hundreds of developers, have large binary assets, or are doing something like Twitter and have all your developers push all their in-progress feature branches to the monorepo (madness if you ask me), a Git monorepo will scale for you. A Mercurial monorepo will scale even better. Keep in mind that most people aren't {Twitter, Facebook, Google, Mozilla}, so chances are you won't ever reach these scaling limits.
- neilk 11y ago
  This has to be a sick joke. The monorepo at Google was the bane of my existence, and everyone else that I knew. I don't know what it's like now, but at the time Google was on svn, and there had to be a specially hacked version of it. It used a home-grown internal requirements system, to pull down the tree in a sparse way. And to accomplish that you had to re-specify all your dependencies in these special text files (because specifying deps in two places is obviously better.) And then the tool would spend a lifetime figuring it all out, and then you would still download like 35% of the repo anyway, because the nature of giant repos is to get tangled together. With languages like PHP it might not be so bad, as you typically are editing one file at a time. But most of Google is Java, and most people like using IDEs for that, so it was a disaster. An IDE likes to touch everything all the time, to re-index and re-compile. And, fun fact: at the time Google recommended you store your entire home directory on an NFS network drive, for security. Everyone quietly ignored that. It was better in one way: everyone could see everyone else's source, re-using libraries was more common, and branching and contributing patches was more common. It prevented people from hoarding their source code, and just throwing .jars or .so's over the wall. But I think the Github era has shown that this did not require everything being in one giant repo! I mean, at the time, Google had Codesearch and other great tools; even though git wasn't super common at the time, this could have been solved in other ways. My current company is in the process of breaking up their monorepo, even though our codebase is nowhere near the same size. Lately I get to taunt my co-workers because I created a new project in the new broken-up way. All the tests on my micro-repo pass in 5 seconds, and it takes many minutes for the big one.

35 points by luu 11y ago

dlitz 11y ago

> Facebook, Google and Twitter Aren't these the companies all terrible at taking outside contributions, except for the projects that they don't manage in their monorepos?

daleharvey 11y ago
I see similiar issues at mozilla. Monorepos are far easier to work with from an engineering standpoint, for external contributors they are definitely prohibitive.
shoo 11y ago
Conway's Law for infrastructure / tooling, perhaps. If you have complete control over the organisation's dev process, and are willing to invest enough, you can build something better than the default free / distributed / popular tooling. This is probably partly enabled by the fact that they don't have to care about trading patches easily with the outside world. Then, by rolling their own in-house workflow/infrastructure/tooling, they make it harder for outsiders to contribute, even if this were organisationally possible and desirable.
falcolas 11y ago
Based on some brief experience at Oracle, accepting outside contributions is a legal nightmare - it is frequently less expensive to do a clean-room re-implementation of a feature than go through the lawyers and contract negotiations now and in the future to get code ownership rights worked out.
- mcs_ 11y ago
  agree with this. I think monolithic repos are part of legal strategy in enterprise.
- jsprogrammer 11y ago
  That's because it's Oracle. Everything associated with Oracle is a legal nightmare.

zokier 11y ago

> Require Trunk-Based Development This I think needs bit of expanding. Does he mean in contrast to feature-branches? Why monorepos would not be compatible with feature-branches?

shoo 11y ago
Not to answer your questions directly, but Paul Hammant has written a bunch about this over the years: http://paulhammant.com/2013/04/05/what-is-trunk-based-development/ http://paulhammant.com/2013/04/05/what-is-trunk-based-develo... http://paulhammant.com/2013/05/06/googles-scaled-trunk-based-development/ http://paulhammant.com/2013/05/06/googles-scaled-trunk-based... http://paulhammant.com/2014/04/03/microsofts-trunk-based-development/ http://paulhammant.com/2014/04/03/microsofts-trunk-based-dev... http://paulhammant.com/2014/01/06/googlers-subset-their-trunk/ http://paulhammant.com/2014/01/06/googlers-subset-their-trun... https://dzone.com/articles/legacy-app-rejuvenation https://dzone.com/articles/legacy-app-rejuvenation
- zokier 11y ago
  I just read the first link (adding the others to my reading backlog), based on it it seems like trunk based development is less of a development model and more of a release model. Especially it does not seem to conflict directly with feature branch model.
  - shoo 11y ago
    Perhaps partly. But Paul also mentions "Developers do not break the build with any commit" -- that places a pretty heavy constraint upon regular development outside of releases. See also the so-called "not rocket science rule of software engineering": https://graydon.livejournal.com/186550.html https://graydon.livejournal.com/186550.html
    - speedkills 11y ago
      See also got-dmz or atlassian bamboo's gatekeeper feature. Team city has something similar too. Surprised the guy who wrote bors hadn't seen more examples, I have seen the idea come up over and over again (and strongly support it).

StavrosK 11y ago

Is it really a single big repo for the whole organization? What happens if you want to deploy your code on servers? Do you have to upload all the mobile app code and history as well? Doesn't the history get very polluted this way?

NickNameNick 11y ago
Are you really using a checkout on a production server to install and configure your software? This feels like a "Doctor, doctor - it hurts when I stick a spoon in my eye" moment - "Don't do that!" Build tools and build servers exist for a reason. Subversion has "svn export" if you insist on working that way. git can git clone repos --depth 1; rm -rf repos/.git if you really, really must. And what you are calling polluted history, they are calling efficient collaboration.
- shoo 11y ago
  If there exists appropriate tooling (e.g. for filtering commit logs based on what touched the code belonging to a single project / what touched the code belonging to a single project and its transitive dependencies) then I dare say a huge monorepo history over many projects and libraries isn't polluted history, it is remarkably rich and useful history. In big places with solid engineering teams and a long-term investment in infrastructure it seems achievable to build the tools to do this properly. (famously: google, which isn't a very fair example). I think many people's default reaction to monorepos -- fear, loathing and disbelief, say -- is based upon working with open source tools / related cloud services / distributed workflows - that scale poorly when dealing with large repositories, at least with default configuration and without a lot of mucking about. That said, this default reaction is arguably a healthy reaction if the reality is that most teams / organisations do not have the resources to invest in setting up tooling / infrastructure beyond what they can get for free / rent for a few hundred dollars a month.
- StavrosK 11y ago
  > Build tools and build servers exist for a reason. What's the build server going to do? Copy the Python code from the repo to the other server? It doesn't seem like setting up a build server just to do "git checkout; rm .git" is worth the effort.

fsloth 11y ago

The key is enabling maximum developer leverage - and one aspect of this is to get rid of as many systems errors and inefficiencies as possible. I think build systems and internal code management schemes are large interconnected systems. It feels fairly pointless just to expose one facet of the entire systems since the total system encompasses all of the individual entities dealing with the codebase - including developers, platform - the whole works. For example, I see no overhead in my daily work from the fact that our codebase is split to several libraries with dependencies through explicit built release versions. On the contrary,there monorepository would be just horrible. So, before saying X is great one should firsr define what type of software is built, what is the specific toolchain, what are the specific configurations, how many developers modify the same codebase frequently, and so on.

beberlei 11y ago
hi, author of those slides here, I actually mention this during the talk, that the talk is over-emphesizing the awesome point to take a point across. It definately is not a technique to use everywhere, so I agree with you. It works for some kind of projects and i was motiviated for this talk, because I saw so many projects using manyrepos in cases where it may not fit very well.
- fsloth 11y ago
  Hi, yeah, sure, effective communication requires simplification. A few points got me concerned in the slides - namely, concerning emphasizing how easy this makes to change things. I would consider that it is not a positive thing that adding a library is cheap and easy, and I find that component versioning can be bypassed even more so. It all sounds like grievous local greedy optimization without a view of the entire lifecycle of the codebase. The slides say change is the only constant. I would claim that unless the change is backed by good and strong architecture - which throwing all architectural firewalls away does not support - then the end result will be a mess of spaghetti the older the codebase gets. But, I may just be too much set in my ways of coding and I respect the fact that these may be non-issues in the particular domain these slides address.

bsimpson 11y ago

Everyone I've ever met who's worked at Twitter hates the monorepo. They have a single git repo with way more files than that tool was ever designed to maintain. As such, normally instantaneous commands like status, log, ls, etc. take an inordinately long amount of time to execute. Apparently their onboarding process is "have someone copy his working copy to a flash drive, then copy it from there to your laptop" because git pull would take too long.

beberlei 11y ago
interesting, i can incorporate this into future versions of the talk about pain points developers might have. The list of users is value free, it is just a mention of companies I know using it. Both the Google and FB talks about this mention the technical challenges, so i guess this is really a downside at very big scales.
indygreg2 11y ago
Twitter is maintaining an internal fork/distribution of Git with patches to attempt to make it scale to their needs. It is described a bit at http://git-merge.com/videos/scaling-git-at-twitter-wilhelm-bierbaum.html http://git-merge.com/videos/scaling-git-at-twitter-wilhelm-b.... But even then, as you mention, Git doesn't scale well for them. This is why Facebook is using Mercurial. This is why the Chromium and Android projects are spread out over hundreds of Git repos. This is why Google's internal monorepo is using a Perforce-like homegrown tool backed by Google's distributed wizardry. Git will start to see significant scaling problems beyond say 1 GB total data, 100,000 files, and X number of refs (there has been significant upstream work around scaling refs - a bunch from Twitter I believe - and I haven't measured how well newer versions handle thousands of refs). The good news is that unless you are a software project with hundreds of developers, have large binary assets, or are doing something like Twitter and have all your developers push all their in-progress feature branches to the monorepo (madness if you ask me), a Git monorepo will scale for you. A Mercurial monorepo will scale even better. Keep in mind that most people aren't {Twitter, Facebook, Google, Mozilla}, so chances are you won't ever reach these scaling limits.
neilk 11y ago
This has to be a sick joke. The monorepo at Google was the bane of my existence, and everyone else that I knew. I don't know what it's like now, but at the time Google was on svn, and there had to be a specially hacked version of it. It used a home-grown internal requirements system, to pull down the tree in a sparse way. And to accomplish that you had to re-specify all your dependencies in these special text files (because specifying deps in two places is obviously better.) And then the tool would spend a lifetime figuring it all out, and then you would still download like 35% of the repo anyway, because the nature of giant repos is to get tangled together. With languages like PHP it might not be so bad, as you typically are editing one file at a time. But most of Google is Java, and most people like using IDEs for that, so it was a disaster. An IDE likes to touch everything all the time, to re-index and re-compile. And, fun fact: at the time Google recommended you store your entire home directory on an NFS network drive, for security. Everyone quietly ignored that. It was better in one way: everyone could see everyone else's source, re-using libraries was more common, and branching and contributing patches was more common. It prevented people from hoarding their source code, and just throwing .jars or .so's over the wall. But I think the Github era has shown that this did not require everything being in one giant repo! I mean, at the time, Google had Codesearch and other great tools; even though git wasn't super common at the time, this could have been solved in other ways. My current company is in the process of breaking up their monorepo, even though our codebase is nowhere near the same size. Lately I get to taunt my co-workers because I created a new project in the new broken-up way. All the tests on my micro-repo pass in 5 seconds, and it takes many minutes for the big one.