13 ms·

Monorepo: please do

234 points by Soliah 8y ago

sharpercoder 8y ago
In my experience, this discussion gets convoluted by confusing modularity with monorepo. They are orthogonal to each other; you can have a very modular codebase in a monorepo but also a very coupled (non-modular) codebase with polyrepo. Though it's true that monorepos without proper discipline can tend towards coupling. Yet, when discussing mono vs poly, we should keep this in mind.
- benmarten 8y ago
  It's not, why do I have to checkout terabyte of code that I don't need, even if the code is modularized?
  - slobotron 8y ago
    Chances are you will end up downloading a lot of dependencies anyways, why not have git deliver it all?
    - nkozyra 8y ago
      Huh? You'd download dependencies for the repos you need, not the code and dependencies for the entire company. It could be several orders of magnitude larger and with a larger organization could be a lot of unnecessary code that any given Dev may never touch.
      - pvorb 8y ago
        But imagine the increased productivity of your devs if they only had to check out a single repo. Anyone has the same organization of projects on their machine. All tools are in one place...
        nkozyra 8y ago
        I don't understand. Where is the argument for more productivity?
        Tempest1981 8y ago
        We have one large-ish repo that keeps showing "This repository currently has approximately 547 loose objects.” We keep pruning and gc'ing with different flags, but pulls just seem far slower than other smaller repos.
        Too 8y ago
        A: You avoid issues such as Readme files stating, "before compiling you have to git clone ../commonA, ../commonB". These always tend to get stale so in reality you also have to git clone ../commonC wasting you tons of hours of troubleshooting. B: Developer working on daily basis in component A finds a bug in component B. He just has to change the code and commit it for review, instead of understanding the specifics of working with component B repository.
  - erulabs 8y ago
    If a mono-repo has a terabyte of code, or if 10 small repos have 1/10th a terabyte each, what have you really gained? In any case, git LFS solves large file storage effectively, as do a number of other artifact storage solutions, and a repo with a terabyte of code is _not_ going to be trivially split apart, since it would be by a factor of thousands, the biggest codebase ever created by humankind.
    - twblalock 8y ago
      If I only need to check out one of the smaller repos then I've gained quite a lot in terms of download speed, storage size, etc. Git LFS adds a lot of complexity I'd rather avoid.
      - erulabs 8y ago
        Sure but then you only have some small portion of the total infrastructure, which adds its own layer of complexity for the people reviewing your changes :P It's all trade offs, is all I'm saying - I honestly still can't decide between the two, although for all companies sub 20 people, I'd for sure stick with a single repo.
        tracker1 8y ago
        If I'm working on Application X, wtf do I care about infrastructure code? Or for that matter, as a specific... if someone is working on Google Maps, should they care about the codebase for Google Inbox for Android?
        matthewmacleod 8y ago
        Does Application X rely on particular infrastructure configuration? Or does Google Inbox on Android integrate with Google Maps? There are dependencies everywhere. Monorepos are one of the tools which can be used to make dealing with them easier in some cases. They’re not an absolute solution not appropriate for all circumstances, but no tool is!
        8y ago
  - nimchimpsky 8y ago
    A terabyte of code ? jesus.
  - jchw 8y ago
    No need to checkout a terabyte of code. If your repo is scaling that high, you're going to want a VFS layer. Microsoft made a VFS layer for Git. As you might imagine, you simply grab files as needed, and your version control just deals with diffs for the most part. Google's own monorepo is proprietary but the Bazel build system is open source and would work great with a VCS hooked up with a VFS layer.
    - jacques_chester 8y ago
      I want to like Bazel. I really do. But on first encounter the syntax is filled with sigils that don't seem to have obvious differences or purpose for existence. Then it turns out that I and others have spent as much time fighting it as using it. Lastly the coverage of ecosystems is sparse and there does not seem to be a lot of activity around extending them -- doing the boring, tedious, unloved work of dealing with everyone's quirks and bugs and corner cases and annoyances (been there, done that). Again: I wish it was a smooth experience. Because I like the ideas very much. But it wasn't when I tried and I don't know anyone -- outside of Google -- for whom it was a smooth experience.
      - iainmerrick 8y ago
        I can’t speak to the actual implementation, but I’m surprised at your description of the syntax as “filled with sigils”, as the syntax is basically Python -- isn’t that about as easy as you can get? I find Bazel’s syntax much easier to deal with than other build languages that use JSON (essentially the same Python syntax but with lots of extra quotes everywhere and extra fussiness about where commas are allowed).
        jacques_chester 8y ago
        bazel build //main:hello-world I'm sure the double slashes and colon have important differences. It is not obvious what they are. cc_binary( name = "hello-world", srcs = ["hello-world.cc"], deps = [ ":hello-greet", "//lib:hello-time", ], ) It's not instantly obvious why one is :hello-greet and the other is //lib:hello-time. I could swear I've seen @ floating around as well. As I said above, I am sure these are all very sensible. But I am just tired of memorising minilanguages embedded in strings. I don't want to any more.
  - skj 8y ago
    Sounds like a tooling problem. We shouldn't use the current state of tooling as an excuse.
    - jessaustin 8y ago
      Isn't the entire argument about the current (or maybe "immediately foreseeable") state of tooling? We don't really care one way or the other, in a philosophical sense. What works?
      - skj 8y ago
        When the tools aren't good enough, we can either toss up our hands and say "I guess it's always going to be like this!", or we can get to work and make better tools.
        jessaustin 8y ago
        This is an argument about how to use current tools. TFA doesn't argue that mono will be great once we work really hard. It argues that mono is great now. Thread parent has a specific objection to that argument. You don't reasonably counter that objection with statements about morality.
        skj 8y ago
        A few things to note: - I was replying to a comment, not the article. - The article spoke about points that were largely independent of the current or future state of tooling. Instead, it focused on fundamental issues with mono- vs poly-repo systems. Most directly, being forced to fix migrations and incompatibilities immediately rather than letting versions skew. If you want to batter someone for not arguing for or against the points in the article, you can do it with the comment I was replying to, or with your own comment just now.
  - mwkaufma 8y ago
    We have a perforce monorepo with ~80gb total payload for the whole thing, but everyone uses streams to filter it, so that's not a problem.
    - hinkley 8y ago
      I think there's a false dichotomy here. In the post yesterday one of the arguments was that if nobody checks out all of the code then what's the value of having the code all in one place? Last monorepo I worked on, individual contributors checked out just the tree they were working on (we had a suite of applications with several shared modules). We made it simple and straightforward for them to get what they wanted and ignore people whose work didn't impact them. But the senior people, who were better with architecture and version control trivia, checked out the entire thing. They would steward any cross-cutting changes that needed to be done, and make sure any callers to shared libraries were updated in the face of breaking changes. They were also backstopped by the build plans, (some of) which also checked out the entire thing.
      - mwkaufma 8y ago
        Streams aren't modules -- they're views. If someone takes you as a dependency and wants you to have visibility on them they add themselves to your stream so you pull down their directory as well.
  - rhacker 8y ago
    It's not for everyone, but damn, why is there a TERABYTE of code? Just curious - assets? checking in binaries?
    - malkia 8y ago
      Test protos. Evaluated configs. Golden data. JAR archives, etc.
  - alexnewman 8y ago
    Signs your build system is never going to be adopted outside of people cargo culting you? - [x] Namespaces and the like without much security benefit - [x] Giant Java dependency - [x] Strange syntax and glyphs
- hinkley 8y ago
  First monorepo I worked on, we used separate compilation units for each 'module'. We paid a tax on build time but it added a bit of friction to adding new cross-module dependencies willy nilly. I don't know how you maintain that arm's length separation if you don't have compilation units in your language of choice, and that may contribute to some of the muddiness in this kind of discussion. "It depends."
  - lstamour 8y ago
    I think the private visibility and shared build chain that Bazel offers could step in here, in that it makes it harder to build a project without specifying every dependency, when combined with linting tools and clearly assigning code ownership...?
benmarten 8y ago
Please don't. It's just too slow and not efficient. Instead use common open source best practices of shared library architecture. Problem solved! Putting everything into one repo is just lack of organization and creates a huge mess.
- zamadatix 8y ago
  Too slow as in "to do it" or too slow as in "to use it". In either case I think if that were true there wouldn't be monorepo's at Google, Facebook, and Microsoft. I will say it's true that didn't come for free, e.g. Microsoft had to make GVFS due to the sheer enormity of their codebase but that's already done and works pretty well. I agree share library style makes more sense in most cases though. The main problem with it is forcing everyone to use the latest library versions but that isn't insurmountable by any means.
  - mlthoughts2018 8y ago
    My old boss was an engineering manager at Google in the 90s and early 2000s. He used to tell us that _everyone_ he interacted with at Google _hated_ the monorepo, and that Google’s in-house tooling did not actually produce anything approaching a sane developer experience. He used to laugh so cynically at stories or that big ACM article touting Google’s use of a monorepo (which was a historical unplanned accident based on toppling a poorly planned Perforce repository way back when), because in his mind, his experience with monorepos at Google was exactly why his engineering department (several hundred engineers) in my old company did not use a monorepo.
    - user5994461 8y ago
      His experience from the 90s and early 2000s is meaningless in the current era. Version control and Google were in their infancy. SVN was first released in 2000. Git in 2008. Branching, tagging and diffing were nowhere near what is possible now. That goes back to desktop with a disk smaller than a GB, CPU in the tens of MHz with a network so slow and reliable, if you have one at all.
iseeyoubydesign 8y ago
what about the discussion of microservices? Why would you ever have these products separate by servers all in one repo? These technical constraints dont really help and are just another attempt to create a silver bullet.
mmmeff 8y ago
Thank you so much for writing this. As someone who’s worked in the best and worst of these two words, the productivity gains are absolutely insane and the limitations, as stated by the author, are no more painful than limitations of federated/polyrepo code. Fighting back against monorepo design is dangerous - embrace experimentation.
- shados 8y ago
  > Fighting back against monorepo design is dangerous What's dangerous about it? Monorepos have a lot of benefits, and should absolutely be considered. Maybe even by most. But right now in the community it's almost pushed as the "only true way with all benefits and no drawbacks", and that's absolutely not true. To the point the knowledge of why and how to poly repo is already starting to get lost. That's dangerous.
  - Benjammer 8y ago
    The real danger here is anyone talking about any system architectures or tooling as "dangerous" (or "not dangerous") absent any other context... What do you even mean by "dangerous"? To a business? To your health? What is the deal with people trying to make these sorts of global assertions in a vacuum about what's "good" and "bad"? This doesn't make any engineering sense in any way to me. You have a problem and you figure out the best way for your business to solve that problem given some bounded resources. Nothing in the basic problem solving process (scientific method?) necessitates all the arbitrary "should" axioms. Why don't people just analyze their specific situation and figure out a solution? It's like people arguing vehemently about the optimal design that every company "should" be using for all windshields for all personal vehicles on the road, without even remotely discussing various vehicle body shapes and sizes.
    - jessaustin 8y ago
      This complaint belongs one level higher in the thread.
    - sebastos 8y ago
totallysnowman 8y ago
I think that the reason of the argument is that both authors understand the definition of "large repository" very differently. With 100 engineers a monorepo might seem a good idea. With 500 it becomes nearly impossible to do anything involving a build. Some isolation is needed. Also from my experience many engineers just don't give a shit about architecture. They create entangled mess, that kind of works for the customer, and go home. Without some enforced isolation it is impossible to maintain it. That being said I am more inclined to polyrepos.
- woolvalley 8y ago
  Why does the build not work at 500 engineers? Just because there are 5 apps / services and their 25 libraries doesn't mean you have build the 4 other apps when doing development just because you did a git pull. Libraries would still need to build their dependants in a CI system with poly or mono repos.
  - totallysnowman 8y ago
    The 5 apps and their 25 libraries are in fact one entangled monolith. That is what I am talking about. It is only marketed as 5 apps. That has some hundreds external dependencies, of course.
- thurn 8y ago
  the fact that essentially 100% of big tech companies use monorepos seems like evidence that it is at least possible to do it in a scalable way...
  - denimnerd42 8y ago
    yeah by writing custom version control software. Am I going to convince my company to do that (which has like 50k software engineers) probably not.
    - rabidrat 8y ago
      How many companies have 50k software engineers? Seems like the handful that do, should do whatever works for them. The rest of us can just use a monorepo.
marcosdumay 8y ago
So... An article based on equating change recording medium with integration testing procedures.
jonthepirate 8y ago
The reason I am upvoting this is that it is written in a positive tone. Too many people - especially in the world of DevOps, trash everything. (X is the worst, don't do that, etc) and more often than not do not offer better guidance following their whiney tone. We need more "please do's" in this industry. Thank you Adam.
kokokokoko 8y ago
Its almost as if both approaches have positives and negatives. Some of which are more important depending on your project and organization. I'd be more interested to read about a project or company that failed due to making one choice or the other. And then by switching things to the other way, things were fixed. Otherwise, as someone who was worked with both, I imagine there are a host of other decisions that will be much determinant on your success. Let's not get too wrapped up in what color to paint the shed.
- deleted 8y ago
  [deleted]
- natalyarostova 8y ago
  >Its almost as if Please don't do this.
skybrian 8y ago
I wonder if a star pattern would work, where you have a single, shared repo for all your libraries and a repo for each app. This would help people working on smaller apps, since they don't need to look at other apps unless they're working on shared library code. Of course, once you are working on library code, you have to build and test all the apps that use it. But even at Google, the people working on the lowest levels of the system can't use the standard tools anyway.
- ceronman 8y ago
  A star pattern still has most of the downsides of the multirepo approach. Specifically, it has the problem of needing a parallel version control (e.g. SemVer) on top of your individual repositories. This creates fragmentation, where different applications have dependencies on different versions of the libraries which ends up in dependency hell, technical debt, and CI hell.
  - skybrian 8y ago
    An alternative would be to have a policy where all the app repos must use the same version (nobody can upgrade until they all upgrade). This makes things harder for the library maintainers, but no more than a monorepo. I don't see why you'd need semver. The apps could sync to a particular commit in the library repo.
    - Too 8y ago
      What you propose is just a fake monorepo, containing the global policy of allowed version of X, disguised as multiple subrepos. OP discusses this.
est31 8y ago
There aren't good monorepo solutions out there (yet). Git LFS is great for few large files, but it doesn't help with tons of smaller files. Git submodules are crap when it comes to usability, and have been for a long time, it's even mentioned in the famous Torvalds Git Talk. Git had a sparse checkouts feature since a long time, but it only affected the checkout itself, all the blobs would still be synced. Now, Git is gaining good monorepo capabilities with the git partial clone feature [1]. Their idea is that with them you can only clone the parts of a repository that are interesting to you. This has been brewing for a while already but I'm not sure how ready it is. There doesn't seem to be user-level documentation for it yet, to my knowledge, so I am linking to the technical docs. [1]: https://github.com/git/git/blob/master/Documentation/technical/partial-clone.txt https://github.com/git/git/blob/master/Documentation/technic...
- itsdrewmiller 8y ago
  https://vfsforgit.org/ https://vfsforgit.org/ is another option here - MS-originated and Github is adopting it - https://venturebeat.com/2017/11/15/github-adopts-microsofts-tool-for-supporting-massive-software-projects/ https://venturebeat.com/2017/11/15/github-adopts-microsofts-...
- dangoor 8y ago
  From earlier discussions around monorepos, I saw references that Google, Facebook, and other large monorepo orgs have been making use of Mercurial.
  - est31 8y ago
    Yes, Facebook is mercurial based to my knowledge. Google is using its custom solution called piper I think: https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...
klodolph 8y ago
I have personally migrated a medium size polyrepo code base (something like ~20 repos?) into a monorepo and I agonized over the decision. But it lifted a huge weight off my shoulders. I feel like if you are working completely in the open-source world, and you are contributing one open-source project to a larger array of available projects, then the decision to use a polyrepo makes a lot of sense. You can submit libraries to a package repository like Yarn/NPM/PyPI or you can use Git references for e.g. Go's package manager. But what I experienced with polyrepos outside this world is that we ended up with a weird DAG of repos. It was always unclear whether a specific piece of code that was duplicated between projects should be moved into one dependency or another, or whether it should have its own repo. Transitive dependencies were no fun at all, if you used git modules you might end up with two copies of the same dependency. You might have to make a sequence of commits to different repos, remembering to update cross-repo references as you go, and if you got stuck somewhere you had to work backwards. This feels like a step backwards, like the step backwards from CVS to RCS. Again, in the open-source world you might have some of this taken care of by using a package manager like Yarn. But if your transitive dependencies aren't suitable for being published that way, it can be tough. Monorepo + Bazel right now is a bit rough around the edges but overall it's reduced the amount of engineering time spent on build systems. On the other hand, it's not like Bazel can't handle polyrepos. In fact, they work quite nicely, and Bazel can automatically do partial checkouts of sets of related polyrepos, if that's your thing. As for VCS scalability problems, I expect that Git is really just the popular VCS du jour and some white horse will show up any day now with a good story for large, centralized repos with a VFS layer. In the meantime any company large enough to experience VCS performance problems but not large enough have their own VCS team (like Google and Facebook) will suffer, or possibly pay for Perforce.
- malkia 8y ago
  BAZEL has WORKSPACE file that can work with multi-repos, but AFAIK things are still rough there, though would get better eventually (I'm bit hand-wavy on the details).
  - klodolph 8y ago
    Yes, exactly. Unfortunately, sometimes the partial checkouts can be somewhat limited by the fact that your WORKSPACE code will import Starlark defined in other repos. This can get a bit ridiculous if your repo uses a bunch of different languages, if you browse through e.g. the TypeScript support instructions for Bazel you’ll see some of what you’re in for. If your project is mostly something like C++ (which has support built-in to Bazel) then the WORKSPACE rules will be much more manageable and partial checkouts become a lot easier.
woolvalley 8y ago
My org went from polyrepo 10 commit semver dependency hell when updating an internal API to monorepo and it saves a lot of time. Unmigrated semver breaking changes are a form of technical debt, and it takes a lot more total man hours to do the 'proper' one by one many commit poly repo migration than the other way around. If we had the tooling to do multirepo atomic commits and reviews then maybe we would of stuck with polyrepos, but it doesn't really exist out in the wild, so monorepo it was.
- mlthoughts2018 8y ago
  My org went from a monorepo where every project had to obey the same CI model and you could not introduce entirely new CI tools for new prototypes over to a polyrepo with separated semver library repos for shared dependencies, and it simplified everything so much. Adding additional PRs across different repos is functionally no different than the same PR with scattered dependencies in a monorepo, except that separating the PRs makes each isolated set of changes more atomic and focused, which has led to fewer bugs and better quality code review and, the hugest win, each repo is free to use whatever CI & deployment tooling it needs, with absolutely no constraints based on whatever CI or deployment tool another chunk of code in some other repo uses. The last point is not trivial. Lots of people glibly assume you can create monorepo solutions where arbitrary new projects inside the monorepo can be free to use whatever resource provisioning strategy or language or tooling or whatever, but in reality this not true, both because there is implicit bias to rely on the existing tooling (even if it’s not right for the job) and monorepos beget monopolicies where experimentation that violates some monorepo decision can be wholly prevented due to political blockers in the name of the monorepo. One example that has frustrated me personally is when working on machine learning projects that require complex runtime environments with custom compiled dependencies, GPU settings, etc. The clear choice for us was to use Docker containers to deliver the built artifacts to the necessary runtime machines, but the whole project was killed when someone from our central IT monorepo tooling team said no. His reasoning was that all the existing model training jobs in our monorepo worked as luigi tasks executed in hadoop. We tried explaining that our model training was not amenable to a map reduce style calculation, and our plan was for a luigi task to invoke the entrypoint command of the container to initiate a single, non-distributed training process (I have specific expertise in this type of model training, so I know from experience this is an effective solution and that map reduce would not be appropriate). But it didn’t matter. The monorepo was set up to assume model training compute jobs had to work one way and only one way, and so it set us back months from training a simple model directly relevant to urgent customer product requests. Had we been able to set this up as a separate repo where there were no global rules over how all compute jobs must be organized, and used our own choice of deployment (containers) with no concern over whatever other projects were using / doing, we could have solved it in a matter of a few days. In my experience, this type of policy blocker is uniquely common to monorepos, and easily avoided in polyrepo situations. It’s just a whole class of problem that rarely applies in a polyrepo setting, but almost always causes huge issues with monorepo policies and fixed tooling choices that end up being a poor fit for necessary experiments or innovative projects that happen later.
cryptonector 8y ago
I agree, use a monorepo. I anxiously await MSFT's git megamonorepo functionality. Until then there's things like git meta[0]. [0] http://twosigma.github.io/git-meta/ http://twosigma.github.io/git-meta/
mindcrime 8y ago
"Shared responsbility" is one of those ideas that sounds good on paper, but doesn't really scale terribly well in the real world. As the old saying goes "when everybody is responsible, nobody is responsible". More to the point, as the author of TFA allows, once a system reaches a certain size, nobody can understand it all. At some point you have to engage division of labor /specialization, and once you do that, it doesn't make sense to have just anybody randomly making changes in parts of the code-base they don't normally work in. I'd rather see a poly-repo approach, with a designated owner for discrete modules, but where anybody can clone any repo, make a proposed fix, and submit a PR. Basically "internal open source" or "inner source"[1]. In my experience, this is about as close as you can get to a "best of both worlds" situation. But, as the author of TFA also says, you absolutely can make either approach work. [1]:https://en.wikipedia.org/wiki/Inner_source https://en.wikipedia.org/wiki/Inner_source
- lhorie 8y ago
  > poly-repo approach, with a designated owner for discrete modules, but where anybody can clone any repo, make a proposed fix, and submit a PR This is how it works at my company. The issue we run into is that PRs coming from non-core maintainers tend to either get over-scrutinized (e.g. "this diff may work for you but it's not generic enough for X/Y/Z") or flat out ignored at the code review stage and sometimes don't land in a timely enough manner. Another challenges with this approach is when you have deeply nested dependencies and need to "propagate" an upgrade in some deep dep up the tree. In the JS/Node world, this usually means fixing an issue involves hacking on transpiled files in the node_modules folder of a project to figure out what change needs to be made, and then mirroring said change into the actual repo and then tweaking things until type checking/linting/CI pass. Not really conducive for collaboration. One other problem is that security/bug fix rollouts are a bit more challenging. We had a case a while back where a crash-inducing bug was fixed and published but people still experienced crashes due to not having upgraded the one out dozens of packages required by their projects.
- falsedan 8y ago
  That's what we did at my last workplace; 800-odd devs, 3000-odd repos, and a service we wrote to record and gather all the ownership metadata (which team owns this repo? where can I contact them? who should be on the code reviews? how do I page them? which project should tickets be created in?) It was a gigantic pain trying to find owners for half-dead repos for services still running and in use, where the original authors had left years ago & from teams 4 or 5 restructures ago. The one thing I learned was: never make a user the owner of a repo (unless it is in their personal space), always find a team to accept responsibility for it.
randyrand 8y ago
The more complicated answer is sometimes you should use a mono repo and other times you shouldn't.
pdpi 8y ago
Can we just move along and get to "Monorepo: Maybe do it, maybe don't. Just think it through and own your decision"? Both monorepos and polyrepos have advantages and disadvantages. Many factors — scale, overall team quality and experience, level of integration between projects are a few that come to mind — will affect how much those advantages and disadvantages matter to any given company at any given point in time. The right choice for you isn't necessarily the right choice for me. Much more important than which approach you choose is understanding, and accepting, the consequences of your choice. You'll want to extract value out of the advantages, you'll need to mitigate the disadvantages. You won't be able to adopt tools and processes meant for the other approach without some degree of friction.
- 0xFACEFEED 8y ago
  That's what most people do. They just don't blog about it.
e3b0c 8y ago
Monorepo could be a decent choice if your software stack does not require too much external dependencies. Or more precisely, the ratio of own code to the third-party code is reasonably high. Let me give a concrete example. The Android open source project (AOSP) which builds the system of Android devices has the code size close to the scale of tens of GB (let alone all the histories!). It is already a massive monorepo in itself. And typically you would have many of them from different OEM/SoC vendors of different major releases. In such a scenario, it would turn into 'a monorepo of monorepos,' which is quite unpleasant to imagine.
malkia 8y ago
Monorepo is a total win, if you have something like https://github.com/Microsoft/VFSForGit https://github.com/Microsoft/VFSForGit (ex GVFS) - e.g. any monorepo that overlays changes, and the rest are simply file names with no actual contents is a win. You can certainly achieve this with Perforce, SVN, HG, any repo system there too. Linux: FUSE + ? Windows: Dokan? CBFS? Or the new fangled https://docs.microsoft.com/en-us/windows/desktop/projfs/projected-file-system https://docs.microsoft.com/en-us/windows/desktop/projfs/proj... which VFSForGit uses
Tempest1981 8y ago
For those using monorepos, what is your branch strategy? Say that 3 projects share a library, and release on different schedules. How does each project freeze shared library changes? Do you keep N version branches? How does the library team know which consumers a commit may break? What tools are recommended?
ceronman 8y ago
I work at a large organization (2000+ devs). We have used both a Monorepo and Polyrepo. After some extensive experience with both models my conclusion is that the Monorepo is by far a superior model, specially for a large organization. Of course the Monorepo is not free of downsides, those mentioned in the original article are real, although a bit exaggerated in my opinion. VCS operations can be slow and scaling a VCS system is challenging, but possible. And the risk of high coupling and a tangled architecture is also very real if you don't use a dependency management system like Bazel/Buck/Pants. But in my opinion the downsides of the Polyrepo are much worse and much much harder to fix. The main problem is that you need a parallel version control system like SemVer on top of your VCS. SemVer is fine for open source projects but for a dynamic organization is a nightmare because it is a manual process prone to failure. SemVer dependency hell is really hard to deal with and creates a lot of technical debt. Additionally, once you go Polyrepo you lose true CI/CD. Yes, you still have CI/CD pipelines but those apply only to a fraction of the code. Once you get used to run `bazel test` and you know you will run every single test of any piece of code that could depend on the code you just changed, you never want to go back. Yes, you could have true CI/CD with Polyrepos, but it requires a lot of work and writing a lot of tooling that does not exist in the wild. It is cheaper to invest in scaling your VCS in a multi-repo.
fxfan 8y ago
There's a lot of discussion of bazel and co inside sub-comments but i have a question that isn't addressed- How do the "global build tools" play with language specific build tools? My primary stack is Rust and Scala. Both have excellent build capabilities in their native tools. How well do pants/bazel integrate with them? I wouldn't want to rewrite complex builds nor would I expect these tools to have 100% functionality of native ones.
- laurentlb 8y ago
  Bazel has some level of support for many languages: https://docs.bazel.build/versions/master/be/overview.html#additional-rules https://docs.bazel.build/versions/master/be/overview.html#ad... I know the Scala rules are used in production by multiple companies. Rust support is improving quickly, but it's not perfect. See the dedicated GitHub repositories for more information. (I work on Bazel)
Rapzid 8y ago
Any good mono repo build tools out there? I've been thinking about this for the past few weeks. Considering creating a general purpose monorepo tool chain and potentially a mono repo first CI system. Unfortunately some of the most popular CI/CD services out there(Travis, Circle, etc) don't even support cross-repo pipelines, much less mono repo builds.
- fxfan 8y ago
  Pants and bazel sound like favorites
  - Rapzid 8y ago
    Interesting, thanks! Didn't realize Bazel was open sourced.. Those both look way more in the weeds than what I would have imagined.. I guess for Bazel at least it makes sense given Googles scale how fine-grain they would get into caching and incremental builds.. For my needs a simple tool that would allow discovering "WORKSPACES" and constructing a build graph based on what's changed, while handing off the actual building to some entry point in the workspace, would be good enough. Have a weird collection Gradle projects, node projects, test suites, docs, and etc with their own build processes already in place. Some things are also on a "critical" path while others can run async given the context(branch, tag, etc)... I'm rambling though.
  - jpeeler 8y ago
    Does anyone know how please (https://please.build https://please.build) compares?
sigil 8y ago
Observe how the verb "force" gets used 6 times. Monorepos "force the conversation." You the individual contributor are "forced to deal with the situation" and "forced to see the upfront cost" of breaking contracts. Your team is forced to "look up from their component, and see the perspectives of other teams and consumers." All this forcing people to do things the Right Way (my way) is surely part of the pushback against monorepos. But set that aside for the moment. Let's suppose defaults should force people to do things the Right Way, and that we also know what the Right Way is. Instead of letting anyone sloppily depend on any code checked into the monorepo, shouldn't we force people to think long and hard about contracts between components -- the default concern in a polyrepo architecture? When and how to make contracts, when and how to break contracts? Isn't this how Amazon moved past their monorepo woes, adopted SOA, built AWS, and became one of the largest companies on earth? Heck, isn't this how the Internet itself was built?
- holoway 8y ago
  Author here. The Right Way :tm: is situational - there isn't one right answer to things like when and how to make contracts, or how to break them. When I used the term "force", you'll see that I'm usually talking about dialog between people and teams. It's not that it's a single right way to do it. There isn't, and anyone who tells you there is has something to sell you, or is inexperienced enough to not have seen enough of the problem domain. What is for certain: teams need to have tooling that causes the conversations and behavior that lead to the outcomes we want. As systems and teams scale large enough, this tooling becomes essential - without it, teams go their own way, and in so doing, may or may not create the culture needed for the outcomes you want. I have never once in my career, so far, had to tell a team to communicate less. When we're talking about engineering organizations that are large enough to diverge, you must solve these problems somehow, and it needs to be systemic and intentional.
  - sigil 8y ago
    Thanks for the response. Out of curiosity, how does your engineering organization introduce new dependencies within the monorepo? Can B, C and D all depend on A without A's consent or even awareness? (Suppose A is some checked in code that's useful, going to see updates in future, but is dormant at present.) Your post puts a lot of the onus on A for breaking B, C, and D, but I think equal care and consideration needs to come from the other side of the contract. Eg, What are you depending on? Is it a dependency you want to take on, or are you and the shared code likely to diverge in life? These are top of mind decisions in a polyrepo architecture, but from my experience they're often not even considered in a monorepo. Anything checked in is fair game for reuse. This is why I suspect you may be "forcing" the wrong thing. For reference I've worked in companies large and small, both monorepo and polyrepo. When I worked on Windows back in the 00's the monorepo tooling (SourceDepot) was quite amazing for the time, but the costs of that sort of coordination were also painfully apparent to everyone. The place I currently work has a monorepo for desktop software and polyrepos for everything else. It isn't a straight up A/B experiment, but anecdotally the pain is higher and shipping velocity lower in the monorepo half of the world. Most of the monorepo pain is related to CI or other costs of global coordination, the kind of things Matt touches on midway (albeit probably too subtlely). I'd be interested to see your counterarguments to those points as well. Do you need fancy dependency management tooling to make your global CI builds fast and reproducible? Matt argues those end up being equivalent to the kind of dependency tooling that's intrinsic to polyrepo architectures anyway.
rdsubhas 8y ago
This is starting to get a debate of "principles", like forcing A and B to talk, or forcing A and B to have more explicit boundaries, and so on. Guess where that ends (hint: it doesn't). With a monorepo, the basic effort you have to put in to start scaling is quite high. To properly do a local build, you need bazel or something. But bazel doesn't stop at just building, but it manages dependencies all the way down to libraries and stuff. Let's say you're using certain maven plugins, like code coverage, shading, etc. Would bazel have all the build plugins your project needs? Most likely not. You have to backport a bunch of plugins from maven to bazel and so on. Guess how many IDEs support bazel? Not a lot. Then you need to run a different kind of build farm. When you check-in stuff to a monorepo, you need to split and distribute one single build. Compared to a polyrepo where one build == one job, a monorepo is like one build == a distributed pool of jobs, which again needs very deep integration with the build tool (bazel again here), to fan out, fan in across multiple machines, aggregate artifacts, and so on. Then the deployment. Same again. There is no "just works" hosted CI or hosted git or anything for monorepos. People still dabble with concourse or so on. And guess what, for a component in its own repo, you don't need to do anything. Existing industry and OSS tooling is built from ground up for that. Just go and use them. To provide a developer a "basic experience" to go from working on, building and deploying a single component – the upfront investment you need to provide with a monorepo is very high. Most companies cannot spend time on that, because scale means different things to different companies. There is a vast gap in the amount of ops/dev tooling you have for independent hosted components vs monorepo tools. Just search for "monorepo tools" or DAG and see how many you can come up with. So what really happens with a monorepo is, most companies go with multi-module maven and jenkins multi-job. The results are easy to predict. I'm not saying that maven/jenkins are bad, but they are _not_ sophisticated, and are not anywhere close to what Twitter/Facebook/Google or any modern company uses to deal with a monorepo (for a good reason). They are just not good at DAG. If you're relying on maven+jenkins as your monorepo solution, all I can say is "good luck". Instead, if you start by putting one component in one repo, you keep scaling for _much longer_ before you hit a barrier. In principle, monorepos are better. In practice, they don't have the basic "table stakes" tooling that you need to get going. Maybe monorepo devops tooling is a next developer productivity startup space. But until then, it's not mainstream for very good reasons.
coldtea 8y ago
>But I think Matt’s argument misses the #1 reason I’ve flipped quite hard to a monorepo perspective as my own level in the organization has gotten higher Perhaps the fact that since their level was now higher, they wouldn't have to deal with the nitty gritty details and pain of working with a monorepo as a developer? E.g. I wasn't for it when I was a dev, but now that I can just impose it on others, I love it. Same with how various 'development process' rituals are adopted...
AzzieElbab 8y ago
As engineers we spend wast amounts of time in constant search for a rival to "tabs vs spaces" debate
luord 8y ago
Yet another chapter in one of the big flamewars. Seeing as I fall in the monorepo camp, I must say I mostly agree; also, I much prefer this tone for an article. I find it enjoyable how plenty of comments both here and in the other discussion are of people saying "We had a mono/polyrepo and things improved tremendously when we migrated towards a poly/monorepo". The issue might be one of growth and complacency: a drastic change like that forces the team to face the technical debt that was being ignored and do a better implementation using what was learned from past mistakes.

234 points by Soliah 8y ago

In my experience, this discussion gets convoluted by confusing modularity with monorepo. They are orthogonal to each other; you can have a very modular codebase in a monorepo but also a very coupled (non-modular) codebase with polyrepo. Though it's true that monorepos without proper discipline can tend towards coupling. Yet, when discussing mono vs poly, we should keep this in mind.

benmarten 8y ago
It's not, why do I have to checkout terabyte of code that I don't need, even if the code is modularized?
- slobotron 8y ago
  Chances are you will end up downloading a lot of dependencies anyways, why not have git deliver it all?
  - nkozyra 8y ago
    Huh? You'd download dependencies for the repos you need, not the code and dependencies for the entire company. It could be several orders of magnitude larger and with a larger organization could be a lot of unnecessary code that any given Dev may never touch.
    - pvorb 8y ago
      But imagine the increased productivity of your devs if they only had to check out a single repo. Anyone has the same organization of projects on their machine. All tools are in one place...
      - nkozyra 8y ago
        I don't understand. Where is the argument for more productivity?
        Tempest1981 8y ago
        We have one large-ish repo that keeps showing "This repository currently has approximately 547 loose objects.” We keep pruning and gc'ing with different flags, but pulls just seem far slower than other smaller repos.
        Too 8y ago
        A: You avoid issues such as Readme files stating, "before compiling you have to git clone ../commonA, ../commonB". These always tend to get stale so in reality you also have to git clone ../commonC wasting you tons of hours of troubleshooting. B: Developer working on daily basis in component A finds a bug in component B. He just has to change the code and commit it for review, instead of understanding the specifics of working with component B repository.
- erulabs 8y ago
  If a mono-repo has a terabyte of code, or if 10 small repos have 1/10th a terabyte each, what have you really gained? In any case, git LFS solves large file storage effectively, as do a number of other artifact storage solutions, and a repo with a terabyte of code is _not_ going to be trivially split apart, since it would be by a factor of thousands, the biggest codebase ever created by humankind.
  - twblalock 8y ago
    If I only need to check out one of the smaller repos then I've gained quite a lot in terms of download speed, storage size, etc. Git LFS adds a lot of complexity I'd rather avoid.
    - erulabs 8y ago
      Sure but then you only have some small portion of the total infrastructure, which adds its own layer of complexity for the people reviewing your changes :P It's all trade offs, is all I'm saying - I honestly still can't decide between the two, although for all companies sub 20 people, I'd for sure stick with a single repo.
      - tracker1 8y ago
        If I'm working on Application X, wtf do I care about infrastructure code? Or for that matter, as a specific... if someone is working on Google Maps, should they care about the codebase for Google Inbox for Android?
        matthewmacleod 8y ago
        Does Application X rely on particular infrastructure configuration? Or does Google Inbox on Android integrate with Google Maps? There are dependencies everywhere. Monorepos are one of the tools which can be used to make dealing with them easier in some cases. They’re not an absolute solution not appropriate for all circumstances, but no tool is!
        8y ago
- nimchimpsky 8y ago
  A terabyte of code ? jesus.
- jchw 8y ago
  No need to checkout a terabyte of code. If your repo is scaling that high, you're going to want a VFS layer. Microsoft made a VFS layer for Git. As you might imagine, you simply grab files as needed, and your version control just deals with diffs for the most part. Google's own monorepo is proprietary but the Bazel build system is open source and would work great with a VCS hooked up with a VFS layer.
  - jacques_chester 8y ago
    I want to like Bazel. I really do. But on first encounter the syntax is filled with sigils that don't seem to have obvious differences or purpose for existence. Then it turns out that I and others have spent as much time fighting it as using it. Lastly the coverage of ecosystems is sparse and there does not seem to be a lot of activity around extending them -- doing the boring, tedious, unloved work of dealing with everyone's quirks and bugs and corner cases and annoyances (been there, done that). Again: I wish it was a smooth experience. Because I like the ideas very much. But it wasn't when I tried and I don't know anyone -- outside of Google -- for whom it was a smooth experience.
    - iainmerrick 8y ago
      I can’t speak to the actual implementation, but I’m surprised at your description of the syntax as “filled with sigils”, as the syntax is basically Python -- isn’t that about as easy as you can get? I find Bazel’s syntax much easier to deal with than other build languages that use JSON (essentially the same Python syntax but with lots of extra quotes everywhere and extra fussiness about where commas are allowed).
      - jacques_chester 8y ago
        bazel build //main:hello-world I'm sure the double slashes and colon have important differences. It is not obvious what they are. cc_binary( name = "hello-world", srcs = ["hello-world.cc"], deps = [ ":hello-greet", "//lib:hello-time", ], ) It's not instantly obvious why one is :hello-greet and the other is //lib:hello-time. I could swear I've seen @ floating around as well. As I said above, I am sure these are all very sensible. But I am just tired of memorising minilanguages embedded in strings. I don't want to any more.
- skj 8y ago
  Sounds like a tooling problem. We shouldn't use the current state of tooling as an excuse.
  - jessaustin 8y ago
    Isn't the entire argument about the current (or maybe "immediately foreseeable") state of tooling? We don't really care one way or the other, in a philosophical sense. What works?
    - skj 8y ago
      When the tools aren't good enough, we can either toss up our hands and say "I guess it's always going to be like this!", or we can get to work and make better tools.
      - jessaustin 8y ago
        This is an argument about how to use current tools. TFA doesn't argue that mono will be great once we work really hard. It argues that mono is great now. Thread parent has a specific objection to that argument. You don't reasonably counter that objection with statements about morality.
        skj 8y ago
        A few things to note: - I was replying to a comment, not the article. - The article spoke about points that were largely independent of the current or future state of tooling. Instead, it focused on fundamental issues with mono- vs poly-repo systems. Most directly, being forced to fix migrations and incompatibilities immediately rather than letting versions skew. If you want to batter someone for not arguing for or against the points in the article, you can do it with the comment I was replying to, or with your own comment just now.
- mwkaufma 8y ago
  We have a perforce monorepo with ~80gb total payload for the whole thing, but everyone uses streams to filter it, so that's not a problem.
  - hinkley 8y ago
    I think there's a false dichotomy here. In the post yesterday one of the arguments was that if nobody checks out all of the code then what's the value of having the code all in one place? Last monorepo I worked on, individual contributors checked out just the tree they were working on (we had a suite of applications with several shared modules). We made it simple and straightforward for them to get what they wanted and ignore people whose work didn't impact them. But the senior people, who were better with architecture and version control trivia, checked out the entire thing. They would steward any cross-cutting changes that needed to be done, and make sure any callers to shared libraries were updated in the face of breaking changes. They were also backstopped by the build plans, (some of) which also checked out the entire thing.
    - mwkaufma 8y ago
      Streams aren't modules -- they're views. If someone takes you as a dependency and wants you to have visibility on them they add themselves to your stream so you pull down their directory as well.
- rhacker 8y ago
  It's not for everyone, but damn, why is there a TERABYTE of code? Just curious - assets? checking in binaries?
  - malkia 8y ago
    Test protos. Evaluated configs. Golden data. JAR archives, etc.
- alexnewman 8y ago
  Signs your build system is never going to be adopted outside of people cargo culting you? - [x] Namespaces and the like without much security benefit - [x] Giant Java dependency - [x] Strange syntax and glyphs
hinkley 8y ago
First monorepo I worked on, we used separate compilation units for each 'module'. We paid a tax on build time but it added a bit of friction to adding new cross-module dependencies willy nilly. I don't know how you maintain that arm's length separation if you don't have compilation units in your language of choice, and that may contribute to some of the muddiness in this kind of discussion. "It depends."
- lstamour 8y ago
  I think the private visibility and shared build chain that Bazel offers could step in here, in that it makes it harder to build a project without specifying every dependency, when combined with linting tools and clearly assigning code ownership...?

benmarten 8y ago

Please don't. It's just too slow and not efficient. Instead use common open source best practices of shared library architecture. Problem solved! Putting everything into one repo is just lack of organization and creates a huge mess.

zamadatix 8y ago
Too slow as in "to do it" or too slow as in "to use it". In either case I think if that were true there wouldn't be monorepo's at Google, Facebook, and Microsoft. I will say it's true that didn't come for free, e.g. Microsoft had to make GVFS due to the sheer enormity of their codebase but that's already done and works pretty well. I agree share library style makes more sense in most cases though. The main problem with it is forcing everyone to use the latest library versions but that isn't insurmountable by any means.
- mlthoughts2018 8y ago
  My old boss was an engineering manager at Google in the 90s and early 2000s. He used to tell us that _everyone_ he interacted with at Google _hated_ the monorepo, and that Google’s in-house tooling did not actually produce anything approaching a sane developer experience. He used to laugh so cynically at stories or that big ACM article touting Google’s use of a monorepo (which was a historical unplanned accident based on toppling a poorly planned Perforce repository way back when), because in his mind, his experience with monorepos at Google was exactly why his engineering department (several hundred engineers) in my old company did not use a monorepo.
  - user5994461 8y ago
    His experience from the 90s and early 2000s is meaningless in the current era. Version control and Google were in their infancy. SVN was first released in 2000. Git in 2008. Branching, tagging and diffing were nowhere near what is possible now. That goes back to desktop with a disk smaller than a GB, CPU in the tens of MHz with a network so slow and reliable, if you have one at all.

iseeyoubydesign 8y ago

what about the discussion of microservices? Why would you ever have these products separate by servers all in one repo? These technical constraints dont really help and are just another attempt to create a silver bullet.

mmmeff 8y ago

Thank you so much for writing this. As someone who’s worked in the best and worst of these two words, the productivity gains are absolutely insane and the limitations, as stated by the author, are no more painful than limitations of federated/polyrepo code. Fighting back against monorepo design is dangerous - embrace experimentation.

shados 8y ago
> Fighting back against monorepo design is dangerous What's dangerous about it? Monorepos have a lot of benefits, and should absolutely be considered. Maybe even by most. But right now in the community it's almost pushed as the "only true way with all benefits and no drawbacks", and that's absolutely not true. To the point the knowledge of why and how to poly repo is already starting to get lost. That's dangerous.
- Benjammer 8y ago
  The real danger here is anyone talking about any system architectures or tooling as "dangerous" (or "not dangerous") absent any other context... What do you even mean by "dangerous"? To a business? To your health? What is the deal with people trying to make these sorts of global assertions in a vacuum about what's "good" and "bad"? This doesn't make any engineering sense in any way to me. You have a problem and you figure out the best way for your business to solve that problem given some bounded resources. Nothing in the basic problem solving process (scientific method?) necessitates all the arbitrary "should" axioms. Why don't people just analyze their specific situation and figure out a solution? It's like people arguing vehemently about the optimal design that every company "should" be using for all windshields for all personal vehicles on the road, without even remotely discussing various vehicle body shapes and sizes.
  - jessaustin 8y ago
    This complaint belongs one level higher in the thread.
  - sebastos 8y ago

totallysnowman 8y ago

I think that the reason of the argument is that both authors understand the definition of "large repository" very differently. With 100 engineers a monorepo might seem a good idea. With 500 it becomes nearly impossible to do anything involving a build. Some isolation is needed. Also from my experience many engineers just don't give a shit about architecture. They create entangled mess, that kind of works for the customer, and go home. Without some enforced isolation it is impossible to maintain it. That being said I am more inclined to polyrepos.

woolvalley 8y ago
Why does the build not work at 500 engineers? Just because there are 5 apps / services and their 25 libraries doesn't mean you have build the 4 other apps when doing development just because you did a git pull. Libraries would still need to build their dependants in a CI system with poly or mono repos.
- totallysnowman 8y ago
  The 5 apps and their 25 libraries are in fact one entangled monolith. That is what I am talking about. It is only marketed as 5 apps. That has some hundreds external dependencies, of course.
thurn 8y ago
the fact that essentially 100% of big tech companies use monorepos seems like evidence that it is at least possible to do it in a scalable way...
- denimnerd42 8y ago
  yeah by writing custom version control software. Am I going to convince my company to do that (which has like 50k software engineers) probably not.
  - rabidrat 8y ago
    How many companies have 50k software engineers? Seems like the handful that do, should do whatever works for them. The rest of us can just use a monorepo.

marcosdumay 8y ago

So... An article based on equating change recording medium with integration testing procedures.

jonthepirate 8y ago

The reason I am upvoting this is that it is written in a positive tone. Too many people - especially in the world of DevOps, trash everything. (X is the worst, don't do that, etc) and more often than not do not offer better guidance following their whiney tone. We need more "please do's" in this industry. Thank you Adam.

kokokokoko 8y ago

Its almost as if both approaches have positives and negatives. Some of which are more important depending on your project and organization. I'd be more interested to read about a project or company that failed due to making one choice or the other. And then by switching things to the other way, things were fixed. Otherwise, as someone who was worked with both, I imagine there are a host of other decisions that will be much determinant on your success. Let's not get too wrapped up in what color to paint the shed.

deleted 8y ago
[deleted]
natalyarostova 8y ago
>Its almost as if Please don't do this.

skybrian 8y ago

I wonder if a star pattern would work, where you have a single, shared repo for all your libraries and a repo for each app. This would help people working on smaller apps, since they don't need to look at other apps unless they're working on shared library code. Of course, once you are working on library code, you have to build and test all the apps that use it. But even at Google, the people working on the lowest levels of the system can't use the standard tools anyway.

ceronman 8y ago
A star pattern still has most of the downsides of the multirepo approach. Specifically, it has the problem of needing a parallel version control (e.g. SemVer) on top of your individual repositories. This creates fragmentation, where different applications have dependencies on different versions of the libraries which ends up in dependency hell, technical debt, and CI hell.
- skybrian 8y ago
  An alternative would be to have a policy where all the app repos must use the same version (nobody can upgrade until they all upgrade). This makes things harder for the library maintainers, but no more than a monorepo. I don't see why you'd need semver. The apps could sync to a particular commit in the library repo.
  - Too 8y ago
    What you propose is just a fake monorepo, containing the global policy of allowed version of X, disguised as multiple subrepos. OP discusses this.

est31 8y ago

There aren't good monorepo solutions out there (yet). Git LFS is great for few large files, but it doesn't help with tons of smaller files. Git submodules are crap when it comes to usability, and have been for a long time, it's even mentioned in the famous Torvalds Git Talk. Git had a sparse checkouts feature since a long time, but it only affected the checkout itself, all the blobs would still be synced. Now, Git is gaining good monorepo capabilities with the git partial clone feature [1]. Their idea is that with them you can only clone the parts of a repository that are interesting to you. This has been brewing for a while already but I'm not sure how ready it is. There doesn't seem to be user-level documentation for it yet, to my knowledge, so I am linking to the technical docs. [1]: https://github.com/git/git/blob/master/Documentation/technical/partial-clone.txt https://github.com/git/git/blob/master/Documentation/technic...

itsdrewmiller 8y ago
https://vfsforgit.org/ https://vfsforgit.org/ is another option here - MS-originated and Github is adopting it - https://venturebeat.com/2017/11/15/github-adopts-microsofts-tool-for-supporting-massive-software-projects/ https://venturebeat.com/2017/11/15/github-adopts-microsofts-...
dangoor 8y ago
From earlier discussions around monorepos, I saw references that Google, Facebook, and other large monorepo orgs have been making use of Mercurial.
- est31 8y ago
  Yes, Facebook is mercurial based to my knowledge. Google is using its custom solution called piper I think: https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...

klodolph 8y ago

I have personally migrated a medium size polyrepo code base (something like ~20 repos?) into a monorepo and I agonized over the decision. But it lifted a huge weight off my shoulders. I feel like if you are working completely in the open-source world, and you are contributing one open-source project to a larger array of available projects, then the decision to use a polyrepo makes a lot of sense. You can submit libraries to a package repository like Yarn/NPM/PyPI or you can use Git references for e.g. Go's package manager. But what I experienced with polyrepos outside this world is that we ended up with a weird DAG of repos. It was always unclear whether a specific piece of code that was duplicated between projects should be moved into one dependency or another, or whether it should have its own repo. Transitive dependencies were no fun at all, if you used git modules you might end up with two copies of the same dependency. You might have to make a sequence of commits to different repos, remembering to update cross-repo references as you go, and if you got stuck somewhere you had to work backwards. This feels like a step backwards, like the step backwards from CVS to RCS. Again, in the open-source world you might have some of this taken care of by using a package manager like Yarn. But if your transitive dependencies aren't suitable for being published that way, it can be tough. Monorepo + Bazel right now is a bit rough around the edges but overall it's reduced the amount of engineering time spent on build systems. On the other hand, it's not like Bazel can't handle polyrepos. In fact, they work quite nicely, and Bazel can automatically do partial checkouts of sets of related polyrepos, if that's your thing. As for VCS scalability problems, I expect that Git is really just the popular VCS du jour and some white horse will show up any day now with a good story for large, centralized repos with a VFS layer. In the meantime any company large enough to experience VCS performance problems but not large enough have their own VCS team (like Google and Facebook) will suffer, or possibly pay for Perforce.

malkia 8y ago
BAZEL has WORKSPACE file that can work with multi-repos, but AFAIK things are still rough there, though would get better eventually (I'm bit hand-wavy on the details).
- klodolph 8y ago
  Yes, exactly. Unfortunately, sometimes the partial checkouts can be somewhat limited by the fact that your WORKSPACE code will import Starlark defined in other repos. This can get a bit ridiculous if your repo uses a bunch of different languages, if you browse through e.g. the TypeScript support instructions for Bazel you’ll see some of what you’re in for. If your project is mostly something like C++ (which has support built-in to Bazel) then the WORKSPACE rules will be much more manageable and partial checkouts become a lot easier.

woolvalley 8y ago

My org went from polyrepo 10 commit semver dependency hell when updating an internal API to monorepo and it saves a lot of time. Unmigrated semver breaking changes are a form of technical debt, and it takes a lot more total man hours to do the 'proper' one by one many commit poly repo migration than the other way around. If we had the tooling to do multirepo atomic commits and reviews then maybe we would of stuck with polyrepos, but it doesn't really exist out in the wild, so monorepo it was.

mlthoughts2018 8y ago
My org went from a monorepo where every project had to obey the same CI model and you could not introduce entirely new CI tools for new prototypes over to a polyrepo with separated semver library repos for shared dependencies, and it simplified everything so much. Adding additional PRs across different repos is functionally no different than the same PR with scattered dependencies in a monorepo, except that separating the PRs makes each isolated set of changes more atomic and focused, which has led to fewer bugs and better quality code review and, the hugest win, each repo is free to use whatever CI & deployment tooling it needs, with absolutely no constraints based on whatever CI or deployment tool another chunk of code in some other repo uses. The last point is not trivial. Lots of people glibly assume you can create monorepo solutions where arbitrary new projects inside the monorepo can be free to use whatever resource provisioning strategy or language or tooling or whatever, but in reality this not true, both because there is implicit bias to rely on the existing tooling (even if it’s not right for the job) and monorepos beget monopolicies where experimentation that violates some monorepo decision can be wholly prevented due to political blockers in the name of the monorepo. One example that has frustrated me personally is when working on machine learning projects that require complex runtime environments with custom compiled dependencies, GPU settings, etc. The clear choice for us was to use Docker containers to deliver the built artifacts to the necessary runtime machines, but the whole project was killed when someone from our central IT monorepo tooling team said no. His reasoning was that all the existing model training jobs in our monorepo worked as luigi tasks executed in hadoop. We tried explaining that our model training was not amenable to a map reduce style calculation, and our plan was for a luigi task to invoke the entrypoint command of the container to initiate a single, non-distributed training process (I have specific expertise in this type of model training, so I know from experience this is an effective solution and that map reduce would not be appropriate). But it didn’t matter. The monorepo was set up to assume model training compute jobs had to work one way and only one way, and so it set us back months from training a simple model directly relevant to urgent customer product requests. Had we been able to set this up as a separate repo where there were no global rules over how all compute jobs must be organized, and used our own choice of deployment (containers) with no concern over whatever other projects were using / doing, we could have solved it in a matter of a few days. In my experience, this type of policy blocker is uniquely common to monorepos, and easily avoided in polyrepo situations. It’s just a whole class of problem that rarely applies in a polyrepo setting, but almost always causes huge issues with monorepo policies and fixed tooling choices that end up being a poor fit for necessary experiments or innovative projects that happen later.

cryptonector 8y ago

I agree, use a monorepo. I anxiously await MSFT's git megamonorepo functionality. Until then there's things like git meta[0]. [0] http://twosigma.github.io/git-meta/ http://twosigma.github.io/git-meta/

mindcrime 8y ago

"Shared responsbility" is one of those ideas that sounds good on paper, but doesn't really scale terribly well in the real world. As the old saying goes "when everybody is responsible, nobody is responsible". More to the point, as the author of TFA allows, once a system reaches a certain size, nobody can understand it all. At some point you have to engage division of labor /specialization, and once you do that, it doesn't make sense to have just anybody randomly making changes in parts of the code-base they don't normally work in. I'd rather see a poly-repo approach, with a designated owner for discrete modules, but where anybody can clone any repo, make a proposed fix, and submit a PR. Basically "internal open source" or "inner source"[1]. In my experience, this is about as close as you can get to a "best of both worlds" situation. But, as the author of TFA also says, you absolutely can make either approach work. [1]:https://en.wikipedia.org/wiki/Inner_source https://en.wikipedia.org/wiki/Inner_source

lhorie 8y ago
> poly-repo approach, with a designated owner for discrete modules, but where anybody can clone any repo, make a proposed fix, and submit a PR This is how it works at my company. The issue we run into is that PRs coming from non-core maintainers tend to either get over-scrutinized (e.g. "this diff may work for you but it's not generic enough for X/Y/Z") or flat out ignored at the code review stage and sometimes don't land in a timely enough manner. Another challenges with this approach is when you have deeply nested dependencies and need to "propagate" an upgrade in some deep dep up the tree. In the JS/Node world, this usually means fixing an issue involves hacking on transpiled files in the node_modules folder of a project to figure out what change needs to be made, and then mirroring said change into the actual repo and then tweaking things until type checking/linting/CI pass. Not really conducive for collaboration. One other problem is that security/bug fix rollouts are a bit more challenging. We had a case a while back where a crash-inducing bug was fixed and published but people still experienced crashes due to not having upgraded the one out dozens of packages required by their projects.
falsedan 8y ago
That's what we did at my last workplace; 800-odd devs, 3000-odd repos, and a service we wrote to record and gather all the ownership metadata (which team owns this repo? where can I contact them? who should be on the code reviews? how do I page them? which project should tickets be created in?) It was a gigantic pain trying to find owners for half-dead repos for services still running and in use, where the original authors had left years ago & from teams 4 or 5 restructures ago. The one thing I learned was: never make a user the owner of a repo (unless it is in their personal space), always find a team to accept responsibility for it.

randyrand 8y ago

The more complicated answer is sometimes you should use a mono repo and other times you shouldn't.

pdpi 8y ago

Can we just move along and get to "Monorepo: Maybe do it, maybe don't. Just think it through and own your decision"? Both monorepos and polyrepos have advantages and disadvantages. Many factors — scale, overall team quality and experience, level of integration between projects are a few that come to mind — will affect how much those advantages and disadvantages matter to any given company at any given point in time. The right choice for you isn't necessarily the right choice for me. Much more important than which approach you choose is understanding, and accepting, the consequences of your choice. You'll want to extract value out of the advantages, you'll need to mitigate the disadvantages. You won't be able to adopt tools and processes meant for the other approach without some degree of friction.

0xFACEFEED 8y ago
That's what most people do. They just don't blog about it.

e3b0c 8y ago

Monorepo could be a decent choice if your software stack does not require too much external dependencies. Or more precisely, the ratio of own code to the third-party code is reasonably high. Let me give a concrete example. The Android open source project (AOSP) which builds the system of Android devices has the code size close to the scale of tens of GB (let alone all the histories!). It is already a massive monorepo in itself. And typically you would have many of them from different OEM/SoC vendors of different major releases. In such a scenario, it would turn into 'a monorepo of monorepos,' which is quite unpleasant to imagine.

malkia 8y ago

Monorepo is a total win, if you have something like https://github.com/Microsoft/VFSForGit https://github.com/Microsoft/VFSForGit (ex GVFS) - e.g. any monorepo that overlays changes, and the rest are simply file names with no actual contents is a win. You can certainly achieve this with Perforce, SVN, HG, any repo system there too. Linux: FUSE + ? Windows: Dokan? CBFS? Or the new fangled https://docs.microsoft.com/en-us/windows/desktop/projfs/projected-file-system https://docs.microsoft.com/en-us/windows/desktop/projfs/proj... which VFSForGit uses

Tempest1981 8y ago

For those using monorepos, what is your branch strategy? Say that 3 projects share a library, and release on different schedules. How does each project freeze shared library changes? Do you keep N version branches? How does the library team know which consumers a commit may break? What tools are recommended?

ceronman 8y ago

I work at a large organization (2000+ devs). We have used both a Monorepo and Polyrepo. After some extensive experience with both models my conclusion is that the Monorepo is by far a superior model, specially for a large organization. Of course the Monorepo is not free of downsides, those mentioned in the original article are real, although a bit exaggerated in my opinion. VCS operations can be slow and scaling a VCS system is challenging, but possible. And the risk of high coupling and a tangled architecture is also very real if you don't use a dependency management system like Bazel/Buck/Pants. But in my opinion the downsides of the Polyrepo are much worse and much much harder to fix. The main problem is that you need a parallel version control system like SemVer on top of your VCS. SemVer is fine for open source projects but for a dynamic organization is a nightmare because it is a manual process prone to failure. SemVer dependency hell is really hard to deal with and creates a lot of technical debt. Additionally, once you go Polyrepo you lose true CI/CD. Yes, you still have CI/CD pipelines but those apply only to a fraction of the code. Once you get used to run `bazel test` and you know you will run every single test of any piece of code that could depend on the code you just changed, you never want to go back. Yes, you could have true CI/CD with Polyrepos, but it requires a lot of work and writing a lot of tooling that does not exist in the wild. It is cheaper to invest in scaling your VCS in a multi-repo.

fxfan 8y ago

There's a lot of discussion of bazel and co inside sub-comments but i have a question that isn't addressed- How do the "global build tools" play with language specific build tools? My primary stack is Rust and Scala. Both have excellent build capabilities in their native tools. How well do pants/bazel integrate with them? I wouldn't want to rewrite complex builds nor would I expect these tools to have 100% functionality of native ones.

laurentlb 8y ago
Bazel has some level of support for many languages: https://docs.bazel.build/versions/master/be/overview.html#additional-rules https://docs.bazel.build/versions/master/be/overview.html#ad... I know the Scala rules are used in production by multiple companies. Rust support is improving quickly, but it's not perfect. See the dedicated GitHub repositories for more information. (I work on Bazel)

Rapzid 8y ago

Any good mono repo build tools out there? I've been thinking about this for the past few weeks. Considering creating a general purpose monorepo tool chain and potentially a mono repo first CI system. Unfortunately some of the most popular CI/CD services out there(Travis, Circle, etc) don't even support cross-repo pipelines, much less mono repo builds.

fxfan 8y ago
Pants and bazel sound like favorites
- Rapzid 8y ago
  Interesting, thanks! Didn't realize Bazel was open sourced.. Those both look way more in the weeds than what I would have imagined.. I guess for Bazel at least it makes sense given Googles scale how fine-grain they would get into caching and incremental builds.. For my needs a simple tool that would allow discovering "WORKSPACES" and constructing a build graph based on what's changed, while handing off the actual building to some entry point in the workspace, would be good enough. Have a weird collection Gradle projects, node projects, test suites, docs, and etc with their own build processes already in place. Some things are also on a "critical" path while others can run async given the context(branch, tag, etc)... I'm rambling though.
- jpeeler 8y ago
  Does anyone know how please (https://please.build https://please.build) compares?

sigil 8y ago

Observe how the verb "force" gets used 6 times. Monorepos "force the conversation." You the individual contributor are "forced to deal with the situation" and "forced to see the upfront cost" of breaking contracts. Your team is forced to "look up from their component, and see the perspectives of other teams and consumers." All this forcing people to do things the Right Way (my way) is surely part of the pushback against monorepos. But set that aside for the moment. Let's suppose defaults should force people to do things the Right Way, and that we also know what the Right Way is. Instead of letting anyone sloppily depend on any code checked into the monorepo, shouldn't we force people to think long and hard about contracts between components -- the default concern in a polyrepo architecture? When and how to make contracts, when and how to break contracts? Isn't this how Amazon moved past their monorepo woes, adopted SOA, built AWS, and became one of the largest companies on earth? Heck, isn't this how the Internet itself was built?

holoway 8y ago
Author here. The Right Way :tm: is situational - there isn't one right answer to things like when and how to make contracts, or how to break them. When I used the term "force", you'll see that I'm usually talking about dialog between people and teams. It's not that it's a single right way to do it. There isn't, and anyone who tells you there is has something to sell you, or is inexperienced enough to not have seen enough of the problem domain. What is for certain: teams need to have tooling that causes the conversations and behavior that lead to the outcomes we want. As systems and teams scale large enough, this tooling becomes essential - without it, teams go their own way, and in so doing, may or may not create the culture needed for the outcomes you want. I have never once in my career, so far, had to tell a team to communicate less. When we're talking about engineering organizations that are large enough to diverge, you must solve these problems somehow, and it needs to be systemic and intentional.
- sigil 8y ago
  Thanks for the response. Out of curiosity, how does your engineering organization introduce new dependencies within the monorepo? Can B, C and D all depend on A without A's consent or even awareness? (Suppose A is some checked in code that's useful, going to see updates in future, but is dormant at present.) Your post puts a lot of the onus on A for breaking B, C, and D, but I think equal care and consideration needs to come from the other side of the contract. Eg, What are you depending on? Is it a dependency you want to take on, or are you and the shared code likely to diverge in life? These are top of mind decisions in a polyrepo architecture, but from my experience they're often not even considered in a monorepo. Anything checked in is fair game for reuse. This is why I suspect you may be "forcing" the wrong thing. For reference I've worked in companies large and small, both monorepo and polyrepo. When I worked on Windows back in the 00's the monorepo tooling (SourceDepot) was quite amazing for the time, but the costs of that sort of coordination were also painfully apparent to everyone. The place I currently work has a monorepo for desktop software and polyrepos for everything else. It isn't a straight up A/B experiment, but anecdotally the pain is higher and shipping velocity lower in the monorepo half of the world. Most of the monorepo pain is related to CI or other costs of global coordination, the kind of things Matt touches on midway (albeit probably too subtlely). I'd be interested to see your counterarguments to those points as well. Do you need fancy dependency management tooling to make your global CI builds fast and reproducible? Matt argues those end up being equivalent to the kind of dependency tooling that's intrinsic to polyrepo architectures anyway.

rdsubhas 8y ago

This is starting to get a debate of "principles", like forcing A and B to talk, or forcing A and B to have more explicit boundaries, and so on. Guess where that ends (hint: it doesn't). With a monorepo, the basic effort you have to put in to start scaling is quite high. To properly do a local build, you need bazel or something. But bazel doesn't stop at just building, but it manages dependencies all the way down to libraries and stuff. Let's say you're using certain maven plugins, like code coverage, shading, etc. Would bazel have all the build plugins your project needs? Most likely not. You have to backport a bunch of plugins from maven to bazel and so on. Guess how many IDEs support bazel? Not a lot. Then you need to run a different kind of build farm. When you check-in stuff to a monorepo, you need to split and distribute one single build. Compared to a polyrepo where one build == one job, a monorepo is like one build == a distributed pool of jobs, which again needs very deep integration with the build tool (bazel again here), to fan out, fan in across multiple machines, aggregate artifacts, and so on. Then the deployment. Same again. There is no "just works" hosted CI or hosted git or anything for monorepos. People still dabble with concourse or so on. And guess what, for a component in its own repo, you don't need to do anything. Existing industry and OSS tooling is built from ground up for that. Just go and use them. To provide a developer a "basic experience" to go from working on, building and deploying a single component – the upfront investment you need to provide with a monorepo is very high. Most companies cannot spend time on that, because scale means different things to different companies. There is a vast gap in the amount of ops/dev tooling you have for independent hosted components vs monorepo tools. Just search for "monorepo tools" or DAG and see how many you can come up with. So what really happens with a monorepo is, most companies go with multi-module maven and jenkins multi-job. The results are easy to predict. I'm not saying that maven/jenkins are bad, but they are _not_ sophisticated, and are not anywhere close to what Twitter/Facebook/Google or any modern company uses to deal with a monorepo (for a good reason). They are just not good at DAG. If you're relying on maven+jenkins as your monorepo solution, all I can say is "good luck". Instead, if you start by putting one component in one repo, you keep scaling for _much longer_ before you hit a barrier. In principle, monorepos are better. In practice, they don't have the basic "table stakes" tooling that you need to get going. Maybe monorepo devops tooling is a next developer productivity startup space. But until then, it's not mainstream for very good reasons.

coldtea 8y ago

>But I think Matt’s argument misses the #1 reason I’ve flipped quite hard to a monorepo perspective as my own level in the organization has gotten higher Perhaps the fact that since their level was now higher, they wouldn't have to deal with the nitty gritty details and pain of working with a monorepo as a developer? E.g. I wasn't for it when I was a dev, but now that I can just impose it on others, I love it. Same with how various 'development process' rituals are adopted...

AzzieElbab 8y ago

As engineers we spend wast amounts of time in constant search for a rival to "tabs vs spaces" debate

luord 8y ago

Yet another chapter in one of the big flamewars. Seeing as I fall in the monorepo camp, I must say I mostly agree; also, I much prefer this tone for an article. I find it enjoyable how plenty of comments both here and in the other discussion are of people saying "We had a mono/polyrepo and things improved tremendously when we migrated towards a poly/monorepo". The issue might be one of growth and complacency: a drastic change like that forces the team to face the technical debt that was being ignored and do a better implementation using what was learned from past mistakes.