31 ms·

Monorepos: Please don’t

• superasn 8y ago
  The biggest advantages monorepos have offered is development of tools like lerna(1) or yarn workspaces. Before that there used to be a node_modules folder with GBs of [useless] data in all my projects. Now there is just one folder on top and that's it. Also if you're developing lots of modules or plugins it makes it super to work without committing changes since they are symlinked. (1) https://lernajs.io https://lernajs.io
  • WorldMaker 8y ago
    node_modules is an interesting worst case of package management systems. There's some good exploratory work currently happening on making node_modules and the node package ecosystem better in general, but especially in the polyrepo case. Yarn "Plug'n'Play" is one, and Tink [1] the other. [1] https://npm.community/t/tink-faq-a-package-unwinder-for-javascript/3191 https://npm.community/t/tink-faq-a-package-unwinder-for-java...
• erik_seaberg 8y ago
  I wish this had touched on polyrepos' ability to pin known-good versions of dependencies; that tends to be the Achilles' heel of monorepos.
  • muro 8y ago
    Yet who unpins them or updates when a new good version is available?
    • erik_seaberg 8y ago
      There's a cost for being out of date, but there's also a cost for learning the hard way whether a new version breaks prod. Pay it down like any other tech debt. Maybe I could test literally every release version of each of my dependencies, but that isn't really my job.
      • WorldMaker 8y ago
        Greenkeeper (and similar systems) comes to mind, too, in the polyrepo case. You can still CI with "the latest" in the polyrepo case. We have the technology to automate that. Including situations like 'let me know when the next version of my dependency that passes this test is released and send me a PR to update my pinned version when it happens'.
    • adrianN 8y ago
      The person responsible for dependency management in your team. You have one right?
  • rhinoceraptor 8y ago
    NPM might not be the best package manager, but if you're using something like Lerna you can get the best of both worlds. Your local copy of an internal dependency can either be symlinked to the local source code, or a copy of the published package. That makes it a lot easier to work on a package and its consumers at the same time.
    • mrgriffin 8y ago
      Oh God! I had forgotten how much frustration was in my old Python team until I checked the docs and discovered that you can make pip "install" your local copy of one repo as the dependency for another. The poor developer who prompted me to check was testing by doing CI builds and pulling down the new eggs. Of course this isn't really a point against polyrepos, since it had a solution, but it's definitely something that I could imagine catching out lots of juniors.
  • paulddraper 8y ago
    That can be good, or that can be bad. Someone makes an incompatible change, but you do not find out months later, because the client of that incompatibility is not using the latest versions. In the meantime, the development of the module and its downstreams has marched past any sort of easy resolution, and you essentially are now maintaining two different copies of your code.
  • rossjudson 8y ago
    Hmm. Monorepos strongly favor "always-good". Core library breakage gets detected and rolled back really fast. It's been months since I've been affected by one. Known-good is in the eye of the beholder, and is just another dimension that generates breakage.
    • erik_seaberg 8y ago
      google3 breaking was a bimonthly event that left the entire team aground. Bad enough that it happened, but the worst part is that we couldn't do anything about it but wait.
      • rossjudson 8y ago
        I guess my end of google3 hasn't been like that in a long time.
• pbiggar 8y ago
  Alternate title: monorepos - ideal for teams under 100 devs
  • nomel 8y ago
    The title should have been "Monorepos: Please don't, at scale", but I suspect it was made intentionally controversial.
    • tensor 8y ago
      Which large companies use polyrepos? Google, Facebook, Microsoft, Uber, all use monorepos.
      • hiyou102 8y ago
        Microsoft doesn't use a monorepo. Bloomberg, Dropbox, and Amazon don't either.
        • ec109685 8y ago
          Microsoft switched all of windows to a mono repo.
          • lostmsu 8y ago
            I think Windows was already a single repo forever. IMHO, that's exactly what is slowing it right now. Other products have their own repos, sometimes multiple. As of 2016.
            • ec109685 8y ago
              No, they had multiple for windows: https://blogs.msdn.microsoft.com/vsappcenter/how-mono-repo-and-one-infra-help-us-deliver-a-better-developer-experience/ https://blogs.msdn.microsoft.com/vsappcenter/how-mono-repo-a...
      • 8y ago
• sterlind 8y ago
  Could you get the best of both worlds by having a monorepo of submodules? Code would live in separate repos, but references would be declared in the monorepo. Checkins and rollbacks to the monorepo would trigger CI.
  • joeblau 8y ago
    We actually did this. When I started at Uber ATG one of our devs made a submodule called `uber_monorepo` that was linked from the root of our git repo. In our repo's `.buckconfig` file we had access to everything that the mobile developers at Uber had access to by prefixing our targets with `//uber_monorepo/` We did however run into the standard dependency resolution issue when you have any loosely coupled dependency. Updating our submodule usually required a 1-2 day effort because we were out of sync for a month or two.
  • naniwaduni 8y ago
    There's not much good to either world. You need fairly extensive tooling to make working with a repo of submodules comfortable at any scale. At large scale, that tooling can be simpler than the equivalent monorepo tooling, assuming that your individual repos remain "small" but also appropriately granular (not a given--organizing is hard, especially if you leave it to individual project teams). However, in the process of getting there, a monorepo requires no particular bespoke tooling at small or even medium scale (it's just "a repo"), and the performance intolerability pretty much scales smoothly from there. And those can be treated as technical problems if you don't want to approach social problems. To put it another way, we're comparing asymptotic O(n) with something bigger, neglecting huge constant factors on the former. There's a lot of path-dependence, since restructuring all your repos with new tooling is hard to appreciate.
  • rkangel 8y ago
    This is the approach taken by a number of projects. The ones I am most familiar with are the OpenEmbedded/Yocto/Angstrom family that build Linux for embedded devices. They have a 'root' repo that references the layer repos (using metadata files rather than submodules), and there is a tool that does the pulling. It's optimised for pulling not committing though, I don't think the tooling helps much with bumping versions. It can be misused though - the releases of the root repository reference the children by tags usually. Someone retagged a child repo and we suddenly had build failures.
• towaway1138 8y ago
  My polyrepo cautionary tale: Two repos, one for fooclient, one for fooserver, talking to each other over protocol. Fooserver can do scary dangerous permanent things to company server instances, of which there are thousands. Fooserver sprouts a query syntax ("just do this for test servers A and B"), pushed to production. Fooclient sprouts code that relies on this, pushed to production. A bit later, Fooserver is rolled back, blowing away query syntax, pushed to production. "Just do this for test servers A and B" now becomes "Do this for every server in the company". Hilarity ensues.
  • lioeters 8y ago
    Ouch. I suppose the lesson is that a monorepo with both client and server being developed and tested together would have reduced such risk.
    • yellowapple 8y ago
      Versioning the client/server interface would've also reduced such risk.
• im_down_w_otp 8y ago
  The biggest gripe I have with modern day monorepos is that people are trying to use Git to interact with them, which doesn't make a tremendous amount of sense, and results in either an immense amount of pain and/or the creation of a bunch of tools to try to coerce Git into behaving basically like SVN. Which of course begs the question, rather than trying to perform a bunch of unnatural acts, why not just use SVN to start with? It works extremely well with monorepo & subtree workflows. Sure it has some warts in a few dimensions around branching, versioning, etc. compared to Git when using Git in ways aligned with how Git wants to work, but those warts are minimal in comparison to what's required to pretzel Git monorepos into scaling effectively.
• thehazard 8y ago
  Better title would be "Monorepos don't fit with my particular use case."
  • WorldMaker 8y ago
    I disagree. The thesis statement repeated several times throughout is "Monorepos don't scale in exactly the same ways that polyrepos don't scale, the tools to solve the scaling problems are the exact same except monorepos need more of them and encourage bad habits along the way." You may disagree with that thesis, but it definitely seems to cover more than one use case.
    • thehazard 8y ago
      ...the author's title is literally "monorepos: please don't!"
  • forrestthewoods 8y ago
    I strongly agree. I hate this style of blog post. Telling people what they should or should not do is generally absurd. Every situation is unique and you can't possibly know another project's requirements or acceptable trade-offs. A better approach, in my opinion, is "Here's what we did and why". The author clearly has experience in the area. Great! Tell me about your problems. Tell me about your attempted solutions and what did or did not work. Tell me what you wish you had done! I'd love to use knowledge of your situation to inform my own decision making. But don't be surprised if my circumstances are different and lead me to prefer different trade-offs and choose a different solution. That doesn't make me a zealot or an idiot.
    • icebraining 8y ago
      Isn't your own post telling people what they should and should not do (specifically on how to give advice)?
      • deleted 8y ago
        [deleted]
• mrgriffin 8y ago
  My problem with polyrepos is that often organizations end up splitting things too finely, and now I'm unable to make a single commit to introduce a feature because my changes have to live across several repositories. Which makes code review more annoying because you have to tab back and forth to see all the context. It's doubly frustrating when I'm (or my team is) the only people working on those repositories, because now it doesn't feel like it gained any advantages. I know the author addresses this, but I can't imagine projects are typically at the scale they're describing. Certainly it's not my experience. Also I definitely miss the ability to make changes to fundamental (internal) libraries used by every project. It's too much hassle to track down all the uses of a particular function, so I end up putting that change elsewhere, which means someone else will do it a little different in their corner of the world, which utterly confuses the first person who's unlucky enough to work in both code bases (at the same time, or after moving teams).
  • WorldMaker 8y ago
    It's an interesting social problem in how you manage those project / library / repository boundaries. On the flipside, though, it's been well documented that among many of the major monorepos those boundaries still exist, they just become far more opaque because no one has to track them. You find the weird gatekeepers in the dark that spring out only when you get late in your code review process because you touched "their" file and they got an automated notice from a hidden rules engine in your CI process you didn't even realize existed. In the polyrepo case those boundaries have to be made explicit (otherwise no one gets anything done) and those owners should be easily visible. You may not like the friction they sometimes bring to the table, but at least it won't be a surprise.
    • rgoulter 8y ago
      http://wiki.c2.com/?ConwaysLaw http://wiki.c2.com/?ConwaysLaw "Conway's Law" is something like "organisation of code will match the organisation of people". It's a neat description. I think it's more common to merge or split modules and classes than repositories. I wonder if there'd be less tension if repos and teams were 1:1 though.
• aaron695 8y ago
  Without naming the size, this just causes people who actually agree to argue. I assume this does not apply to 99% of people using a repo.
• jonex 8y ago
  Seems like the main point is that you'll still need to add additional tooling (search, local cloning, build, etc) to handle scaling, something you can do just as well with polyrepos. Conversely, for polyrepos, you can add tooling to fix issues with dependency management and multi-project changes/reviews. However, the author figures that monorepos engourage bad code culture and points out that Git is hard to build a monorepo on. To me this message seems a bit shallow, of course we can build tooling to hide the fact that we have a polyrepo. Given well enough built tooling and consistent enough polyrepo structure (all using same VCS, all being linked from common tooling, following common coding standards and using the same build tooling, etc.) the distinction from having a monorepo is more of an implementation detail. Given the choice between a consistent monorepo where everyone is running everything at HEAD and a polyrepo where each project have their own rules and there's no tooling to make a multi-project atomic change, I'd go for the former. Given the choice between identical working environments but different underlying implementations I would go for whatever the tools team think is easier to maintain.
  • woolvalley 8y ago
    What is the tooling for multi-repo atomic synchronized commits? Monorepo's give you that for free, which is the reason why I think monorepo projects exist. SVN kind of gave you partial checkouts, which was helpful.
    • jonex 8y ago
      Yes I think so too. But of course, as the article points out, nothing is entirely free. At some point we will have to build tools to handle scaling, and then the trade offs between a mono and polyrepo becomes less obvious. I'd lean towards monorepos as a base either way, but given sufficiently well working tooling it might not matter much.
    • bluGill 8y ago
      Polyrepo argues that is a non-feature and don't give it to you. You can figure out where things are, but you never get synchronization. This is a good thing because when you have to make the multirepo commit you make the change and then update each downside one at a time. Each change is much smaller and so easier to review (and also easier to find the right reviewer). Of course the downside is you either have to maintain both ABIs (not just API), have a rollout scheme with two version of the upstream library exist side by side, or don't release. Nothing is perfect.
• paulddraper 8y ago
  > is there any real difference between checking out a portion of the tree via a VFS or checking out multiple repositories? There is no difference. How big is your monorepo? Assume each line of code is a full 80 characters, stored via ASCII/UTF-8. That 67 million lines of code in 5GB. I can fit five of those on a Blu-ray. > The end result is that the realities of build/deploy management at scale are largely identical whether using a monorepo or polyrepo. True. > It might be deployed over a period of hours, days, or months. Thus, modern developers must think about backwards compatibility in the wild. Depends entire on the application. Lots of changes are deployed within short periods of time with low compatibility requirements. > Downside 1: Tight coupling Monorepos do often have tightly coupled software. Polyrepos also often have tightly coupled software. Polyrepos look more decoupled, but pragmatically I can't say I've noticed a much of difference. > Downside 2: VCS scalability I've also heard Twitter engineers complain about the VCS. But what is the scope of the author's discussion? 1,000 engineer orgs? Or 20 engineer orgs? Those are vastly different levels of engineering collaboration. I assume the article was not written to cover both of those. Or was it? --- Ultimately, I think the author implicitly assumed a universe of discourse of gigantic repos with hundreds and hundreds of daily contributors. When people talk about the spectrum of monorepo vs polyrepo architectures, that is very extreme. For example, last I knew, Uber has more repos than it did engineers. And I don't assume that "polyrepos" always means multiple repos per engineer.
• rbetts 8y ago
  Unless you are pure OSS or pure closed source - you end up with a poly-repo strategy regardless as you split open and closed code, suffering the annoyances of both systems.
  • madhadron 8y ago
    No, what you end up with is a system for mirroring open source code into your repo, and a system for mirroring commits that should be open source from your code into external repos. All active work still happens in a monorepo.
    • rbetts 8y ago
      Any good examples of small companies that pull this off and the bots/CI/tool whatever they use to do it?
• ajuc 8y ago
  I call bullshit on "our repository is too big for one machine". Seriously, you have over 1 TB of code and 100 people wrote it?
  • dmarlow 8y ago
    You can have large repos and it not only be code. I remember seeing repos many tens of gigs because all of VS was versioned as well for "reproducibility".
  • Xylakant 8y ago
    adding raw versions of binary assets (designs, video, ...) can quickly lift a repo beyond a TB. Now, you could say "don't do that", but there's valid use cases where you'd want to track all binary assets as part of the development cycle.
    • tome 8y ago
      Ouch, well, yes that is a very good situation in which not to take the "mono" part of "monorepo" too seriously.
      • daemin 8y ago
        You can do that with content, you just partition the workspace/view of the monorepo to what each person needs rather than checking the entire thing out git style.
      • Xylakant 8y ago
        or use a VCS that allows partial checkouts of repositories. There's no DVCS that I know of that can do that, but for example SVN can. Git LFS might be an option, too. There are also commercial products that target that market. I just wanted to point out that reaching a measly TB of data doesn't require much effort. (worked on a product that would version rendered clips for special effect production).
    • dllthomas 8y ago
      See also: https://git-annex.branchable.com https://git-annex.branchable.com
• curtis 8y ago
  My advice is that if components need to release together, then they ought to be in the same repo. I'd probably go further and say that if you just think components might need to release together then they should go in the same repo, because you can in fact pretty easily manage projects with different release schedules from the same repo if you really need to. On the other hand if you've got a whole bunch of components in different repos which need to release together it suddenly becomes a real pain. If you've got components that will never need to release together, then of course you can stick them in different repositories. But if you do this and you want to share common code between the repositories then you will need to manage that code with some sort of robust versioning system, and robust versioning systems are hard. Only do something like that when the value is high enough to justify the overhead. If you're in a startup, chances are very good that the value is not high enough. As a final observation, you can split big repositories into smaller ones quite easily (in Git anyway) but sticking small repositories together into a bigger one is a lot harder. So start out with a monorepo and only split smaller repositories out when it's clear that it really makes sense.
  • forty 8y ago
    My rule of thumb is: if you need to do PRs in several repositories to do one features, you should probably merge the repositories. At work, we have code spread among a bunch of repositories, and having to link to the 2/3 related PRs in other repos is a major PITA, and even more so for the reviewers.
    • tigershark 8y ago
      Not always. It makes absolutely sense to have a repository for the gui and one for the server. When writing a new feature you usually write some gui code and some server code and create different pull requests. I think monorepos are seriously wrong and I completely agree with this article.
      • Vinnl 8y ago
        Well... Why does that make sense? I have a repository containing both the GUI and the server, and sometimes I have to make changes to both. Locating those related changes together in the same commit and/or PR makes a lot of sense to me: the changes depend on each other, and thus should be reviewed together. What's the advantage of splitting them up?
• sandov 8y ago
  I agree, but also... Medium: Please don't
  • erikpukinskis 8y ago
    Why?
    • sandov 8y ago
      TL;DR: Really bad user experience. For a longer explanation see https://medium.com/@nikitonsky/medium-is-a-poor-choice-for-blogging-bb0048d19133 https://medium.com/@nikitonsky/medium-is-a-poor-choice-for-b... Alternatives better than medium: Wordpress, Blogger, github pages, plain html files.
      • garmaine 8y ago
        “For a longer explanation: see this Medium post.” There is a certain irony there which betrays one of the problems left unaddressed.
        • sandov 8y ago
          The purpose of it being a Medium post is so you can instantly see what he's talking about.
  • benatkin 8y ago
    Agree 100%. "Pardon the interruption" followed by an article with a fixed top bar asking me to "become a member" (supporting an anti-open web tech company like it's a charity) and a fixed bottom bar asking me to sign up.
• olingern 8y ago
  I’ve found monorepos to be extremely valuable in an immature, high-churn codebase. Need to change a function signature or interface? Cool, global find & replace. At some point monorepos outgrow their usefulness. The sheer amount of files in something that’s 10K+ LOC ( not that large, I know ) warrants breaking apart the codebase into packages. Still, I almost err on the side of monorepos because of the convenience that editors like vscode offer: autocomplete, auto-updating imports, etc.
  • kazagistar 8y ago
    Monorepos and packages are not mutually exclusive. You can and should have many different projects in subfolders I'm your monorepo, each with their own builds and tests and artifacts (though hopefully somewhat standardized). The point is that now it's easy to release changes across multiple projects, integration test between them on a specific global patch, etc, without a whole pile of complex tooling.
    • olingern 8y ago
      Agreed. When I wrote the parent comment, I was thinking of a time I prematurely abstracted an API wrapper to a private git repo and how painful simple, frequent changes were. Though, as you say and I commented below they’re not mutually exclusive. A wrapper or even an entirely separate service can exist alongside others. One dark side of this is being able to “reach inside” other parts of the monorepo and blur application boundaries.
  • _untra_ 8y ago
    This guy gets it. Software Engineering is about using the appropriate tools and techniques for the task at hand. If your repo gets so large it can't be comfortably checked out, something needs to get split apart. Monorepos are also a great technique for tackling large legacy codebases. When the rot is all in one designated place, it becomes easier to encourage good developer habits on new code created in new, separated repo(s). Speaking from experience I've worked on a team operating through a monorepo project that came out real well. The codebase was mostly golang, so everything lived in the GO_PATH, but for the most part the typescript in the UI side of the repo didn't complain. Testing and code quality was a higher priority, as well, which may have contributed to its success. I have also worked on a monorepo project that had minimal tests and automation, that soon grew monstrous and ultimately needed refactoring. That was a big pile of coffeescript, es6 and java that ultimately refactored into three different node modules and two microservices. Javascript and its module packaging tends to conform better to polyrepo patterns. golang code all wants to be in the same place, and java repos have their own desired nested directory structures. These two languages tend to encourage monorepo design patterns. Monorepo or Polyrepo, the correct answer is whatever works for your team and task at hand.
• kerng 8y ago
  Static analysis is easier on monorepo. At least one can run it on all code. Polyrepo has the problem that some code is off the radar. That might be the only advantage of monorepo in my opinion.
• sfrench 8y ago
  My last 2 jobs have been working on developer productivity for 100+ developer organizations. One is a monorepo, one is not. Neither really seems to result in less work, or a better experience. But I've found that your choice just dictates what type of problems you have to solve. Monorepos are going to be mostly challenges around scaling the org in a single repo. Polyrepos are going to be mostly challenges with coordination. But the absolute worst thing to do is not commit to a course of action and have to solve both sets of challenges (eg: having one pretty big repo with 80% of your code, and then the other 20% in a series of smaller repos)
  • 01100011 8y ago
    Jesus, this. Look, you're going to run into issues either way, because you're trying to solve a difficult problem. It's like thinking OOP or functional programming is going to solve all your issues... I mean, in some limited cases they could, but realistically you're just smooshing the difficulties around and hopefully moving them to somewhere where you are more able to deal with them. FWIW, I've worked in a many-repo org and it sucked worse than huge companies with monorepos and good tooling, but I'm not going to make some blanket statement because it depends on the specifics of your code/release process/developer familiarity etc.
    • brodo 8y ago
      This. Every decision is a trade-off. There is no silver bullet. Context matters.
  • btschaegg 8y ago
    Sounds reasonable. I'll have to add, though, that the underlying technology factors into this as well. For example: If you're stuck with a TFS monorepo (you poor soul), you actually get to deal with both problems to some extent, since TFS doesn't enforce that you check out the intire repository at once. This can have very "funny" situations because someone forgot to checkout new changes in some folder. OTOH, at least for releases, you can remedy this by using CI everywhere.
  • jeklj 8y ago
• sftwds 8y ago
  >Scaling a single VCS to hundreds of developers, hundreds of millions lines of code... Maybe I am way out of my element here, but is this a common problem? Do companies with only “hundreds of engineers” really have “hundreds of millions of lines of code”?
  • hermitdev 8y ago
    From personal experience, it can happen. At one point, I was personally responsible for about 2 million lines of code. Over several years, I was able to reduce it to about 500k through generous use of code generation for ORM type work. The generated code never ended up in VCS, but the generator and model did. Certainly helped checkout/update times as there was several thousand fewer files to deal with. I was one of about 900 engineers at a financial company of about 1500 employees at the time. I don't honestly know how many lines of code there were across the company, but I imagine it easily exceeded 100M. It took us a full week to do a full recompile of everything. We had no CI... Was always a problem approaching release time.
    • tome 8y ago
      > I was one of about 900 engineers at a financial company of about 1500 employees at the time. Can you say which company it is, or give a few more details? I'm fascinated to know which financial company can consist of 60% engineers!
      • hermitdev 8y ago
        Sorry, no, NDA at what not. But, it was a very technology driven hedge fund, not one of the big banks you read about in the news.
• titzer 8y ago
  No silver bullet here, I think. It's definitely the case that a mega monorepo doesn't, in practice, have the atomic commit property. E.g. once you add owner files and separate code reviews, you're in for a world of hurt. Case in point, Google developed an internal tool to split cross-cutting CLs into manageable pieces, wrangle all the owners and approvals, presubmits, etc, and then submit the CL piecemeal--i.e. not atomic. Chromium uses a different model. It just DEPS's in other repos at pinned versions. That has a whole other set of problems.
  • EricBurnett 8y ago
    (Disclaimer: at Google) It's not quite so black and white. It's true that repo-wide refactorings often get carved into little changes and so aren't made atomically, but they're the exception rather than the rule. Any small change, e.g. changing an interface and the 5 callers of it, _can_ be made atomically. And changing code that's reused a small number of times is a far more common case then changing core libraries the whole company uses, so atomic submit ends up being hugely valuable.
• 0xFACEFEED 8y ago
  At least the author gave us the courtesy of italicizing his broken assumption from the outset of the post. > Because, at scale, a monorepo must solve every problem that a polyrepo must solve, with the downside of encouraging tight coupling, and the additional herculean effort of tackling VCS scalability. Right. But you have to get to "scale" first (as it relates to VCSs). Most companies don't. Even if they're successful. Introducing polyrepos front loads the scaling problems for no reason whatsoever. A giant waste of time. Checkmate! I didn't even need a snarky poll. The irony of that poll is that it clearly demonstrates his zealotry, not other people's.
  • adrianN 8y ago
    You can't split monorepos after the fact, at least not without immense costs. You can always just put all your small repos into a big one.
    • btschaegg 8y ago
      I think there's a nuance to this that should be pointed out: Monorepos allow you to do very bad hacks (I need this other component over there; let me just put in a Symlink. Done.). And if people can, they will use those hacks. If you split your repo up from the get go, the worst thing you can get that you'll have to assemble multiple distinct, well-encapsulated (in terms of project structure) things into one. In Git, that could lead to multiple root commits, but that's about it.
      • wcdolphin 8y ago
        No. The worst case is that the engineering team spent more time working on “well encapsulated projects” than on the most important project for their business and are all now out of jobs. Most companies don’t fail because of tech debt. And certainly not because of version control tech debt.
        • 0xFACEFEED 8y ago
          Exactly. Whenever I see an engineer take a hardline position (eg: "no monorepos you zealots!") I always ask myself: is this person just annoyed? Most of the time they're just annoyed. One side effect of every successful business are annoyed worker ants that are sick of dealing with growth problems. I've been there. I know how annoying it can be. Personally I've found comfort in embracing the chaos and learning to manage it responsibly. No dogma. No absolutes. Know how to do monorepos well. Know how to do polyrepos well. Learn the pitfalls of both. Don't assume other people are stupid zealots.
• peterwwillis 8y ago
  "The frank reality is that, at scale, how well an organization does with code sharing, collaboration, tight coupling, etc. is a direct result of engineering culture and leadership, and has nothing to do with whether a monorepo or a polyrepo is used. The two solutions end up looking identical to the developer. In the face of this, why use a monorepo in the first place?" .....because, as the author directly stated, the type of repo has nothing to do with the product being successful. So stop bikeshedding, pick a model, and get on with the real business of delivering a successful product.
• shereadsthenews 8y ago
  This argument boils down to people who have used Perforce, who believe in the benefits of a monorepo, and people who have only ever used git, who do not. While it's true that git is a terrible program that does not lead to conclusions about the merits of a monorepo.
• karmakaze 8y ago
  TL;DR The post just reads like some opinionated piece for traffic. The author has never even used a monorepo as far as I can tell, so can only argue from one side, the best one ever used: polyrepo. Then goes on to list 'theoretical' benefits and the downsides (which should also be theoretical if having never been used) of monorepos. It concludes with "The two solutions end up looking identical to the developer. In the face of this, why use a monorepo in the first place? Please don’t!" implying that 'Google, Facebook, Twitter, and others' do it for no benefit.
• wtracy 8y ago
  I'm not familiar with how monorepos work in practice, but it seems obvious to me that it's going to complicate everyday tasks. Ready to commit? Whoops, another team made a bunch of commits to their project, and you need to rebase your project before you can commit. (I'm having flashbacks to Clearcase already.) Need to roll back the last two commits you made? Sure, that takes two seconds--oh, wait, another team made multiple commits that got interleaved with yours. Have fun cherry picking the files you want to revert. Of course, I'm apparently a curmodgeon, because as soon as someone starts talking about running a find/replace globally across multiple projects, I want to grab something sharp.
  • headcanon 8y ago
    It seems facebook built tooling to address that issue. I doubt they would tolerate that lifestyle either: https://code.fb.com/web/rapid-release-at-massive-scale/ https://code.fb.com/web/rapid-release-at-massive-scale/
  • joshuamorton 8y ago
    Why would you need to rebase or cherrypick unless you and the other team were touching the same files?
    • notacoward 8y ago
      Problem is, places that use monorepos also tend to have whole teams full of people who feel entitled to f#ck with stuff across the entire repo, often without going through the normal review or other processes for each component. Thus it's not uncommon to commit code early in the day so that you can build packages for system test, find a problem in system test, then come back later the same day to find one of those randos has already "fixed" your code to adhere to some new standard they decided on at lunchtime. Now reverting only your own commit is a mess, and reverting theirs as well invites a s###storm of epic proportions because they're higher-caste than you.
      • eadmund 8y ago
        > places that use monorepos also tend to have whole teams full of people who feel entitled to f#ck with stuff across the entire repo That's only true to the extent that the statement 'places that use multirepos also tend to be full of people who feel entitled to f#ck with stuff across the entire codebase' is. Bad colleagues can cause damage either way. I appreciate that with a single repo, bad colleagues are at least forced to have passing tests after their changes, rather than leaving it to me to pick up the pieces.
• vikingcaffiene 8y ago
  I am sure this author means well but I respectfully disagree with this advice. The author is arguing against the monorepo approach and then proceeds to list out some of the most successful software companies on earth as reasons NOT to do it. The reason they were able to get to their lofty heights was in some part because they used a monorepo. The biggest advantage of a monorepo is you can move quickly and understand the implications of changes since everything is housed under one roof. That's critical for startups IMO. By the time you reach the "scale" the author is talking about, you have the resources to deal with it. Is it hard? Yes. Is it worth throwing out the baby with the bathwater? No IMO. I currently work in a polyrepo word that the author is encouraging. I can tell you it f*cking sucks. Just take the very simple example of firing up your dev environment. In a polyrepo world, you have to individually fire up each codebase or write up some sort of script to do that for you. The former example sucks for obvious reasons and the latter example makes the case for a monorepo since one dev could author a script that could then be used by all (since he/she will know the paths to all things that need to start). Don't even get me started on setting up an environment from scratch. Containers make this easier but again, it would be nice to just rock `./start.sh` and be off to the races. A monorepo can give you that. Pulling/pushing changes to your vcs becomes a tiresome error prone nightmare since now you need to remember to run git pull on all the codebases that touch the area you are working on. You might forget to pull on one of those codebases and everything starts breaking and now you need to stop and track it down. Dumb error? Yep. Not a thing in a monorepo? Yep. PR's become really sucky because now you need to harass your team for n PR's instead of just the one if the feature you are working on cuts across codebases. I've worked in some fairly large monorepo codebases with lifespans of >10 years and I can tell you that I have yet to encounter any of the issues with VCS scaling the author speaks of. In the future if I find myself in a situation like that you know what I'll do? Migrate to a more performant solution like Mercurial or something. Will it suck? Sure. But not as much as dealing with a polyrepo. Then there's dependency management. Holy sweet mother of god dependency management is the worst. Lets say you need to make a breaking change to one of your codebases, in a monorepo (with decent test coverage or a type system worth a damn) you have a decent chance of tracking down everything that needs to get patched. In a polyrepo? Phttt! Enjoy those bug reports from your customers and 1am hotfixes bruh. I really really wish people on here would stop trying to solve problems of "scale" when that's literally the last thing you need to worry about. Being able to respond quickly to business requirements is the only thing you should be worrying about until its obvious that you've made it. Then feel free to worry about scale.
• laurencei 8y ago
  Can anyone here explain to me how a monorepo like Google or Facebook handles security? If I pull the repo - I have the entire contents of Google or Facebook? Is that right? Surely that lacks the normal security measures around what must be highly sensitive information, so there must be more to it than I know of?
  • madhadron 8y ago
    Sensitive information like user data or certificates doesn't live in a source repository. It lives in databases and systems for managing certificates. Treating straight source code as sensitive information is security by obscurity.
    • vageli 8y ago
      > Treating straight source code as sensitive information is security by obscurity. For many companies, their source code is a large part of their business.
      • deleted 8y ago
        [deleted]
  • joshuamorton 8y ago
    (there's an acm paper about Google's repo that dives deeper into this). First thing, you can't just "pull the whole repo" at Google or fb scale. It doesn't fit on a single hard drive. This means the enitre repo is normally accessed via networked means. As a result, builds can also be done over the network transparently. So building and testing is done as a different user. That user can have different privileges than the individual requesting the build. So there is a way to hide source code so that only the output artifacts (compiled binaries) can be accessed. But I think the other part of that is that that's normally a tiny minority of code. The other option is of course to live outside the monorepo, as some projects do.
    • ddulay 8y ago
• malkia 8y ago
  I do really like mono-repos, but google's other significant new project: fuchsia - is set-up as multi-git repo (and I believe chromium too, maybe android (haven't checked)). For fuchsia, they use a tool called "jiri"[1] to update the repos, previously (and maybe still in use) is the "gclient" sync tool [2] way from depot_tools[3] [1] - https://fuchsia.googlesource.com/jiri/ https://fuchsia.googlesource.com/jiri/ [2] - https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/master/gclient https://chromium.googlesource.com/chromium/tools/depot_tools... [3] - https://chromium.googlesource.com/chromium/tools/depot_tools.git https://chromium.googlesource.com/chromium/tools/depot_tools... It even reflects a bit to the build system of choice, GN (used in the above), previously gyp, feels similar on the surface (script) to Bazel, but has some significant differences (gn has some more imperative parts, and it's a ninja-build generator, while bazel, like pants/bucks/please.build is a build system on it's own). Simply fascinated :), and can't wait to see what the resolution of all this would be... Bazel is getting there to support monorepos (through WORKSPACEs), but there are some hard problems there...
  • robaato 8y ago
    Having worked with some organisations building on Android (>1,000 repos), life is not easy when you are trying to build on top of it and regularly take updates etc. I asked one company how many changes required changes to more than one repo and was told "a small percentage". We then did some basic analysis of issue IDs across commits and discovered that it was in reality nearer 30% of changes. Keeping those together was just plain very hard. Start to scale this by teams of hundreds or thousands of devs and you get a lot of pain. Managing branches is also hard - easy to create (with repo tool) - but hard to track changes.
  • IshKebab 8y ago
    Funny that there are so many reimplementations of git submodules but with support for "just give me HEAD" - Google has two (jiri and repo), my company has a home-grown one too.
  • malkia 8y ago
    "Bazel is getting there to support monorepos" -> support multiple repos (was tired when I wrote this I guess)...
• dajonker 8y ago
  I've been in a project where some (authoritative) people had a tendency to split things into separate repositories for very small things, e.g. repositories with a single class. This was pure developer hell. Any change meant changing at least 3 repositories, including a review for each change. Never understood this decision as all parts needed to be on the latest version anyway. Caused lots of dependency and versioning issues too.
• madhadron 8y ago
  The truth is that you're not going to get to make this decision. If you're starting greenfield, you're going to start a single repo for your project. If that greenfield is the whole company and everything is part of that project, you get a giant monorepo. If greenfield is a new division that's not part of another project, you're going to create a new repo, and now you're in a polyrepo environment. Which way it goes is determined by the environment, wherein the engineers do the sensible thing at the time. Then you do the engineering to solve the problems with whatever way you went.
  • icebraining 8y ago
    I've started a greenfield project that involved multiple modules, and I created a different repo for each, but I could just as easily have created a single repo.
  • sfink 8y ago
    At Mozilla, we started with a monorepo, then went to a mostly-monorepo, then consolidated just about everything back into a monorepo, and have since "decayed" a bit to a mostly-monorepo. There were major gains from merging our source tree with the continuous integration support code and configuration. We've pretty much always vendored selected third-party code, so that didn't really change. Large collections of tests and their infrastructure have been much easier to manage as part of a monorepo. Given that our tooling efforts are mostly in handling a monorepo, I can't really judge how differently things would be if we had gone full multirepo -- my experience with multirepos has been pretty awful, but that's an unfair comparison since we intentionally haven't worked on tooling for it. We solved our worst multirepo problems by going monorepo, but I'm sure other projects have solved their worst monorepo problems by going multirepo, so neither really proves anything. The push for separate repos these days mostly comes from social reasons -- if you have a separable piece and want external contributors, that's a strong motivator for putting it on github since that's just Where Things Are these days. No matter how good your tooling and issue tracking and whatever is, even if it's far superior to github's, it doesn't really matter. People have to learn yours; they already know github's. I don't particularly like github's workflow, but I still use it.
• sytse 8y ago
  The article is great summary of the pros and cons. What is still missing from the default tooling is a way to make a change across repos. At GitLab we're working on group merge requests to solve this https://gitlab.com/gitlab-org/gitlab-ee/issues/3427 https://gitlab.com/gitlab-org/gitlab-ee/issues/3427
• sierdolij 8y ago
  Polyrepos are the way to go: - Semantic versions. - Group components into reusable packages. - Don't use git modules or other source cloning in builds, use native/platform package management. - Access control is made much easier. - Sign commits and tags. - Code review either before- or after-the-fact, just do it(tm). - Reproducible builds - strip out timestamps/random tokens/unsorted metadata. - Create CHANGELOGs semi/automatically. - Eliminate manual steps altogether. - Distributed builds/build caching (distcc, ccache). - TDD smoke tests should run automatically in dev on save with 10 seconds. Bonus points for running personal TDD sandbox on faster remote servers via rsync and trigger on file-save. - Standardize on 1-3 languages. - Services composed of simpler 12factor microservices, not monorepo megaservices. Deploy fuse switching, proxying, HA/redundancy, rate limiting, monitoring and performance stats collection just like macroservices.
  • joshuamorton 8y ago
    About half of these aren't specific to polyrepos. changelogs, reproducible builds, code review, signing, grouped components, distributed builds, TDD smoke test, standardized languages, and microservices are all possible (and just as easy) in monorepos. You no longer need to worry about versioning, which means no manual updates of either your package or updating dependency versions. Although access control is more difficult, that doesn't seem like a good enough reason to make this kind of decision.
• chvid 8y ago
  Does a lot of the pain from a monorepo come from trying to use a tool - Git - that is explicitly designed to support distributed repositories? Wouldn't things be easier if you used eg. Subversion instead? That is a tool that was designed around a client/server paradigm and had a single repository as its main use case.
  • icebraining 8y ago
    Git was designed for a monorepo (the kernel). When people talk about monorepo, they mean a single history line.
    • chvid 8y ago
      It is just the feeling I get when I see how it is used. That it is being misused. If you have a corporate setting with a single Jira/Stash/Bamboo installation keeping all the company's stuff in a single central place. Then this design where git copies (clones) the full repository on the local machine just seems like a misfit and source of lots of pain (ie. just being slow and at times unstable, local repository not pushed back to origin, multiple history lines etc.). It seems to me that things would have been better served with a more primitive client/server orientated tool like Subversion.
    • rkangel 8y ago
      The kernel is not a monorepo. The kernel is a large(ish) repo. A monorepo would be if you put the kernel AND userspace in the same repo (e.g. all the code for a Yocto distro). To me, when people talk about a monorepo they are talking about putting separate pieces of the architecture in a single repo. It's a great example actually. If it was all in a monorepo and you could release it together then you wouldn't have to worry about the breaking userspace, you could make the changes to both sides at once. In practice because that's not how releasing works in that environment, you can't do that.
  • rkangel 8y ago
    Anecdotally - I worked on a ~1 million lines of C project in SVN for a number of years (it was a Bluetooth stack). SVN handled it (mostly) fine.
• EngineerBetter 8y ago
  Visited a customer recently who had inherited a monorepo. All their CI and release problems traced back to it. At the risk of sounding like an old git, package coupling and package cohesion principles were defined for a reason. I do feel like a lot of patterns in contemporary development are kneejerk reactions to how last generation's programmers did things. Exceptions? Nah, multiple returns! Dependency management? Who needs it... Oh, wait. Many small, single-responsibility repos? Wang it all in one, and then invent your own tooling to cope with it!
  • rauhl 8y ago
    > Wang it all in one, and then invent your own tooling to cope with it! Why don’t you believe that additional tooling is required to manage software whose sources are composed from multiple repositories?
    • EngineerBetter 8y ago
      Thing is, that's been solved problem in open source at least since Ivy.
  • fnord123 8y ago
    >Exceptions? Nah, multiple returns! Dependency management? Who needs it... Oh, wait. I thought consensus was that exceptions, like OOP, is an antipattern. I guess there's room for different opinions. :-/
    • EngineerBetter 8y ago
      Consensus could be a synonym here for 'current trends'. I see no advantage of returning error types over checked exceptions. I'm happy to be informed, though.
      • jenscow 8y ago
        You obviously haven't written any win32 applications :) Where a function may return zero, non-zero, not-ERROR_SUCCESS, INVALID_HANDLE, NULL, (there's more), to indicate an error. You then need to call GetLastError which gives you a number that you input into google to get a generic error message. You "should" also wrap every api call in this error checking so you know where the failure began.
• nathan_f77 8y ago
  I'm using a monorepo as a solo developer, and it's been pretty good. I like having everything in one place, so I can work on everything in a branch, including the feature, updates to API clients, documentation, blog post, etc. One problem is that my test suite is very inefficient. I have to run through every integration test, even if I haven't changed any code that might cause these tests to fail. It's especially weird that CI runs all my tests whenever I write a new blog post. So I'm very tempted to split up some things into internal libraries and keep them in a separate repo, and add all these repos as submodules. I know this can be pretty dangerous, and it's easy to break things when you update dependencies, OS versions, language versions, etc. If I go down this road, I have to be extremely careful to enumerate all the things that might break the library, and prevent any of these things from being updated automatically. I'll set a very strict version constraint in the package.json / gemspec, and throw an error if I detect a different version of Node, Python, Ruby, system libraries, etc. Then I'm forced to run all the library tests and explicitly bump the versions if I want to update anything. I should also only do this when the library is a pure function with no side effects. The really tricky part is figuring out how to write robust integration tests. API boundaries can be a big source of bugs. I think I'll do something similar to VCR [1], where the first integration test executes all of the code without any mocks, and then records the response. The response would then include those exact arguments, and it would also be tied to a specific commit hash for the library. If I change anything in the library, then I just need to re-run the slow tests, and then everything will be cached. I guess a real advantage of putting things in a separate library is that you know exactly what files are required for a specific feature, and the commit hash gives you a "fingerprint" of those files that you can use for caching in your tests. Just have to be super careful about any dependencies that might break the library. Also I really need to start running all my tests in a Docker container which matches CI and production. I even have some screenshot tests where I have alternative versions for Mac and Linux. Would be nice to delete those. The experience was really bad when I tried to do this in the past, so I need to figure out a better way. Anyway, sorry for the train of thought! Would be interested to hear your thoughts, and if there's anything else I should watch out for. [1] https://github.com/vcr/vcr https://github.com/vcr/vcr
• ascorbic 8y ago
  One thing I really dislike about monorepos for node modules is that you can't npm install from them directly. Unless it's a project with very fast PR merging and releases you can be stuck with a broken module that has an open or even merged PR to fix it that you can't install because it hasn't been pushed to npm. npm link might work locally, but that doesn't help if it needs to build on a CI server. If it's one repo per module then I can just npm install the git url and it works fine.
• mrbanks 8y ago
  Overly opinionated garbage, imagine having to work with this guy.
• isacikgoz 8y ago
  Just read the article and it was really great read. We're using polyrepos and dealing with so many repos was not great. That's why I created "gitbatch". Gitbatch allows you to manage multiple repositories in an easy way. https://github.com/isacikgoz/gitbatch https://github.com/isacikgoz/gitbatch
• jayd16 8y ago
  There's a lot wrong with this article. Most of the arguments are either not backed up or are misleading. I haven't heard anyone argue they can drop dependency management because of a monorepo. The author lists downsides of monorepos without listing the upsides and downsides of polyrepos so its really half complete. I don't think anyone who likes a monorepo is suggesting you just commit breaking changes to master and ignore downstream teams. What it does do is give the ability to see who those downstream teams (if any) might be. The crux of the author's argument is that added information is harmful because you might use it wrong. Its just as easy (far easier in fact) to ignore your partners without the information a monorepo gives. Its not really an argument at all. There's really nothing here but "there be dragons". Monorepo's provide some cross functional information for a maintenance price. Its up to you whether the benefit is worth the overhead.
  • briantakita 8y ago
    "... Please don't" titles also give off a condescending vibe, which usually means the author has erected strawmen, is appealing to emotion, & has not thought things through.
• harunurhan 8y ago
  I have worked with polyrepo madness... I do remember doing commits to up to 5 different repos just for a feature. And to roll this feature to prod, few of these repos had to go through release process. On top of everything we couldn't really write tests to ensure if the feature works. The best we could do is write tests on the "user facing" repo and keep fixing and releasing others until those pass. Well, I am sure many companies doing better than we did, with poly repos though.
  • twic 8y ago
    Monorepo = all applications in one repo Polyrepo = each application in its own repo Whatever-madness-you-had-repo = each application across multiple repos I'm sorry to hear you had to suffer what you did, but that was not the only alternative to a monorepo!
  • deleted 8y ago
    [deleted]
• sbr464 8y ago
  One glaring omission of the monorepo design, not sure why really, is if you want open and closed source software in the same monorepo, it doesn’t seem possible. Curious as to why this design choice was made.
  • lclarkmichalek 8y ago
    It's entirely possible, and repositories like https://github.com/facebook/fbthrift/ https://github.com/facebook/fbthrift/ are an example of an open source project that is synced commit for commit with a private monorepo. It just requires some tooling (like everything with monorepos)
    • sbr464 8y ago
      Thanks for sharing
• rkangel 8y ago
  To me, the key point is this: Splitting your code into multiple repos draws a permanent architectural boundary, and it's done at the start of a project (when you know the least about the right solution). The upsides and downsides of this are an interesting debate, but there is a cost to polyrepos if you want to change the system architecture. There is a cost to monorepos too as argued by this post, and its up to the tech leads as to which cost is greater.
• rossjudson 8y ago
  Hilariously misguided. Pretty funny to read that the things I do every day are impossible. Monorepo and tight coupling are orthogonal issues. Limits on coupling come from the build system, not from the source repository. Yes, you should assume there is a sophisticated "VFS". What is this "checkout" you speak of? I have no time for that. I am too busy grepping the entire code base, which is apparently not possible. If the "the realities of build/deploy management at scale are largely identical whether using a monorepo or polyrepo", then why on earth would google invest enormous effort constructing an entire ecosystem around a monorepo? Choices: 1) Google is dumb. 2) Mono and poly are not identical.
  • kiallmacinnes 8y ago
    > then why on earth would google invest enormous effort constructing an entire ecosystem around a monorepo? Choices: 1) Google is dumb. 2) Mono and poly are not identical. I think, once you've chosen a path of mono or poly, you have quite a challenge ahead of you to migrate to the other. At that point, the tradeoffs arent based purely on the technical benefits - and "invest in monorepo tooling" may become a perfectly valid decision, as it's cheaper than "migrate to a polyrepo setup'. I'm not arguing either way for or against monorepo, just pointing out that "must be a good idea because Google does it" is invalid - technical merit is just one of the thousands of concerns to be balanced.
    • nindalf 8y ago
      This makes sense to me. If you're considering a monorepo with millions of lines of code used concurrently by thousands of developers, it's absolutely possible. You just need a handful of developers working on the infra to make it happen. I do agree with GP though. I wish the author hadn't decided the things I do everyday are impossible.
      • oblio 8y ago
        > You just need a handful of developers working on the infra to make it happen. With thousands of developers banging on the code base, it's going to be more than "a handful of developers". It's going to be at least a few "handfuls" of developers full time plus probably many, many other full time equivalents spread out throughout the whole user base (testing, supporting other users, etc.).
• m0zg 8y ago
  "Scalability" issues aren't encountered until your repo has many millions of LOCs and a lot of churn. For 99.99% of organizations this is not an issue and will never be an issue.
• username90 8y ago
  With the right tooling for both types each directory in a monorepo is equivalent to a repository in a multirepo setup. The only difference is that in the monorepo it is easier to create new repos and dependencies between repos (just add a directory in a commit or add a dependency on another directory). The author of this piece apparently think that the ease to work in a monorepo is a bad thing, I disagree. I think that being able to treat repositories as easily as directories is awesome since it is a lot simpler so requires a lot less training for your devs to understand.
• jamietanna 8y ago
  It's funny you say this, because my most viewed article from organic searches is about converting your polyrepo setup to a monorepo https://www.jvt.me/posts/2018/06/01/git-subtree-monorepo/ https://www.jvt.me/posts/2018/06/01/git-subtree-monorepo/
• daemin 8y ago
  If you spend your time maintaining and improving a low level library then having a monorepo is much better than having a polyrepo. Primarily because as you make changes to the library you can update everyone else's code that uses it rather than having to wait on them to do so. This reduces the need to maintain older versions of these libraries and applications. Additionally you can submit the single change in one go which updates everyone and it is much cleaner than having to find out and know all of the repos that could use your library and manually submit to each one of them, probably breaking some for a time in the process.
• lmilcin 8y ago
  "Yeah, well, that's just, like, your opinion, man" Monorepos are one way of solving some of the problems each organization has. Monorepos require discipline in solving those problems and if the organization is not willing to get there all the way or if it takes too much time then it's just pain and suffering for everybody. I suspect the author works for one of those organizations that wanted to be hip but did not actually understand what it entails. Maybe faking agile and devops sort of works for you (works as in "it's difficult to pinpoint where the problem is") but faking monorepos certainly does not.
• notacoward 8y ago
  You know what's worse than a monorepo? A duorepo. Yes, that's right, two huge repositories embodying all the problems of a monorepo, but coupled in such a way that it's easy to break something if the commits and deployments from one are out of sync with the other. It's like drinking both bottles of poison, yet it (or minor variations such as three or four entangled ginormorepos) is a thing that really exists.
• z3t4 8y ago
  You can make shallow clones, and auto push to a single repo, it's common to for example auto push to Github from an internal repo. It sure has it's issues, but problems are solved with solutions, simply bury your head in the sand, eg. using single repos where a monorepo is the best solution - is not a solution.
• CJefferson 8y ago
  Is there any examples of someone who actually maintained a monorepo for a massive company, who now says they shouldn't? It always seems to be "back seat drivers" against monorepo, not people with practical experience (that I can see at least)
• cmrdporcupine 8y ago
  It's worth pointing out that while Google has a monorepo in Google3, it also doesn't at the same time. We have are other projects such as Android, ones based on Chrome, etc. that are composed of multiple git projects and use repo to manage and sync.
• NicoJuicy 8y ago
  Since i knew that Google did it, i have started to think about it, a lot. And mono-repos really do make sense ( a lot) when you need them tied together. Finding errors in your console immediatly without version numbers gets the job faster done. There are other ways though, like if you use dot net. A mono repo that creates nuget packages and projects that pull the latest build of them into their solution. This way, external parties can re-use the same components. On a beta version, that releases new nuget components, if there is a file change ( and so a version update), notify the external parties. Have one website which mentions the schedule of an update on the live version to reduce email traffic. Oldskool, but it seems to work.
• rakoo 8y ago
  So the conclusion is "monorepo or polyrepo, you'll need a lot of tooling anyway. So why use monorepos?" Very easy: because having everything in a single place is just easier to work with.
  • diminoten 8y ago
    Easier because you can commit more atrocities. Easier is not better, some things should be hard to do to dissuade you from doing them. Stop burdening your co-workers! People in here keep saying "easy" like it's the end goal, but it's not. Correct is. Writing great software is hard, and monorepos make it even harder to do that because a monorepos encourages an "anything goes" vibe.
    • rakoo 8y ago
      I don't understand how you can say that monorepos and correct cannot coexist. Don't you have the minimum that is code quality analysis, automated testing and mandatory code reviews in place? Those must exist to maximize correctness, and they must exist wheter you have 1 or 100 repos. If I can commit more atrocities I can also commit more fine things, and I have infra in place to stop crap making it to master. Like the article said: it's all about the tooling
      • diminoten 8y ago
        I didn't say cannot, I said that correctness in a monorepo exists despite the monorepo, not because of the monorepo.
        • rakoo 8y ago
          And conversely, polyrepo doesn't bring correctness just because it is poly.
          • diminoten 8y ago
            It does promote it, however, due to the abundance of support, tooling, and how well it integrates with most processes and software available. Being a special snowflake (monorepo) makes it a lot harder to write good software.
• jrochkind1 8y ago
  the grass is always greener.
• thedufer 8y ago
  Maybe its just that the author's cutoff is at the wrong team size, but the monorepo I work on (with ~150 devs) has almost none of the problems presented. Unreasonable for a single dev to have the entire repo? I'm looking at a repo with ~10 million LoC and ~1.4 million commits. I have 74 different branches checked out right now. Hard drives are cheap. Code refactors are impossible? I reviewed two of those this morning. They're essentially a non-event. I'm not sure what to make of the merge issue - does code review have to start over after a merge? That seems like a deep issue in your code review process. The service-oriented point seems like a non-sequitur, unless you're telling me I'm supposed to have a service for, say, my queue implementation or time library. The VCS scalability issue is the only real downside I see here. And it is real, but it also seems worth it. It helps that the big players are paving the way here - Facebook's contributions to the scalability of mercurial has definitely made a difference for us.
  • kiallmacinnes 8y ago
    In theory, yes - if the underlying repo changes, code review should start over. In practice though, it's a terrible idea ;) Part of code review is to ensure the code "fits" with all other merged code - so a re-review is "needed" when other changes merge. E.g. if I merge a refactor that changes everything from Pascal case to 100% SHOUTING, reviews now need to take this into account. In practice, this doesn't happen - it's way too much effort for far too little value.
    • jen20 8y ago
      > if I merge a refactor that changes everything from Pascal case to 100% SHOUTING, reviews now need to take this into account. To be fair, if you get away with merging that refactor, the review that needed more attention was of that refactor ;-)
    • thedufer 8y ago
      I think the trick is to only re-review the areas that had merge conflicts, and to do the re-review aware of both the changes you already reviewed and the changes that caused the conflict. Merge conflicts, even in big code refactors, are fairly rare, so this ends up not being much additional work in practice.
• hvindin 8y ago
  I suspect the problem most people end up trying to solve isn't "how do I technically scale my tools", because the author points out that tools and techniques for this already exist and its an already solved problem. Instead my experience has largely been that the problem to solve is "how do I make some few hundred developers behave in a predictable way" in the scenario where you have many ok developers, but that you can't really be sure that none of them will break stuff because you are trying to solve organisational problems of keeping people with merge rights to only the people who wont break things but at the same time not bottle necking development on to small a number of people then sure, split your repos up so that people can only break stuff that they 'own'. But at least be honest about the fact that most of the technical issues of having a monorepo have been solved already so the issues you are probably trying to solve are actually people problems.
• zbentley 8y ago
  The VCS/codebase-tooling-size argument rings a bit hollow. We have really good code-search tools that are heavily optimized and indexed (from ripgrep/silversearcher to more centralized things like hound, when local-disk performance just won't cutit). It's not hard to optimize Git workflows to be faster with relatively simple tricks, and if that absolutely doesn't scale for some reason and VFS isn't an option, there are always centralized VCS systems like Perforce that solve this. P4 gets a lot of shit, but it's really good at solving the gigantic-repo domain; tune your client properly and you can initial-sync 10+ GB repos in the time it takes to get a cup of coffee (and, if your company is large/old enough to have a repo that big, it can probably afford the Perforce licenses).
• qwerty456127 8y ago
  I didn't actually know somebody already does this, but a conceptual idea has came into my mind yesterday: what if there was just one big code repository for a particular programming language and everything anybody writes would immediately become a part of the standard library? It feels kind of a collective brain...
• icedchai 8y ago
  Monorepos are way simpler for small teams to work with. At my startup we have roughly 10 services out of the same repo. It's much easier to "cut a release" across the entire system. It's much easier to share code internally, upgrade dependencies, etc. For a larger company, it might not be a good idea. However, most startups start small and stay that way. Why take on the overhead you don't need?
• yowlingcat 8y ago
  I think this article is complete horseshit. A monorepo will serve you 99% of the time until you hit a certain level of scale when you get to worry about whether a monorepo or a polyrepo is actually material. Most cases are never going to get there. Before that point, a polyrepo is purely a distraction and makes synchronous deployment really painful. We had to migrate a polyrepo to a monorepo and it was not fun because it was a migration that should have never had to be done in the first place. Articles like this are fundamentally irresponsible.
  • drugme 8y ago
    I found it to be neither horseshit nor irresponsible. A bit overdrawn and skewed in some of its arguments, perhaps. But then again... so was your critique. For example: We had to migrate a polyrepo to a monorepo and it was not fun because it was a migration that should have never had to be done in the first place s/polyrepo/monorepo/ in the above and you have an assertion of about equal plausibility and weight.
    • yowlingcat 8y ago
      No, it is horseshit. 99% of companies will never hit big company VCS scaling issues, and once they do, they're on their own. To characterize that scale as common is one of the most embarrassing failures of modern software engineering. People are so embarrassed to use well worn tooling and accept that large scale is both uncommon and something that doesn't invalidate tried and true patterns for smaller scales. It's utterly baffling to me.
      • notacoward 8y ago
        > 99% of companies will never hit big company VCS scaling issues A much higher percentage of developers will. Number of companies is not a good metric for whether a topic is worthy of discussion.
        • sangnoir 8y ago
          Number of companies is a good metric, because companies own the repos and if it becomes a pain-point, only the developers working at that point in time will be hit by this. Anyone who leaves before this inflection point or joins after it's been solved will not be hit, so I don't think the percentage of developers in that intersection is large.
• nunez 8y ago
  These arguments are weak, IMO. Yes, monorepos can be slow to browse through if the VCS isn’t configured to handle the size (sparse pulls aren’t the default with Git; that alone can make a massive difference when your repo is massive). Polyrepos can be just as slow? however; what’s worse is that there are more of them. I remember working with a repo that was >20GB large, mostly from videos (we didn’t know that initially). Pulling that repo took _forever_. Nobody on that team cared because they almost never did a fresh pull and accounted the time it took for their CI/CD to do so in their reports. If it were a monorepo, MANY teams would’ve felt that pain more immediately. Yes, monorepos require some tooling to prevent a gazillion artifacts from being deployed at once (and to specify what’s related to what if code lives across different folders). So do polyrepos! I’ve configured a few Jenkins jobs for my clients to dynamically pull different co-dependent Git repositories at build time. It’s a pain! Especially when multiple credentials are involved! Then there’s the whole “We have a gazillion repos and 20% of them are junk” problem, which requires automated reaping; also a more difficult problem than it seems. Same with refactors. Refactors across polyrepos are just as much of a pain because you’re now subject to n build and review processes/pull requests, and seeing the entire diff is hard/impossible. This introduces mistakes. If anything, refactors in polyrepos are more of an event than they are for monorepos. While monorepos have their problems, I will continue to advocate for them because the ability to see what’s going on in one place and for any developer to propose changes to any part of the code (theoretically) is massively beneficial, ESPECIALLY for complex business domains like healthcare or financial services. Plus, you will have a RelEng/BuildEng team when your codebase and engineering org gets large enough; why add more complexity by creating a gazillion repos that are possibly related to each other? (The large engineering organization without a team focussed on tools and builds doesn’t exist. If it doesn’t, that means that some/many developers are spending way more time spinning their wheels on build systems than they should be.) The real reason why monorepos don’t happen in the aforementioned domains is because there’s no easy way to allow them and pass regulatory audits. Many regulating bodies require hard boundaries enforced by role-based access control, especially for code that deals with personally-identifiable information or code between two or more domains that have a Chinese Wall between them. “All of my developers can check out the entire codebase” is an easy way to get fined hard, and polyrepos are much easier to restrict access into than folders in a monorepo are (one advantage not mentioned in the article). While you _can_ restrict access into directories within a single repo, doing so is not straightforward, and most organizations would rather not waste the engineering effort. I would like to think that Google and Facebook have gotten away with it because they implemented a monorepo from the very beginning and the engineering involved in splitting it up is much more involved than engineering around it. That said, I continue to advocate for them because discoverability is good and it builds a better engineering culture in the end. I would rather hit those walls and make just-in-time exceptions for them than assume that the walls are there and create a worse development experience without exploring better alternatives.
• hennsen 8y ago
  To sum up the discussion of why this is either absolutely right / absolutely wrong, How about: „mono/poly-repo - none of both is THE single solution for every usecase in every organization and project“? Besides that, sure let’s keep analyzing the pros and cons of each in different scenarios...
• etxm 8y ago
  I feel like a lot of arguments against monorepos assume micro services are the _only_ option on the other side. I tend to break up most of my projects at the edge of business logic or domain logic and lean on a package manager to “deploy together” like any other dependency that’s not in your repo. This allows teams to work independently without a large sprawling repo. If you’re following anything semver-ish hopefully your other teams in the company aren’t breaking releases and you can auto-upgrade on patch level changes. If not, we’ll thank goodness CI is there. I’ve always had difficulty working in projects with too many purposes. This helps me focus where to put things and gives an easy point of escape if a dependency needs to become a service in the future.
• 1024core 8y ago
  Google uses a monorepo: https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... I don't think "scale" gets much bigger than Google.
• costrouc 8y ago
  in my opinion several build systems / package managers have already solved this issue. The answer is that it doesn't matter mono repo vs polyrepo. Look at nixpkgs/nixos/nixpkgs if you are interested
• sloanedavids 8y ago
  Please don’t write this article if you don’t know what you’re talking about
• mr_tristan 8y ago
  Open source software workflows are very common and provide a _lot_ of tooling, e.g., Maven, bundler, npm, etc. Add semantic versioning and you have a lot of tooling that you basically get for free for polyrepo setups. With monorepos, you have to really spend a lot of time tooling, because you basically don't use the OSS tools. There's a lot of odd arguments in this blog that are very spurious: "If an organization wishes to create or easily consume OSS, using a polyrepo is required." What? _Consuming_ OSS is usually not that bad. I've even imported the complete history from external repos, pretty easily. (It does suck with git but I wouldn't use git for a monorepo...) _Contributing_ to OSS is tricky, but the fact you use a polyrepos don't really help you much there either. "Polyrepo code layout offers clear team/project/abstraction/ownership boundaries and encourages developers to think carefully about contracts." Clear ownership boundaries has _zero_ to do with polyrepos. In fact, I'd say monorepos can be easier, since you say "everything under this directory is owned by X,Y,Z". There's no search function that's required to figure out where some other team hid their code. So many times, with polyrepos, projects are _hidden_ because they're off in some other grouping unit that you're not a member of. So you don't even know who owns what or where it came from. In the end, I'd still strongly recommend using polyrepos because you get _a lot_ of tooling for free, and most integration issues are solved with semantic version locking and CD automation. But the arguments here are not really great.
• mcguire 8y ago
  "If an individual clone got too far behind, it took hours to catch up (for a time there was even a practice of shipping hard drives to remote employees with a recent clone to start out with). I bring this up not specifically to make fun of Twitter engineering, but to illustrate how hard this problem is." But mostly to make fun of Twitter engineering. Seriously, what advantages would a big bag of billions of lines of code have?
• obeattie 8y ago
  The real issue is that this is more nuanced than is appropriate for one-size-fits-all advice. "Everyone should use a monorepo" isn't helpful, but neither is "everyone should not use a monorepo." Sadly, this article falls into the same trap.
• manigandham 8y ago
  As always, the real world is a whole lot of gray between the black and white articles that are fun but useless. Multirepos like microservices are all about scaling people, not the project. Start with the monolith and monorepo until you need to split, and then focus on separating by groups of logical functionality or team responsibilities (although if those are different then you'll end up with other problems). Also stop taking things literally. Monorepo does not mean you must have everything in a single repo. Even a startup can put the majority of the codebase in one place and have things like a corporate website or small admin backend in another.
• KuhlMensch 8y ago
  Hm, I find monorepos are a natural in javascript land. There is allot less wiring afforded by a little meta-orchestration. This is especially helpful in the repos I've worked on. But from reading the article, it seems like there are legitimate areas where they might not fit.
• userulluipeste 8y ago
  This is just modularity in the broader information-related development handling. People tend to get political about things that make their involvement comfortable and/or when it's not them who are to deal with the pesky consequences. I strongly suspect that that must have been the cause for how giants like Google¹ or Facebook² ended up with monorepos. It is developers' workout or letting them be with their cake; kick the problem down the road and hope to acquire in time the resources necessary to throw at it later. ¹² "Don't be evil" (with developers, among others) and "move fast and break things" most definitely asks for cutting a few corners here and there.
• astrostl 8y ago
  The word "workflow" is suspiciously absent from the OP. Annoying workflow is my #1 complaint against polyrepos.
• groestl 8y ago
  Me, I dream of a monorepo covering the whole world. Give me a single hash, and let me know the state of things as they are, reaching from the toolchain used to compile the bootloader to the state of the database, which has just dropped a row and therefore generated a new commit, forever secure, an immutable history. I accept the infeasibility of my dream. But I'd like my repo to cover as much as my tooling realistically allows.