19 ms·

Just use a monorepo

263 points by jmduke 4y ago

xiphias2 4y ago
''I am here to tell you: if you are running a software business and you aren't at, like, Google-tier scale, just throw it all in a monorepo.'' Google _is_ a monorepo
- BoorishBears 4y ago
  It really paints a picture of the authors credentials to be making this declaration. All signs for the rest of the world point to the opposite conclusion: unless you're Google-scale, you don't have Google level resources. Google has more engineers working on developer experience than most companies will ever have period. And monorepos work best when workflows are carefully thought out with clever application specific tooling. - I think the author is probably working on a 1-10 developer project (and I'm leaning towards 1) and has confused the convenience with having things in reach when the entire system fits in your mind with the general benefits of a monorepo. I also wonder if they read any of the letters they linked too...
  - dastbe 4y ago
    > All signs for the rest of the world point to the opposite conclusion: unless you're Google-scale, you don't have Google level resources. Google has more engineers working on developer experience than most companies will ever have period. I feel like this was true ~5 years ago, but these days the tooling around scaling monorepos is safely supporting O(100) developers without a lot of overhead. > And monorepos work best when workflows are carefully thought out with clever application specific tooling. I don't see any meaningful distinction with how well thought out workflows need to be between mono and polyrepo.
    - BoorishBears 4y ago
      > I don't see any meaningful distinction with how well thought out workflows need to be between mono and polyrepo. You completely failed to parse the sentence. The point being made isn't "well thought workflows are only for monorepos", that applies to the "needing clever application specific tooling" part. Vendoring/Versioning for discrete packages is a heavily invested in problem space for most tech stacks you'll come across. But if you build a monorepo and don't end up with a build system that takes on those responsibilities you end something that doesn't scale to even moderately large interconnected components. OP has such a tiny project that I'm not convinced they're even dealing with dependencies in a traditional sense. But well before you get to Google scale, you'll run into situations where you just want to change one thing and don't want to change every single downstream dependency which normally would have been isolated via a discrete package that doesn't have to change in lockstep. And then that exact same pain starts to exist for deployments and needs to be worked around. > I feel like this was true ~5 years ago, but these days the tooling around scaling monorepos is safely supporting O(100) developers without a lot of overhead. Nothing about the above has changed in the last 5 years, it's kind of the ground truth of monorepos via multiple repos: You're the first person I've ever seen imply monorepos don't offload complexity to tooling, even amongst proponents.
      - dastbe 4y ago
        > Vendoring/Versioning for discrete packages is a heavily invested in problem space for most tech stacks you'll come across. 100% disagree. The problem of "How do I define my dependencies and have a package manager reify that into a concrete set of versioned dependencies" may be a solved problem, but the tools for tracking dependencies across many repos and driving upgrades is neolithic. About the only company I've seen that does this well is Amazon, and they have yet to sell us version sets as a service. > OP has such a tiny project that I'm not convinced they're even dealing with dependencies in a traditional sense. But well before you get to Google scale, you'll run into situations where you just want to change one thing and don't want to change every single downstream dependency which normally would have been isolated via a discrete package that doesn't have to change in lockstep. And then that exact same pain starts to exist for deployments and needs to be worked around. As I alluded to above, the dual of this is that getting everyone to update their dependencies is orders of magnitude more difficult when you have a polyrepo setup, even if we're working under the ideal situation where repo setup is standardize to such a degree that a person can parachute into a repo and become effective within minutes. > Nothing about the above has changed in the last 5 years, it's kind of the ground truth of monorepos via multiple repos: You're the first person I've ever seen imply monorepos don't offload complexity to tooling, even amongst proponents. Both polyrepo and monorepo have complexity in scaling that is handled by their tooling. The difference historically is that OSS polyrepo tooling has been better and better integrated because that's just how most things are built in any language, but that has been improving over the past ~half decade * Bazel maintenance complexity has dropped precipitously, and many of the initial bottlenecks you hit with it have been solved in OSS. * If you're anti-bazel, gradle and cargo monorepo support is decently good. I believe the same is true in js these days, but I don't have hands-on experience * Services for managing monorepos like sourcegraph for codesearch or mergify for submit queue now exist that make it easy to adopt the patterns that work well at large companies. * Microsoft and others have invested in git to improve scalability of developing against large repos. * you have OSS tools like git-branchless that further improve the experience of working in a monorepo There are a bunch of companies in the O(100) - O(1000) developer range that are using this stuff and it works very well.
      - tylerhou 4y ago
        > But well before you get to Google scale, ... doesn't have to change in lockstep. Not updating dependencies is the equivalent of never brushing your teeth. Yes, you can ship code faster in the short term, but version skew will be a huge pain in the future. A little maintenance every day is preferable to ten root canals in a few years.
- CobrastanJorji 4y ago
  Yes, absolutely. Nonetheless, I think the author may be right, except that by "Google" they mean "large." This is a fundamental misunderstanding of just how large Google and its peers are. I think it's more interesting to consider three sizes. If you're small, everything will fit nicely in a monorepo. If you're large, you'll want lots of repos. There aren't really any off the shelf monorepo options that scale super well, so using a bunch of small repos is a great way to deal with the problem. Plus, you probably don't have a full time staff babysitting the source repos, so you want some isolation. If someone in another org is breaking stuff left and right, you don't want the other orgs to be affected. If you're GIGANTIC, monorepos are a pretty great option again. You'll probably have to build your own and then have a full time group of people maintain it, but that's not a huge problem for you because you're a gigantic tech company. You can set up an elaborate build system that takes advantage of the fact that the entire system is versioned together, which can let you almost completely eliminate version dependency hell. You can customize all of your tools to understand the rules for your new system. It's a huge undertaking, but it pays off because you've got a hundred thousand software engineers.
  - jeffbee 4y ago
    > There aren't really any off the shelf monorepo options that scale super well How can you say this when Perforce on a single machine took Google to absolutely terrifying scale? There is no chance that your mid-sized software company will even slightly tax the abilities of Perforce. What I believe you meant was there aren't really any good options to make git tolerable for non-trivial projects, and with that I wholeheartedly agree. And that's why these threads are so tiresome: they always boil down to people talking about what git can and cannot do.
    - sanderjd 4y ago
      The Linux kernel seems pretty non trivial...
      - jeffbee 4y ago
        Google3 is no joke 1000x larger than Linux.
- thundergolfer 4y ago
  It's awkwardly phrased, yes, but what the author is saying is that Google-tier scale companies will have an awful time migrating from poly to mono-repo, and a not fun time being mono-repo for the first few years. For company's not at this tier, it isn't that hard to migrate to a monorepo and the benefits will be more immediate because the tools (eg. git) won't be screaming under the load. (My personal 2c is that you can be well below Google-scale and still hit the limits of the common tooling when using monorepos. Canva, Stripe, and Twitter are examples)
harryvederci 4y ago
I use a megarepo, if that's a thing. One repo with multiple unrelated apps, plus a "shared" directory. I could put the shared dir in a package, but so far I don't really see the need.
- jvolkman 4y ago
  Sounds like a monorepo.
  - harryvederci 4y ago
    Maybe I should have put emphasis on the word "unrelated".
throwaway_au_1 4y ago
I would like for any article insisting I adopt any practice to put a little more effort into doing so. This seems to amount to "it's good", "I should have done it sooner", "it's not so bad, trust me", and "smart people like it, too". Could have been a tweet.
- yodsanklai 4y ago
  Agree, not a good article. Hopefully it'll trigger interesting discussions.
arrakeen 4y ago
this is the first i've heard of "microrepositories" but it appears like the author may have just been overzealous with partitioning their repositories and has now settled on what most of us would just call a "repo".
yodsanklai 4y ago
My understanding is that Meta/Google had to rewrite tons of stuff (distributed file systems, CVS, search tools, build tools...) and need many teams to maintain all these systems, and sometime their developer experience is inferior to what you get with off-the-shelf OSS. I'm not super familiar with this topic as I don't work with systems of that scale, but I'm wondering was it worth it or necessary? what would be the alternative?
- hahaxdxd123 4y ago
  > sometime their developer experience is inferior to what you get with off-the-shelf OSS. The internal version control system that I use (Fig, which is apparently based off Mercurial) is so good I've basically never had to think while using it. I can't really say the same about git. CodeSearch is also, AFAIK, a lot better than what you would get OSS. I think the productivity gains across 100K engineers is definitely worth it.
- Shish2k 4y ago
  > sometime their developer experience is inferior to what you get with off-the-shelf OSS Working at Meta, being able to use Sapling (our Mercurial fork) is actually one of the highlights and I was pretty miserable every time I needed to go back to Git for my open-source work — thankfully it was open-sourced with a git backend a couple of weeks ago, so now I get to combine the excellent sapling CLI with the github hosting service :D
- skuzye 4y ago
  I have to say, I really enjoy using Google monorepo and all tools available to me. Having said that, it's not free of course. There are many engineering teams and computer power being invested to provide this environment to all googlers.
- twblalock 4y ago
  I've seen what happens when an org tried to adopt a Google-style monorepo without making the investments in build tooling and cultural change. It was a disaster. All of those things are needed even at orgs much smaller than Google, or you will end up with an unbuildable, unmaintainable, unreleasable mess. For orgs that can't make those investments, I think a repo per team is the best approach. Each team can treat their repo like their own little Google-style monorepo if they want to.
015a 4y ago
IMO the biggest drawback to a monolith, maybe beyond those listed, is losing the 1-1 mapping of changes to CI to releases. If you know "this thing is broken", the commit log is a fantastic resource to figure out what changed which may have broke it. You submit a PR, and CI has to run; getting most CI platforms cleanly configured to, say, "only run the user-service tests if only the user-service changes" isn't straightforward. I understand there are some tools (Bazel?) which can do it, but the vast, vast majority of systems won't near-out-of-the-box, and will require messy pipelines and shell scripts to get rolling. There's also challenges with local dev tooling. Many VSCode language extensions, for example, won't operate at their best if the "language signaler" file (e.g. package.json for JS) isn't in the root of the repo; from just refusing to function, to intellisensing code in another project into this one, all manner of oddities. Meanwhile; I don't think the purported advantages are all that interesting. Being able to write the blog post and make the change in the same PR? I've never been in a role where we'd actually want that; we want the change running silently in prod, publish the blog post, flip a feature flag or A/B test proportions. Even an argument like "you can add schemas and the API changes in one go"; you can do that without a "monorepo", just co-locate your schemas with the API itself, this isn't controversial or weird, this is how everywhere I've worked operates. None of that is to even approach the scale challenges that come with monorepos at even reasonable sizes; which you can hit pretty quickly if you're, say, vendoring dependencies. Trying to debug something while the Github or Bitbucket UI is falling over because the repo is too large, or the file list is too long, isn't fun. I'm not going to assert this is a hill I would die on, but I'm pretty staunchly in the "one service, for one deployment artifact, for one CI pipeline, for one repo" camp.
- dogleash 4y ago
  Your examples of CI and VSCode are on point, and in the bigger picture it's always about tooling. The mono/multi repo argument is fundamentally boring to me because it always boils down to whether the shape of the tooling problem is easier to work with on this or that side of the divide. The answer is always whichever tradeoffs work best for your situation, and the reason at the end of your post is as good of a reason as as any.
- xmodem 4y ago
mr90210 4y ago
Just use what works well for you and your team, how about that?
fullstackchris 4y ago
My 2 cents: monorepos try to solve what is actually a series of devops problems, by essentially abusing the way an organization or individual uses git (or VCS of your choice). These tools that were never designed for devops in the first place. To elaborate: I think you are much better off, whether as a team or solo developer, developing small, isolated, and tested / reliable libraries and / or packages that can be combined as needed. Ideally, each package or library can build, test, and ship itself at the CI/CD level. Trying to encapsulate all that into one giant conglomerate of potentially different languages, file types, frameworks and much more sounds exactly what it is: a mess!
- gorgoiler 4y ago
  Developing components like this puts a lot of pressure on the components to each look like a piece of Real Software. Versioning, stability, changelogs, coordinating new features as beta releases followed by deprecation announcements and finally deprecation. None of this is unreasonable, technically — my Linux OS is made of many small stable projects after all, none of whom communicate with each other beyond broadcasting NEWS.md etc with each release. It’s also exhausting. Do your competitors hamper themselves by cosplaying a bazaar of open source projects coordinated by whatever your internal equivalent of the Debian project is? No! Move everything under one roof to truly reflect your org structure and take advantage of the fact that you all report to one CTO, you all have real-time communication between each other, and you probably know where everyone else is sat for eight hours a day. Move fast.
  - ghc 4y ago
    > take advantage of the fact that you all report to one CTO How many companies even have a CTO that knows or cares about version control? In most companies version control is managed by IT if at all, and code for different products is split across different lines of business.
- hgsgm 4y ago
  Huh? Monorepo refers to storing your source code, regardle5 of how many binaries you build and run, not shipping your system as one giant monolithic binary.
jph 4y ago
Monorepo vs. Polyrepo guidance thanks to many contributors: https://github.com/joelparkerhenderson/monorepo-vs-polyrepo https://github.com/joelparkerhenderson/monorepo-vs-polyrepo
Yahivin 4y ago
Monorepos are great! They're so good one place I worked at had dozens of them.
- __alias 4y ago
  Underrated comment
siliconc0w 4y ago
Probably disagree - Google has built a boat-load of infrastructure to enable a mono-repo which you don't have and (I don't think) aren't even available either OSS or commercial. Use modules/libraries as an alternative to separate repos. This may require a few more commits for distributed changes but that isn't hard and forces you to consider realities like, "what if A service is deployed before B service?" which happen whether or not you're using a mono-repo. Also use data-lakes for data or generated artifacts (something based on S3 or an S3-alike).
- hgsgm 4y ago
  Google used Perforce for many years until they outgrew it. What size org outgrows Perforce but can't develop and maintain their own thing?
  - hocuspocus 4y ago
    It's just one piece of the puzzle though. Have you noticed how companies running Bazel usually have a dedicated team? Many orgs are absolutely not interested in that.
    - jeffbee 4y ago
      No, i haven't noticed that. My team just uses bazel. What necessitates a dedicated bazel team?
- atomicfiredoll 4y ago
  my understanding is Nrwl's Nx is built by ex-Google employees, that their integrated repo style is based on the approach used at Google, and that it's open source (they make money with their paid build caching solution.) Most companies aren't Google. So, the fact that they can't run the exact same tools doesn't seem like a real blocker when alternatives exist. I've used Nx a bit before and intend to for my next project. Even as a solo dev initially, I'm optimistic it will help me keep things organized and catch potential issues with shared code via the dependency graph.
IshKebab 4y ago
> You cannot apply role-based access controls to certain parts of the repository for security/compliance reasons. You can (at least for write access). Maybe he's talking about read access.
- ridiculous_fish 4y ago
  This is a major sticking point for us. Many of our repos are 3rd party, made available to us under different licenses. Some require individuals to be licensed, and so we use ACLs for both read and write access. Some of our source is export-controlled and this also is enforced via ACLs at the repo level.
  - IshKebab 4y ago
    They require a license for read access? What are they? I don't think I've ever seen anything like that.
jgavris 4y ago
Even in a private monorepo, Google has tools like Copybara that make it somewhat easy to 'open source' small pieces of it and sync them in and out of a 'public' repo. The benefits of a monorepo and just having to 'git checkout' and your entire environment 'just works' (I'm looking at you, scripting languages and build systems) is not to be taken lightly.
gorgoiler 4y ago
Splitting your business’s mission across hard repository boundaries implies that… you know what you are doing! If that’s you then congratulations. Also: you’re kidding yourself. For the rest of us, being virtually unable to rethink the structure of our components because they are ossified as named repositories is a technical and social disaster. Whole teams that should have faded and been reabsorbed elsewhere will live forever because the effort to dismantle and re-absorb their code into other components is astronomical. The value we bring as engineers is in making sequences of small changes that keep us moving towards our business goal. Boundaries that get in the way are anathema to good engineering. Its exactly as if you were unable to move code between top level directories of your project. Ridiculous.
- mansoon 4y ago
  But, but, my modularity microservices as my self shibboleth that we are going to google scale next week right after I figure out what our product is!
pancakemouse 4y ago
Anecdotally, as a developer I really appreciate a well-thought-out monorepo. All the code is there. You don't have to dig around anything legacy or outdated. Searching is a breeze and it's _fast_. Updating both simple and complex logic is equally easy. You don't need to somehow discover that some other repo uses the thing you changed. Everything has the same deployment process. The tests test the thing, as soon as you push it - instead of months down the line, when someone finds forgotten repo again and oh, it's never been tested with the new functionality in our main repos. Dependency hell doesn't really exist. There seems to be more discipline around monorepos because they're collectively owned, whereas many repos across an org each become someone's baby, and practices diverge until teams in the _same company_ fork each others' repos because they don't get along.
shagie 4y ago
For me, this boils down to a question of "what set of problems do you want to solve?" and "what set of problems do you want the system to solve for you?" To this end, I am generally a monorepo proponent. The problems with "what gets built" and how do you do versioning of the components" for the systems that I work on is an easier set of problems to solve than "how do I ensure that I don't break anything else when I update this method?" and "if I do break something when I update the method, where is everything that needs to change?" I've had projects that were fractured across half a dozen repos where libraries were versioned and tagged and and deployed, and then the next down stream was updated to use the newly tagged version... and something broke. This was especially bad when the "something broke" was part of another team's code or it had been thrown over a wall. A bunch of changes, many PRs - some of which are just updating a version to pull from the artifact repo. When things were updated, I took the opportunity to move all of those projects into one monorepo and I only had one PR to review - that passed all the tests for all of the projects contained within the monorepo. Yes, deployments and tagging and the "well, just because you reved X to 2.3.0 doesn't mean that Y is at 2.3.0 too" (strict semver can get a bit on the messy side). It's a question of which problems do you want to solve. I tend to find that me solving the "how do I set up the ci file and coordinate version numbering" is an easier problem to solve than "all these builds need to be updated in a bunch of different repos and one of them breaks."
Octabrain 4y ago
I have my doubts about the mono repo approach. From the top of my head: 1. It might increase the complexity of other process on that repo: CI/CD configuration, makefiles, branching strategies, codeowners etc 2. The versioning might lose its meaning 3. Blast radius in case of screwing things up accidentally. I personally split repositories based on responsabilities: here the code base, here the iac, here the configuration, here the manifests. Always using a standard and predefined naming convention for the repository names. That being said, as always, it depends. I might embrace the monorepo if the context demands it and it has been properly discussed and evaluated.
- cosmotic 4y ago
  1. Often it ends up being easier 2. Versioning becomes easier 3. Blast radius is similar but cleanup is easier
  - TylerE 4y ago
    More importantly re 3. things break sooner. This is good. The sooner the breakage is noticed the less merging there is to do…
  - Octabrain 4y ago
    > 1. Often it ends up being easier It hasn't been like that most of the time in my experience. Having to integrate in the same CI/CD pipeline, let's say, from the code base side, linters, tests and build, from the iac side, formatting, validation, plan, from the manifests side, yaml linters, builds (e.g kustomize), dry runs etc. Now, think on all the previous stuff to consider but also adding the logic for something common like "based on the branch, run this and not that over this environment". Of course you could sepparate the previous in different pipelines, but then you might have to control somehow the order of execution of each. It is possible, however, it might be complex. Also, one must consider that sometimes, reaching a consensus between devops and developers can be tricky and you might end up in people stepping in each other toes. 2. Versioning becomes easier How so? From my point of view, if I find a repository containing an specific thing (e.g a Terraform module) and I see the release version "1.0.1" I can get a clear idea of what that it implies. However, if that module is versioned along the terraform files themselves as well as the code base what was fixed on that "1.0.1" version, the Terraform module?, something in the app itself? Of course, I can go to the release notes and spend some time reading them, but you better be following some good practices on that regard, otherwise is gonna be time consuming. > 3. Blast radius is similar but cleanup is easier I could buy this one, however, it is preferable to assume the risk because of how easy is to clean up things in case they go south or is better to avoid the risk in the first place by having everything sepparated on the first place? I guess context (complexity of the project, culture, communication across teams) is the key.
karmakaze 4y ago
I was curious why it was a big deal either way for a smaller codebase/documents to use a monorepo or not. It's mentioned: > You may be reading this short essay and thinking: "Yeah, no shit! Of course you do not need a monorepo when you're talking about, like, 150K lines of code." > Maybe it's some abstract sense of purity; maybe it's because you didn't realize most IaaS let you deploy from a subfolder. Whatever the reason is: trust me. Move it into a monorepo. If you do and you end up regretting it, [email me] and I'll donate to the charity of your choice. So the new thing being said here is that a monorepo makes sense even at smaller scales. This seems to on the far other end of the scale with a main app, supporting infra, docs, marketing, etc. I've certainly used fullstack repos with backend and frontend together and it's fantastic to use and deploy from commits. I can't infer from this article or my experience what it would be like to work with a monorepo at medium scale say 10-100 apps with 10s of teams. That would take more adaptation than simply "deploy from folder". Dependencies as mentioned in another comment and being able to detect if each target should be retested/deployed seems like the tough one.
- TylerE 4y ago
  100 apps across 10 teams doesn’t sound very medium scale to me.
  - karmakaze 4y ago
    I said 10-100 apps, and should maybe have said 5-10 teams rather than 10 teams, but 10 dev teams in a company to me is medium-sized. Or do you mean it's less than medium-sized?
scarmig 4y ago
I always find monorepo/polyrepo discussions tiresome, mostly because people take the limitations of git and existing OSS tooling as a given and project those failings onto whichever paradigm they are arguing against. I'm pretty excited for new OSS source control tools that would hopefully help us move past this discussion. Particularly, Meta's Sapling[0] seems like a pretty exciting step forward, though they've only released a client so far. (MS released its VFS for Git awhile back, but unfortunately now is deprecated.) [0] https://engineering.fb.com/2022/11/15/open-source/sapling-source-control-scalable/ https://engineering.fb.com/2022/11/15/open-source/sapling-so...
- ricardobeat 4y ago
  Sapling is specifically built to deal with issues arising from gigantic monorepos. Why and how do you think that will move us past this discussion?
  - Spivak 4y ago
    Either one of monorepo minus the major downsides or polyrepo minus the major downsides effectively makes the discussion of the trade offs moot and the first one to happen will likely “win” with not enough gained by deviating from the norm to switch once people adopt it.
  - deleted 4y ago
    [deleted]
- mkl 4y ago
  Sapling discussion a couple of months ago: https://news.ycombinator.com/item?id=33612410 https://news.ycombinator.com/item?id=33612410
- xixixao 4y ago
  https://graphite.dev https://graphite.dev is also aiming to provide a FB-like workflow including stacks.
- nixpulvis 4y ago
  Can't we just generalize the package manager already and push it into VCS? I just want to commit some code and roll it out in the next release. Somewhere in the tree is a top level makefile or something. Stop making this complicated. When it comes to scale, force versions to be incremented at the same time across the lot. You can even spin up a new deploy set and gracefully handoff load. My point is. Just let the source code, in whatever language, be distributed as packages in as modular a way as developers want. One way or another you're going to end up with a makefile or shell script that builds the damn thing. If you don't, then someone fucked up and your build is effectively broken. Monorepo or not.
erksa 4y ago
Earlier in my career I was so excited to learn about microservices, as I was working with a huge monolith at the time. Then I went a bit overboard, and for a new prototype I had all these services in different repos and running independently. It was all very cool and I felt very accomplished, only to later (and now painfully obvious) be pointed out it was completely unnecessary, to the point each function basically was its own deployment. But not in a way that made any sense. If you were looking for the _most_ expensive way of running things, maybe.. hah Tunnel-vision is a powerful thing, and if this is how people are splitting repos now a days, it seems I might not have been the only one chopping up perfectly good dish only to be left with the individual ingredients.
- scarmig 4y ago
  You can have microservices and different repos, microservices and a monorepo, or a monolith and monorepo. In principle you could even have a monolith split across different repos (e.g. each library in a separate repo). What's included in a deployed binary has nothing to do with how the source code is structured.
- lloydatkinson 4y ago
  The post is about monorepos not microservices.
  - erksa 4y ago
    Yes, that's not lost on me, however my point might have been. The point was, when learning this by yourself, you can really go down the wrong path with the right advice at the wrong time. This seems to hold true for the monorepo discussion as it did/does for microservices.
SpeedilyDamage 4y ago
No! I'm tired of being told how to architect systems because other folks refuse to grow their knowledge. Because I'm good at my job, a monorepo would be pants-on-head stupid. Because I know how to make separate codebases interact with one another successfully, it would be a massive mistake to lose those advantages, all because you (not me) run into problems when you try to do what I find perfectly natural. And if you work with me, you will learn, and I will guide you, and in time you will experience systems design as I do, and no longer be shackled to only doing the things you understand today. But no, "do this because I don't know how to do that" will never be a winning argument.
- lliamander 4y ago
  I am myself am skeptical of monorepos, and i have only worked with polyrepos, but I am curious what strategies you use that help make you effective with multiple code bases?
  - SpeedilyDamage 4y ago
    Keeping the spirit of microservices alive is super duper important, so not entwining two services with shared code or responsibilities is key, and forcing interaction to happen through their defined APIs.
    - lliamander 4y ago
      Yeah, that's pretty much what I do as well. Shared libraries get their own repo as well and are published in a package manager. Honestly, in terms of enabling concurrent development across multiple teams while simply using common OSS software, polyrepo seems a lot easier.
- sanderjd 4y ago
  I know how to do that stuff, it just sucks more than having a monorepo with good tooling, in my opinion.
  - 4y ago
osigurdson 4y ago
I call bike shed on the entire debate. One repo or many can work fine depending on the situation.
- kmbfjr 4y ago
  The whole debate gives work to people who don’t want to work on the actual product or service.
  - deleted 4y ago
    [deleted]
raydiatian 4y ago
Here’s why I’ll never go back to anything but a monorepo. My previous org decided on this for their source control/build dependency layout: 1. Monolith, home to front and back end, artisanal handcrafted openapi spec that plugged into nothing and was just for show. 2. Hand-painted service-cli library, which was supposedly going to be open source for customers but never was, and only mattered to the front end, back in the monolith. We manually kept it in sync with the backend, to everybody’s benefit (YFR). Hope you see it: backend changes left the monolith into the service-cli, back to the monolith. Face, meet wall. 3. A knapsack full of tiny libraries which all could have been wrapped up in one. They were all dependencies that plugged into the monolith and the service-cli. And they all had subdependencies. 4. Everything version-wise was synced via private npm registry. So if you wanted to make a change that went into the next version of monolith, but the code had to live in a leaf dependency way down at the bottom, you had to run a relay race marathon of red-tape PRs that you had to bribe your coworkers to pay attention to and/or approve. The entire time, several of us on the team were begging the core team to repair this and they basically told us “no we believe in the tenets of sadism, now take your allotted pain, feature-bitch.” This company employed 150+.
bluGill 4y ago
Polyrepo and monorepo both work well if you have the right tools and workflows in place. Both have substantial downsides. If you can't admit the above then you are either ignorant or trying to sell something. Either way I don't want to talk to you. The rest of us can now discuss pros and cons.
- sanderjd 4y ago
  Yeah agreed. But it's also ok to other one over the other on the full balance. But that also doesn't mean that either preference is the one true way.
ridiculous_fish 4y ago
How do monorepos work with something like gitlab or github, which expect a CI file in the repo root? My workplace has dozens of repos, and they all have separate CI needs and test suites. Trying to shoehorn them all into a single set of CI pipelines would be very difficult, even with the include and parent-child pipeline features.
- shagie 4y ago
  The CI file can specify what triggers different workflows. https://docs.github.com/en/actions/using-workflows/triggering-a-workflow#accessing-and-using-event-properties https://docs.github.com/en/actions/using-workflows/triggerin... on: pull_request: types: - opened paths: - '.github/workflows/**' - '.github/CODEOWNERS' - 'package*.json' That triggers when those files change. The "get them all to run in a single repo" is the challenge of the monorepo. The trade off is "if you change a common library, how do you trigger the test suites in all of the downstream repos to run with the new common library and verify that they still work?" With a monorepo, that second part is solved for you... at the cost of needing to solve the problem that you're asking about.
- atomicfiredoll 4y ago
  I have the most expense with Nx and can only speak to that. Nx projects are tied together with a dependency graph. The workflow/pipeline replaces something like "npm build" with with "nx affected --target=build --base=main". Nx compares the code to the base branch, identifies the leaves of the tree (projects) affected by the code change, and runs the specified task for each project. Each project can specify what that task (technically the "target") means in it's own context by directing it to run a specific "executor." For some projects, "build" may run a Weback executor, others may map it to run a Vite executor. Edit: To summarize, if you change something deep down in the tree like a design token, you can run the "test" and "build" tasks for everything affected by that change to make sure none of your apps break. If those projects are SPAs built in different frameworks, each can map the task to run the framework specific "test" or "build" executor. Now you have automatic assurance that all your apps still work. If that sounds like a lot for the CI to do, that's where distributed caching the of task results come in.
ohCh6zos 4y ago
I have worked at places that have micro-repos, places that have polyrepos, and places that have monorepos. From my observation monorepos are a sign of ossification and resistantce to new patterns and new technologies.
jsnell 4y ago
> Rolling out API changes concomitantly with downstream changes to the documentation or the OpenAPI spec. > Introducing feature-level changes and the blog post announcing those changes. These are horrible reasons to use a monorepo. Commits are not units of deployment. Even if you're pushing every system to prod on every commit, you'd still basically always want to make the changes incrementally, system by system, and with a sensibly sequenced rollout plan rather. To take one of the examples above, why would you ever have the code implementing a feature and an annoucement blog post in the same commit? The feature might not work correctly. You'd want to be able to test it in a staging environment first, right? Or if you don't have staging, be able to run it in prod behind a feature flag gated to only test users, or as a dark launch, or something to verify that the feature is working before letting real users at it and having it crash your systems, cause data corruption, or some other critical problem that would necessitate a rollback. But none of this pre-testing is possible if the code changes are really being done in the same commit as the public announcment. And talking of rolling back... When you revert the code changes that are misbehaving, what are you doing with the blog post? Unpublish it? Or do some kind of a dirty partial rollback that just reverts the code and leaves the blog post in place? The same goes for any kind of cross-project change[0], some of which appear more compelling on the surface than the "code and blog post in one" use case (e.g. refactoring an API by changing the interface and callers at the same time). Monorepos allow for making such changes atomically, but you'd quickly find out that it's a bad idea. There are great reasons to use monorepos, but this is not it. [0] I wrote more about this a couple of years back. https://www.snellman.net/blog/archive/2021-07-21-monorepo-atomic/ https://www.snellman.net/blog/archive/2021-07-21-monorepo-at...
- sa46 4y ago
  > you'd still basically always want to make the changes incrementally, system by system, and with a sensibly sequenced rollout plan rather. Depends. It's significantly faster to deploy everything at the same time and accept that unlucky requests might end up in a weird state than to safely sequence changes. In SRE phrasing, I'm choosing to spend our error budget to maximize change velocity by giving up on compatibility during deploys by skipping a multi-stage rollout plan. In return, I can condense a rollout to a single commit and deploy. A 99.9% availability target yields up to 86 seconds per day to pretend that deploys are "atomic".
voz_ 4y ago
Monorepos for life, don't conflate code storage and organization with deployment lifecycles tho.
snow_mac 4y ago
Just don't.
Vekz 4y ago
Anyone have resources on how to actually perform the migration of many repos to monorepo? Currently facing the same issue and want to make the consolidation move but now sure on how to merge the git history. This article and the previous one it links to are missing all the implementation details.
- phphphphp 4y ago
  Don’t, keep things simple. Maintain a separate archive of each individual repository and any history can be referenced in that repository when necessary.
- j1elo 4y ago
  I just finished a migration like that, and after some research ended up writing shell code to implement 3 methods and put them to the test. Maybe it is useful for you so I'll leave it here: https://gist.github.com/j1elo/6156437f26bfc580769613bb30419af8 https://gist.github.com/j1elo/6156437f26bfc580769613bb30419a... My first step was to learn the procedure that devs behind GStreamer used recently, when they went ahead and decided to merge all their disparate repos into a single monorepo. For my needs it is the less adequate of the three because it will break as soon as there is any collision between the desired destination and the already existing files of a repo. The second method is one proposed in the JDriven blog. It doesn't suffer collisions because the movement into destination subdir of each repo is done separately on the original repos, and then they are merged with everything already in place. Lastly, I found a third method which is the one I liked the most. It uses git subtrees, to perform the move and merge all in the target repo (the monorepo itself), without touching the original repos (unlike the JDriven proposal), so everything is tidy and left in place. Method 1 (GStreamer): * https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/474 https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/... * https://blogs.gnome.org/tsaunier/2021/09/29/gstreamer-one-repository-to-rule-them-all/ https://blogs.gnome.org/tsaunier/2021/09/29/gstreamer-one-re... * https://gitlab.freedesktop.org/thiblahute/gst-merger/blob/master/merge.py https://gitlab.freedesktop.org/thiblahute/gst-merger/blob/ma... Method 2 (JDriven): * https://blog.jdriven.com/2021/04/how-to-merge-multiple-git-repositories/ https://blog.jdriven.com/2021/04/how-to-merge-multiple-git-r... Method 3 (Subtrees): * https://mirrors.edge.kernel.org/pub/software/scm/git/docs/howto/using-merge-subtree.html https://mirrors.edge.kernel.org/pub/software/scm/git/docs/ho... * https://docs.github.com/en/get-started/using-git/about-git-subtree-merges https://docs.github.com/en/get-started/using-git/about-git-s...
ankitdce 4y ago
For everyone who complains about monorepos, remember some of the most forward thinking engineering companies like Google and FB also use monorepos. All the arguments that people make in favor of polyrepo are making so because of lack of strong tooling for monorepos. That's also why Google and FB would not have scaled if they were using GitHub / GitLab but had to build their own. Also Google's original source control was built on top of perforce!
- fargo 4y ago
  They way at least FB is using a monorepo is very different than any kind of monorepo most people imagine. It's not just about tooling, git itself could never handle it. I am all about using a monorepo but Google and FB having one is not the an argument for it.
  - phyrex 4y ago
    how so?
- threeseed 4y ago
  a) There is no correlation between mono/poly repos and your ability to scale. There are many examples of successful companies using either approach. b) As a general rule people should be cautious about adopting approaches and technologies from Google, Meta without a clear understanding of why they need it. What works at their scale doesn't always apply to smaller teams.
- afavour 4y ago
  > remember some of the most forward thinking engineering companies like Google and FB also use monorepos As a counterpoint though I’d say that the issues Google and FB face, particularly in terms of the sheer scale of the work they’re doing, is pretty unique. Google literally invents programming languages for domains it feels needs them, I’m not about to blindly follow that practise either.
- scarmig 4y ago
  The issue with building a custom monorepo system that can handle Google's and Facebook's scale is that it fails to scale down, even to moderately large project and organizational size. It's expensive (think at least 7 figures opex) and not what most people should be doing. git, for all its issues (and I'm a git-hater), scales down to an individual coder and scales up (with a lot of hacks, the hacks being used varying depending on whether you're taking a poly or mono approach) to companies that employ thousands of developers.
jshen 4y ago
Does anyone have a pointer to a tutorial or example of what a successful monorepo and process looks like?
0xbadcafebee 4y ago
Do not listen to pithy blog posts. Seriously. There's too many factors to consider that aren't in the blog post. They get upvoted on HN here quickly, and most often when they are written from ignorance, which seems super confident, and leaves out all the detail and nuance that could make such an upvote questionable. I'm pretty sure the upvotes are based on an emotional reaction to the title alone. (The "reasonable drawbacks" part of this post should have been like 5 pages; and the same for multirepo) If you really want to know whether you should use a monorepo, first go buy a book about them. Then talk to senior leaders in organizations that have gone through an entire growth cycle and migrated from multirepo to monorepo or vice versa. The random dude on the random blog post doesn't have the perspective to inform you properly about the implications. Hell, I've done the switch twice, and I still wouldn't call myself an expert.
nonethewiser 4y ago
And trpc. If you're using typescript.
jalapenos 4y ago
I'd like to add my own PSA in concurrence with this. Just use a monorepo. Use tooling to work around its limitations if you reach that point. I work in a SaaS that's polyrepo based, having split from its original monorepo as part of a microservices push (which never succeeded, leaving us stuck half way in the worst of both worlds). Nothing has been more destructive to productivity than the polyrepos and their consequences. We're talking a 20% engineering spend dead weight loss. This is stark obvious to every single engineer, but trying to get people to accept that fact and sign off on a project to coalesce them back into a monorepo is just insurmountable. Polyrepos are irreversible damage, stay away from them. Hold the line on your monorepo. One organisation, one repo.
- eclipticplane 4y ago
  Either I work with you, or the _failed microservice push leaving us in the worst of all worlds_ is unfortunately common.
- quickthrower2 4y ago
  As a special case: polyrepos are fine for when you have a genuine plugin architecture. To be a genuine one: you have a public facing end to end documentation on how to use it. I would use chrome extensions vs. chrome itself as an example. Or same for VSCode. Then vendor plugins can be in their own repos. I think otherwise for simple module A and module B that interact privately, in an adhoc way, the separation of concerns is not there and monorepo is better.
- lloydatkinson 4y ago
  Absolutely this. I am experiencing a polyrepo setup of 254 repositories. The level of productivity loss is at a scale I don't think can be adequately describe in words.
wankle 4y ago
"Maybe it's some abstract sense of purity" Someone gets a few lines of code under their belt and now they overwhelmingly know what is best for all. It's simple, if it gets released together, put it in the same repo (even if it's a big repo like the Linux kernel). If not, separate it out so you can stay sane. If it's not released together then it's a monorepo, also defined as a big ball of spaghetti.
yellowapple 4y ago
Another big disadvantage of monorepos is the size. If you're having to clone the repo in its entirety on a routine basis (think CI/CD pipelines, or there's some other repo that needs to maintain multiple work trees referencing different commits), that repo being some massive monstrosity can be a real pain in the ass.
hansvm 4y ago
You have a monorepo anyway, and the choice is between dealing with it via wrappers automated or manual wrappers around tools like git (e.g., manually cloning relevant repositories and constructing a build environment referencing all the right versions appropriately) or whether you'll use dedicated monorepo tooling. That's still maybe an interesting question, but when appropriately framed it highlights that the real issue is that monorepo tooling may or may not fit your use case well, and you might be better served by hacking together more primitive solutions or making your own tooling. IMO, the major point of the article still stands; git and whatnot are pretty good, and the build/deploy tools interacting with your code are probably terrible, so absent major outside pressures you'll go a long ways figuring out how to leverage git (in a monorepo configuration) against jenkinbuckethubernetes, and that's often much less painful than inevitably figuring out how to marry two ancestrally disjoint codebases.
humanrebar 4y ago
Most opinions about monorepos conflate monocultures in source control with monocultures in build systems or workflows. Those two considerations are technically orthogonal. You can have one big repo with N build systems (like a node app and a python app in the same repo with disjoint workflows). And you can have N repos that have a common build system (each can combine into one big build with cloning and/or submoduling). I find most of these conversations are more interesting when people decide to make that distinction.
berjin 4y ago
I'm going to start a new term: goldilocks repo. Too small and you end up needing to deploy 5 apps to get your feature out and tests are difficult to coordinate. Too large and the huge CI suite takes forever, people step on each others toes and it can be difficult to get things done. Access control and permissions is also difficult if you ever want to transfer ownership to another team. The happy middle ground is to follow Conways law a little and have a few related apps/modules belonging to the same team in one repo and an a CI that has a simple integration test for all of them. It's fairly natural to achieve this if you don't have people following cargo cult memes.
exabrial 4y ago
I mean you can do whatever you want, but versions, tags, releases seem like a lot less work.
v3ss0n 4y ago
How it works with a team More than one person? How many merge conflicts it gonna introduce with just a team of 4? Mono repo also means that merging both frontend and backend into one, even with totally different languages? Should be fine with traditional web framework but how it will become for API+ SPAs?
throwaway892238 4y ago
The whole thing really comes down to this difference: If you want to change one piece of software, it's faster and easier with multirepo; If you want to change many pieces of software, it's faster and easier with monorepo. All a repository really is, is a collection of files in one "bucket". No magic involved, no secret sauce, no woo-woo philosophical jargon to intone, no technical mastery to obtain. There are other features of a repository, like branches, tags, subrepos, LFS, etc. But you don't even really have to use any of those things. At its core it's just files in buckets. The complexity comes in once you have to start working with these buckets of files. How do I change a lot of files in one bucket? How do I change a lot of files in a lot of buckets? When I want to perform an action with the files ("build", "test", "deploy", etc), in one or many buckets, how do I do that? How do I do these things with multiple people simultaneously changing the same files? Or files that depend on files that depend on files? How do I coordinate the subsequent actions of the first actions on the buckets on the files with multiple people testing multiple changes simultaneously? No matter what option you choose, there will be choices you have to make which require compromise. You will eventually have to work around the complicated consequences of these compromises. No matter what you choose, there is always a way to make things easier, but it always requires work to get there. So really the question isn't monorepo or multirepo. It's what actions do you want to make easy first, and what consequences you want to spend engineering effort to fix first. But at the end of the line, assuming your product exists long enough, everybody ends up with basically the same system, because all of them, when people use them in complex ways, require complex solutions.
- jaxn 4y ago
  In my experience is it not about how many projects we want to change at once as much as it is about how many we want to test/build at once.
  - throwawaaarrgh 4y ago
    Aren't you test/building them because you changed them?
    - 4y ago
LarsDu88 4y ago
Meta, Google, Microsoft, Twitter -- all of these companies use the monorepo. I wonder if taking the monorepo (and managing it successfully) actually has something to do with company's becoming successful.
coding123 4y ago
This bugs me about Java software. Instead of having a single library, the library maintainers create some interface and then 30 more jars that implement it - xwidget-dropbox, xwidget-google-drive, xwidget-onedrive, etc... It's just really stupid. Everyone just pull down xwidget and move forward. Same goes for npm packages and front end things, rely on your treeshaker and ES6
hendry 4y ago
Thanks, inspired to create a video because of this post https://youtu.be/pKOF2DQnfno https://youtu.be/pKOF2DQnfno
LarsDu88 4y ago
To all the folks thinking that this project is too small to be representative of the benefits of monorepo -- both real world success stories and theory point towards monorepo being superior. I've seen this at several orgs. A polyrepo runs into dependency hell, so they go to microservice. Now you have dependency hell and microservice hell. Furthermore, your microservice have APIs that need to be updated, so each change needs to be done in multiple microservice repos. So to solve one problem you now created three problems. Monorepo reduce dependency hell to source control. The vast majority of monorepo issues people encounter are due to using 'git' which is why Google and Meta do not use 'git' at all.
- hbrn 4y ago
  All of your dependencies are individual repositories. The whole world runs on polyrepo. If that's not a success story, I don't know what is.
  - LarsDu88 4y ago
    Polyrepo is literally the only way you can run open source projects. Particularly in academia. People constantly confuse the open source world for how they should run their private companies. The evidence from the most successful software companies run directly contrary to this model.
    - hbrn 4y ago
      You're confusing correlation with causation. The evidence suggests that buying expensive cars makes you rich.
TeeMassive 4y ago
Where I work we just package everything (nugets, python packages, npm) on our Artifactory. Contracts dependencies (DLLs, protobufs) are also distributed as packages. We made it easy to fetch and test the source and allow developers to develop, debug and test those dependencies with their own project if needed. Every time we try to assemble repositories in macro-repos we always end up regretting it. Multiple dedicated repositories allow autonomy for teams and enforce modularity and coding as a library. Monorepos have a tendency of becoming huge merge trains easily and often derailed and with lots of fear of being blamed on stepping on someone else's toes. We update often all our projects knowing full well that not doing so is just borrowing development time at high interest rate. As a side-note when we do have to do an assembly of different code base, we use git-subrepo: https://github.com/ingydotnet/git-subrepo https://github.com/ingydotnet/git-subrepo which provide the best of both submodules and subtree.
erik_seaberg 4y ago
People keep raving about google3, but it had recurring problems with changelists (far outside your own project) that break the world and stop you from testing or deploying anything new for hours. Eventually they had to give up on using everything at head, and instead use known-good versions of major shared dependencies that were a few hours or days behind head. I want to bump our dependencies when we have the slack to handle the risks. Alpha testing every commit of every library is not really how our team is supposed to be spending our time.
dima_vm 4y ago
You cannot open-source parts if it's a monorepo.
28304283409234 4y ago
Does this discussion not also apply to most mono vs poly scenarios? Monolith vs microservices comes to mind. The infamous monolith kernel vs micro kernel discussion between Tannenbaum and Torvalds. Even small companies vs large ones. We shift the complexity and pain to where we are most comfortable with it. Or where we see the best benefits from the trade off.
sakex 4y ago
> I am here to tell you: if you are running a software business and you aren't at, like, Google-tier scale, just throw it all in a monorepo. Funny because Google is famous for having a monorepo at scale.
Dave3of5 4y ago
I mean the only ways you can work with a monorepo are either submodules or a package manager. Do devs really think these are better than a folder ? Having used both they both introduce unneeded complexity and get in the way of actual development. If you use a sensible and modern CI they all support a monorepo flow.
deleted 4y ago
[deleted]
dajonker 4y ago
I couldn't agree more with the author, there's nothing worse than the self-flagellation of having a single change in one repository triggering a snowball effect where you have to update a dozen of other repositories, open another dozen pull requests to be approved, and carefully time releases so you don't break anything.
- akoncius 4y ago
  then you are doing at least one thing wrong.. software versioning was invented long time ago and we survived. people implement distributed monoliths and then complain that polyrepo sucks :)
maxyurk 4y ago
IMO it's not really a monorepo. There's a single codebase really. The author didn't meet the real monorepo pain points yet. The author didn't mention git operations slowing down, CI not handling the scale of the commit events from a single source and so on, instability of codebase due to many parallel changes, painful merges etc. The difference between 150K LOC and Google's largest codebase in the world is huge and AFAIK google has a monorepo. From my experience real monorepo requires a lot of tooling and maintenance.
fergie 4y ago
But doesn't Google actually have a monorepo?
jrmcauliffe 4y ago
Once you reach a certain scale in a particular type of company the repo just becomes another type of org unit. Your giant fish swallows a minnow, it looks good on paper, someone does a half arse integration, then that repo becomes the one. The one that your manager says, under no circumstances no-one commits to, because then it's OUR repo (with whatever baggage that brings)... The org depends on the code, but the org actively resists it like a virus. This kills any attempt at the mono repo at any largish non FAANG org.
jay-barronville 4y ago
This might be an unpopular opinion, but to me, this is just an unsolved problem. Between polyrepos and monorepos, I’m not convinced either of them is a clear winner. I’ve used both structures and as a project grows, I dislike them both for different reasons.
Shinchy 4y ago
Having worked for a company that had a monorepo approach and seeing what happens when it hits its apex of issues I don't think I would ever advocate for one again. I'm sure it works for some companies, but when it goes wrong it can literally destroy an entire company.
- capableweb 4y ago
  Mind expanding on exactly what went wrong? How was the choice of monorepo related to the destruction of the company?
ozim 4y ago
So author seems to have problem with premature optimization. It is not that you should not use monorepo for everything - only when starting out maybe you will be better off starting with single repo and then if you have the need to make a separate one.
etherealG 4y ago
For your 2nd drawback, I can recommend checking out git subtree. It allows 2 way split and merge of any subtree of a repo. You can easily open source part of your repo in a “fork” and keep it up to date with your mainline as well as incoming pull requests with very little effort.
patrick451 4y ago
One problem he didn't mention with the poly-repo approach is that repositories are like rabbits: as soon as you have more than one they begin breeding. Repositories, plural, beget more repositories. The problem of course is that the only way to share code between repos is to create another repo. As the saying goes, once you have two things, you will soon want three. "Oh, repo B needs to use some code in repo A. We'll just split that out into a separate repo." Be very wary of this impulse. Soon, you end up in the situation I'm in now, where you have hundreds of tightly coupled repos, some with a single file! Worse, these all compose at runtime into only one application running on one host. Consider instead that perhaps repo A and repo B should be joined. Or, (yes I dare say it), that perhaps the code should be duplicated.

263 points by jmduke 4y ago

''I am here to tell you: if you are running a software business and you aren't at, like, Google-tier scale, just throw it all in a monorepo.'' Google _is_ a monorepo

BoorishBears 4y ago
It really paints a picture of the authors credentials to be making this declaration. All signs for the rest of the world point to the opposite conclusion: unless you're Google-scale, you don't have Google level resources. Google has more engineers working on developer experience than most companies will ever have period. And monorepos work best when workflows are carefully thought out with clever application specific tooling. - I think the author is probably working on a 1-10 developer project (and I'm leaning towards 1) and has confused the convenience with having things in reach when the entire system fits in your mind with the general benefits of a monorepo. I also wonder if they read any of the letters they linked too...
- dastbe 4y ago
  > All signs for the rest of the world point to the opposite conclusion: unless you're Google-scale, you don't have Google level resources. Google has more engineers working on developer experience than most companies will ever have period. I feel like this was true ~5 years ago, but these days the tooling around scaling monorepos is safely supporting O(100) developers without a lot of overhead. > And monorepos work best when workflows are carefully thought out with clever application specific tooling. I don't see any meaningful distinction with how well thought out workflows need to be between mono and polyrepo.
  - BoorishBears 4y ago
    > I don't see any meaningful distinction with how well thought out workflows need to be between mono and polyrepo. You completely failed to parse the sentence. The point being made isn't "well thought workflows are only for monorepos", that applies to the "needing clever application specific tooling" part. Vendoring/Versioning for discrete packages is a heavily invested in problem space for most tech stacks you'll come across. But if you build a monorepo and don't end up with a build system that takes on those responsibilities you end something that doesn't scale to even moderately large interconnected components. OP has such a tiny project that I'm not convinced they're even dealing with dependencies in a traditional sense. But well before you get to Google scale, you'll run into situations where you just want to change one thing and don't want to change every single downstream dependency which normally would have been isolated via a discrete package that doesn't have to change in lockstep. And then that exact same pain starts to exist for deployments and needs to be worked around. > I feel like this was true ~5 years ago, but these days the tooling around scaling monorepos is safely supporting O(100) developers without a lot of overhead. Nothing about the above has changed in the last 5 years, it's kind of the ground truth of monorepos via multiple repos: You're the first person I've ever seen imply monorepos don't offload complexity to tooling, even amongst proponents.
    - dastbe 4y ago
      > Vendoring/Versioning for discrete packages is a heavily invested in problem space for most tech stacks you'll come across. 100% disagree. The problem of "How do I define my dependencies and have a package manager reify that into a concrete set of versioned dependencies" may be a solved problem, but the tools for tracking dependencies across many repos and driving upgrades is neolithic. About the only company I've seen that does this well is Amazon, and they have yet to sell us version sets as a service. > OP has such a tiny project that I'm not convinced they're even dealing with dependencies in a traditional sense. But well before you get to Google scale, you'll run into situations where you just want to change one thing and don't want to change every single downstream dependency which normally would have been isolated via a discrete package that doesn't have to change in lockstep. And then that exact same pain starts to exist for deployments and needs to be worked around. As I alluded to above, the dual of this is that getting everyone to update their dependencies is orders of magnitude more difficult when you have a polyrepo setup, even if we're working under the ideal situation where repo setup is standardize to such a degree that a person can parachute into a repo and become effective within minutes. > Nothing about the above has changed in the last 5 years, it's kind of the ground truth of monorepos via multiple repos: You're the first person I've ever seen imply monorepos don't offload complexity to tooling, even amongst proponents. Both polyrepo and monorepo have complexity in scaling that is handled by their tooling. The difference historically is that OSS polyrepo tooling has been better and better integrated because that's just how most things are built in any language, but that has been improving over the past ~half decade * Bazel maintenance complexity has dropped precipitously, and many of the initial bottlenecks you hit with it have been solved in OSS. * If you're anti-bazel, gradle and cargo monorepo support is decently good. I believe the same is true in js these days, but I don't have hands-on experience * Services for managing monorepos like sourcegraph for codesearch or mergify for submit queue now exist that make it easy to adopt the patterns that work well at large companies. * Microsoft and others have invested in git to improve scalability of developing against large repos. * you have OSS tools like git-branchless that further improve the experience of working in a monorepo There are a bunch of companies in the O(100) - O(1000) developer range that are using this stuff and it works very well.
    - tylerhou 4y ago
      > But well before you get to Google scale, ... doesn't have to change in lockstep. Not updating dependencies is the equivalent of never brushing your teeth. Yes, you can ship code faster in the short term, but version skew will be a huge pain in the future. A little maintenance every day is preferable to ten root canals in a few years.
CobrastanJorji 4y ago
Yes, absolutely. Nonetheless, I think the author may be right, except that by "Google" they mean "large." This is a fundamental misunderstanding of just how large Google and its peers are. I think it's more interesting to consider three sizes. If you're small, everything will fit nicely in a monorepo. If you're large, you'll want lots of repos. There aren't really any off the shelf monorepo options that scale super well, so using a bunch of small repos is a great way to deal with the problem. Plus, you probably don't have a full time staff babysitting the source repos, so you want some isolation. If someone in another org is breaking stuff left and right, you don't want the other orgs to be affected. If you're GIGANTIC, monorepos are a pretty great option again. You'll probably have to build your own and then have a full time group of people maintain it, but that's not a huge problem for you because you're a gigantic tech company. You can set up an elaborate build system that takes advantage of the fact that the entire system is versioned together, which can let you almost completely eliminate version dependency hell. You can customize all of your tools to understand the rules for your new system. It's a huge undertaking, but it pays off because you've got a hundred thousand software engineers.
- jeffbee 4y ago
  > There aren't really any off the shelf monorepo options that scale super well How can you say this when Perforce on a single machine took Google to absolutely terrifying scale? There is no chance that your mid-sized software company will even slightly tax the abilities of Perforce. What I believe you meant was there aren't really any good options to make git tolerable for non-trivial projects, and with that I wholeheartedly agree. And that's why these threads are so tiresome: they always boil down to people talking about what git can and cannot do.
  - sanderjd 4y ago
    The Linux kernel seems pretty non trivial...
    - jeffbee 4y ago
      Google3 is no joke 1000x larger than Linux.
thundergolfer 4y ago
It's awkwardly phrased, yes, but what the author is saying is that Google-tier scale companies will have an awful time migrating from poly to mono-repo, and a not fun time being mono-repo for the first few years. For company's not at this tier, it isn't that hard to migrate to a monorepo and the benefits will be more immediate because the tools (eg. git) won't be screaming under the load. (My personal 2c is that you can be well below Google-scale and still hit the limits of the common tooling when using monorepos. Canva, Stripe, and Twitter are examples)

harryvederci 4y ago

I use a megarepo, if that's a thing. One repo with multiple unrelated apps, plus a "shared" directory. I could put the shared dir in a package, but so far I don't really see the need.

jvolkman 4y ago
Sounds like a monorepo.
- harryvederci 4y ago
  Maybe I should have put emphasis on the word "unrelated".

throwaway_au_1 4y ago

I would like for any article insisting I adopt any practice to put a little more effort into doing so. This seems to amount to "it's good", "I should have done it sooner", "it's not so bad, trust me", and "smart people like it, too". Could have been a tweet.

yodsanklai 4y ago
Agree, not a good article. Hopefully it'll trigger interesting discussions.

arrakeen 4y ago

this is the first i've heard of "microrepositories" but it appears like the author may have just been overzealous with partitioning their repositories and has now settled on what most of us would just call a "repo".

yodsanklai 4y ago

My understanding is that Meta/Google had to rewrite tons of stuff (distributed file systems, CVS, search tools, build tools...) and need many teams to maintain all these systems, and sometime their developer experience is inferior to what you get with off-the-shelf OSS. I'm not super familiar with this topic as I don't work with systems of that scale, but I'm wondering was it worth it or necessary? what would be the alternative?

hahaxdxd123 4y ago
> sometime their developer experience is inferior to what you get with off-the-shelf OSS. The internal version control system that I use (Fig, which is apparently based off Mercurial) is so good I've basically never had to think while using it. I can't really say the same about git. CodeSearch is also, AFAIK, a lot better than what you would get OSS. I think the productivity gains across 100K engineers is definitely worth it.
Shish2k 4y ago
> sometime their developer experience is inferior to what you get with off-the-shelf OSS Working at Meta, being able to use Sapling (our Mercurial fork) is actually one of the highlights and I was pretty miserable every time I needed to go back to Git for my open-source work — thankfully it was open-sourced with a git backend a couple of weeks ago, so now I get to combine the excellent sapling CLI with the github hosting service :D
skuzye 4y ago
I have to say, I really enjoy using Google monorepo and all tools available to me. Having said that, it's not free of course. There are many engineering teams and computer power being invested to provide this environment to all googlers.
twblalock 4y ago
I've seen what happens when an org tried to adopt a Google-style monorepo without making the investments in build tooling and cultural change. It was a disaster. All of those things are needed even at orgs much smaller than Google, or you will end up with an unbuildable, unmaintainable, unreleasable mess. For orgs that can't make those investments, I think a repo per team is the best approach. Each team can treat their repo like their own little Google-style monorepo if they want to.

015a 4y ago

IMO the biggest drawback to a monolith, maybe beyond those listed, is losing the 1-1 mapping of changes to CI to releases. If you know "this thing is broken", the commit log is a fantastic resource to figure out what changed which may have broke it. You submit a PR, and CI has to run; getting most CI platforms cleanly configured to, say, "only run the user-service tests if only the user-service changes" isn't straightforward. I understand there are some tools (Bazel?) which can do it, but the vast, vast majority of systems won't near-out-of-the-box, and will require messy pipelines and shell scripts to get rolling. There's also challenges with local dev tooling. Many VSCode language extensions, for example, won't operate at their best if the "language signaler" file (e.g. package.json for JS) isn't in the root of the repo; from just refusing to function, to intellisensing code in another project into this one, all manner of oddities. Meanwhile; I don't think the purported advantages are all that interesting. Being able to write the blog post and make the change in the same PR? I've never been in a role where we'd actually want that; we want the change running silently in prod, publish the blog post, flip a feature flag or A/B test proportions. Even an argument like "you can add schemas and the API changes in one go"; you can do that without a "monorepo", just co-locate your schemas with the API itself, this isn't controversial or weird, this is how everywhere I've worked operates. None of that is to even approach the scale challenges that come with monorepos at even reasonable sizes; which you can hit pretty quickly if you're, say, vendoring dependencies. Trying to debug something while the Github or Bitbucket UI is falling over because the repo is too large, or the file list is too long, isn't fun. I'm not going to assert this is a hill I would die on, but I'm pretty staunchly in the "one service, for one deployment artifact, for one CI pipeline, for one repo" camp.

dogleash 4y ago
Your examples of CI and VSCode are on point, and in the bigger picture it's always about tooling. The mono/multi repo argument is fundamentally boring to me because it always boils down to whether the shape of the tooling problem is easier to work with on this or that side of the divide. The answer is always whichever tradeoffs work best for your situation, and the reason at the end of your post is as good of a reason as as any.
xmodem 4y ago

mr90210 4y ago

Just use what works well for you and your team, how about that?

fullstackchris 4y ago

My 2 cents: monorepos try to solve what is actually a series of devops problems, by essentially abusing the way an organization or individual uses git (or VCS of your choice). These tools that were never designed for devops in the first place. To elaborate: I think you are much better off, whether as a team or solo developer, developing small, isolated, and tested / reliable libraries and / or packages that can be combined as needed. Ideally, each package or library can build, test, and ship itself at the CI/CD level. Trying to encapsulate all that into one giant conglomerate of potentially different languages, file types, frameworks and much more sounds exactly what it is: a mess!

gorgoiler 4y ago
Developing components like this puts a lot of pressure on the components to each look like a piece of Real Software. Versioning, stability, changelogs, coordinating new features as beta releases followed by deprecation announcements and finally deprecation. None of this is unreasonable, technically — my Linux OS is made of many small stable projects after all, none of whom communicate with each other beyond broadcasting NEWS.md etc with each release. It’s also exhausting. Do your competitors hamper themselves by cosplaying a bazaar of open source projects coordinated by whatever your internal equivalent of the Debian project is? No! Move everything under one roof to truly reflect your org structure and take advantage of the fact that you all report to one CTO, you all have real-time communication between each other, and you probably know where everyone else is sat for eight hours a day. Move fast.
- ghc 4y ago
  > take advantage of the fact that you all report to one CTO How many companies even have a CTO that knows or cares about version control? In most companies version control is managed by IT if at all, and code for different products is split across different lines of business.
hgsgm 4y ago
Huh? Monorepo refers to storing your source code, regardle5 of how many binaries you build and run, not shipping your system as one giant monolithic binary.

jph 4y ago

Monorepo vs. Polyrepo guidance thanks to many contributors: https://github.com/joelparkerhenderson/monorepo-vs-polyrepo https://github.com/joelparkerhenderson/monorepo-vs-polyrepo

Yahivin 4y ago

Monorepos are great! They're so good one place I worked at had dozens of them.

__alias 4y ago
Underrated comment

siliconc0w 4y ago

Probably disagree - Google has built a boat-load of infrastructure to enable a mono-repo which you don't have and (I don't think) aren't even available either OSS or commercial. Use modules/libraries as an alternative to separate repos. This may require a few more commits for distributed changes but that isn't hard and forces you to consider realities like, "what if A service is deployed before B service?" which happen whether or not you're using a mono-repo. Also use data-lakes for data or generated artifacts (something based on S3 or an S3-alike).

hgsgm 4y ago
Google used Perforce for many years until they outgrew it. What size org outgrows Perforce but can't develop and maintain their own thing?
- hocuspocus 4y ago
  It's just one piece of the puzzle though. Have you noticed how companies running Bazel usually have a dedicated team? Many orgs are absolutely not interested in that.
  - jeffbee 4y ago
    No, i haven't noticed that. My team just uses bazel. What necessitates a dedicated bazel team?
atomicfiredoll 4y ago
my understanding is Nrwl's Nx is built by ex-Google employees, that their integrated repo style is based on the approach used at Google, and that it's open source (they make money with their paid build caching solution.) Most companies aren't Google. So, the fact that they can't run the exact same tools doesn't seem like a real blocker when alternatives exist. I've used Nx a bit before and intend to for my next project. Even as a solo dev initially, I'm optimistic it will help me keep things organized and catch potential issues with shared code via the dependency graph.

IshKebab 4y ago

> You cannot apply role-based access controls to certain parts of the repository for security/compliance reasons. You can (at least for write access). Maybe he's talking about read access.

ridiculous_fish 4y ago
This is a major sticking point for us. Many of our repos are 3rd party, made available to us under different licenses. Some require individuals to be licensed, and so we use ACLs for both read and write access. Some of our source is export-controlled and this also is enforced via ACLs at the repo level.
- IshKebab 4y ago
  They require a license for read access? What are they? I don't think I've ever seen anything like that.

jgavris 4y ago

Even in a private monorepo, Google has tools like Copybara that make it somewhat easy to 'open source' small pieces of it and sync them in and out of a 'public' repo. The benefits of a monorepo and just having to 'git checkout' and your entire environment 'just works' (I'm looking at you, scripting languages and build systems) is not to be taken lightly.

gorgoiler 4y ago

Splitting your business’s mission across hard repository boundaries implies that… you know what you are doing! If that’s you then congratulations. Also: you’re kidding yourself. For the rest of us, being virtually unable to rethink the structure of our components because they are ossified as named repositories is a technical and social disaster. Whole teams that should have faded and been reabsorbed elsewhere will live forever because the effort to dismantle and re-absorb their code into other components is astronomical. The value we bring as engineers is in making sequences of small changes that keep us moving towards our business goal. Boundaries that get in the way are anathema to good engineering. Its exactly as if you were unable to move code between top level directories of your project. Ridiculous.

mansoon 4y ago
But, but, my modularity microservices as my self shibboleth that we are going to google scale next week right after I figure out what our product is!

pancakemouse 4y ago

Anecdotally, as a developer I really appreciate a well-thought-out monorepo. All the code is there. You don't have to dig around anything legacy or outdated. Searching is a breeze and it's _fast_. Updating both simple and complex logic is equally easy. You don't need to somehow discover that some other repo uses the thing you changed. Everything has the same deployment process. The tests test the thing, as soon as you push it - instead of months down the line, when someone finds forgotten repo again and oh, it's never been tested with the new functionality in our main repos. Dependency hell doesn't really exist. There seems to be more discipline around monorepos because they're collectively owned, whereas many repos across an org each become someone's baby, and practices diverge until teams in the _same company_ fork each others' repos because they don't get along.

shagie 4y ago

For me, this boils down to a question of "what set of problems do you want to solve?" and "what set of problems do you want the system to solve for you?" To this end, I am generally a monorepo proponent. The problems with "what gets built" and how do you do versioning of the components" for the systems that I work on is an easier set of problems to solve than "how do I ensure that I don't break anything else when I update this method?" and "if I do break something when I update the method, where is everything that needs to change?" I've had projects that were fractured across half a dozen repos where libraries were versioned and tagged and and deployed, and then the next down stream was updated to use the newly tagged version... and something broke. This was especially bad when the "something broke" was part of another team's code or it had been thrown over a wall. A bunch of changes, many PRs - some of which are just updating a version to pull from the artifact repo. When things were updated, I took the opportunity to move all of those projects into one monorepo and I only had one PR to review - that passed all the tests for all of the projects contained within the monorepo. Yes, deployments and tagging and the "well, just because you reved X to 2.3.0 doesn't mean that Y is at 2.3.0 too" (strict semver can get a bit on the messy side). It's a question of which problems do you want to solve. I tend to find that me solving the "how do I set up the ci file and coordinate version numbering" is an easier problem to solve than "all these builds need to be updated in a bunch of different repos and one of them breaks."

Octabrain 4y ago

I have my doubts about the mono repo approach. From the top of my head: 1. It might increase the complexity of other process on that repo: CI/CD configuration, makefiles, branching strategies, codeowners etc 2. The versioning might lose its meaning 3. Blast radius in case of screwing things up accidentally. I personally split repositories based on responsabilities: here the code base, here the iac, here the configuration, here the manifests. Always using a standard and predefined naming convention for the repository names. That being said, as always, it depends. I might embrace the monorepo if the context demands it and it has been properly discussed and evaluated.

cosmotic 4y ago
1. Often it ends up being easier 2. Versioning becomes easier 3. Blast radius is similar but cleanup is easier
- TylerE 4y ago
  More importantly re 3. things break sooner. This is good. The sooner the breakage is noticed the less merging there is to do…
- Octabrain 4y ago
  > 1. Often it ends up being easier It hasn't been like that most of the time in my experience. Having to integrate in the same CI/CD pipeline, let's say, from the code base side, linters, tests and build, from the iac side, formatting, validation, plan, from the manifests side, yaml linters, builds (e.g kustomize), dry runs etc. Now, think on all the previous stuff to consider but also adding the logic for something common like "based on the branch, run this and not that over this environment". Of course you could sepparate the previous in different pipelines, but then you might have to control somehow the order of execution of each. It is possible, however, it might be complex. Also, one must consider that sometimes, reaching a consensus between devops and developers can be tricky and you might end up in people stepping in each other toes. 2. Versioning becomes easier How so? From my point of view, if I find a repository containing an specific thing (e.g a Terraform module) and I see the release version "1.0.1" I can get a clear idea of what that it implies. However, if that module is versioned along the terraform files themselves as well as the code base what was fixed on that "1.0.1" version, the Terraform module?, something in the app itself? Of course, I can go to the release notes and spend some time reading them, but you better be following some good practices on that regard, otherwise is gonna be time consuming. > 3. Blast radius is similar but cleanup is easier I could buy this one, however, it is preferable to assume the risk because of how easy is to clean up things in case they go south or is better to avoid the risk in the first place by having everything sepparated on the first place? I guess context (complexity of the project, culture, communication across teams) is the key.

karmakaze 4y ago

I was curious why it was a big deal either way for a smaller codebase/documents to use a monorepo or not. It's mentioned: > You may be reading this short essay and thinking: "Yeah, no shit! Of course you do not need a monorepo when you're talking about, like, 150K lines of code." > Maybe it's some abstract sense of purity; maybe it's because you didn't realize most IaaS let you deploy from a subfolder. Whatever the reason is: trust me. Move it into a monorepo. If you do and you end up regretting it, [email me] and I'll donate to the charity of your choice. So the new thing being said here is that a monorepo makes sense even at smaller scales. This seems to on the far other end of the scale with a main app, supporting infra, docs, marketing, etc. I've certainly used fullstack repos with backend and frontend together and it's fantastic to use and deploy from commits. I can't infer from this article or my experience what it would be like to work with a monorepo at medium scale say 10-100 apps with 10s of teams. That would take more adaptation than simply "deploy from folder". Dependencies as mentioned in another comment and being able to detect if each target should be retested/deployed seems like the tough one.

TylerE 4y ago
100 apps across 10 teams doesn’t sound very medium scale to me.
- karmakaze 4y ago
  I said 10-100 apps, and should maybe have said 5-10 teams rather than 10 teams, but 10 dev teams in a company to me is medium-sized. Or do you mean it's less than medium-sized?

scarmig 4y ago

I always find monorepo/polyrepo discussions tiresome, mostly because people take the limitations of git and existing OSS tooling as a given and project those failings onto whichever paradigm they are arguing against. I'm pretty excited for new OSS source control tools that would hopefully help us move past this discussion. Particularly, Meta's Sapling[0] seems like a pretty exciting step forward, though they've only released a client so far. (MS released its VFS for Git awhile back, but unfortunately now is deprecated.) [0] https://engineering.fb.com/2022/11/15/open-source/sapling-source-control-scalable/ https://engineering.fb.com/2022/11/15/open-source/sapling-so...

ricardobeat 4y ago
Sapling is specifically built to deal with issues arising from gigantic monorepos. Why and how do you think that will move us past this discussion?
- Spivak 4y ago
  Either one of monorepo minus the major downsides or polyrepo minus the major downsides effectively makes the discussion of the trade offs moot and the first one to happen will likely “win” with not enough gained by deviating from the norm to switch once people adopt it.
- deleted 4y ago
  [deleted]
mkl 4y ago
Sapling discussion a couple of months ago: https://news.ycombinator.com/item?id=33612410 https://news.ycombinator.com/item?id=33612410
xixixao 4y ago
https://graphite.dev https://graphite.dev is also aiming to provide a FB-like workflow including stacks.
nixpulvis 4y ago
Can't we just generalize the package manager already and push it into VCS? I just want to commit some code and roll it out in the next release. Somewhere in the tree is a top level makefile or something. Stop making this complicated. When it comes to scale, force versions to be incremented at the same time across the lot. You can even spin up a new deploy set and gracefully handoff load. My point is. Just let the source code, in whatever language, be distributed as packages in as modular a way as developers want. One way or another you're going to end up with a makefile or shell script that builds the damn thing. If you don't, then someone fucked up and your build is effectively broken. Monorepo or not.

erksa 4y ago

Earlier in my career I was so excited to learn about microservices, as I was working with a huge monolith at the time. Then I went a bit overboard, and for a new prototype I had all these services in different repos and running independently. It was all very cool and I felt very accomplished, only to later (and now painfully obvious) be pointed out it was completely unnecessary, to the point each function basically was its own deployment. But not in a way that made any sense. If you were looking for the _most_ expensive way of running things, maybe.. hah Tunnel-vision is a powerful thing, and if this is how people are splitting repos now a days, it seems I might not have been the only one chopping up perfectly good dish only to be left with the individual ingredients.

scarmig 4y ago
You can have microservices and different repos, microservices and a monorepo, or a monolith and monorepo. In principle you could even have a monolith split across different repos (e.g. each library in a separate repo). What's included in a deployed binary has nothing to do with how the source code is structured.
lloydatkinson 4y ago
The post is about monorepos not microservices.
- erksa 4y ago
  Yes, that's not lost on me, however my point might have been. The point was, when learning this by yourself, you can really go down the wrong path with the right advice at the wrong time. This seems to hold true for the monorepo discussion as it did/does for microservices.

SpeedilyDamage 4y ago

No! I'm tired of being told how to architect systems because other folks refuse to grow their knowledge. Because I'm good at my job, a monorepo would be pants-on-head stupid. Because I know how to make separate codebases interact with one another successfully, it would be a massive mistake to lose those advantages, all because you (not me) run into problems when you try to do what I find perfectly natural. And if you work with me, you will learn, and I will guide you, and in time you will experience systems design as I do, and no longer be shackled to only doing the things you understand today. But no, "do this because I don't know how to do that" will never be a winning argument.

lliamander 4y ago
I am myself am skeptical of monorepos, and i have only worked with polyrepos, but I am curious what strategies you use that help make you effective with multiple code bases?
- SpeedilyDamage 4y ago
  Keeping the spirit of microservices alive is super duper important, so not entwining two services with shared code or responsibilities is key, and forcing interaction to happen through their defined APIs.
  - lliamander 4y ago
    Yeah, that's pretty much what I do as well. Shared libraries get their own repo as well and are published in a package manager. Honestly, in terms of enabling concurrent development across multiple teams while simply using common OSS software, polyrepo seems a lot easier.
sanderjd 4y ago
I know how to do that stuff, it just sucks more than having a monorepo with good tooling, in my opinion.
- 4y ago

osigurdson 4y ago

I call bike shed on the entire debate. One repo or many can work fine depending on the situation.

kmbfjr 4y ago
The whole debate gives work to people who don’t want to work on the actual product or service.
- deleted 4y ago
  [deleted]

raydiatian 4y ago

Here’s why I’ll never go back to anything but a monorepo. My previous org decided on this for their source control/build dependency layout: 1. Monolith, home to front and back end, artisanal handcrafted openapi spec that plugged into nothing and was just for show. 2. Hand-painted service-cli library, which was supposedly going to be open source for customers but never was, and only mattered to the front end, back in the monolith. We manually kept it in sync with the backend, to everybody’s benefit (YFR). Hope you see it: backend changes left the monolith into the service-cli, back to the monolith. Face, meet wall. 3. A knapsack full of tiny libraries which all could have been wrapped up in one. They were all dependencies that plugged into the monolith and the service-cli. And they all had subdependencies. 4. Everything version-wise was synced via private npm registry. So if you wanted to make a change that went into the next version of monolith, but the code had to live in a leaf dependency way down at the bottom, you had to run a relay race marathon of red-tape PRs that you had to bribe your coworkers to pay attention to and/or approve. The entire time, several of us on the team were begging the core team to repair this and they basically told us “no we believe in the tenets of sadism, now take your allotted pain, feature-bitch.” This company employed 150+.

bluGill 4y ago

Polyrepo and monorepo both work well if you have the right tools and workflows in place. Both have substantial downsides. If you can't admit the above then you are either ignorant or trying to sell something. Either way I don't want to talk to you. The rest of us can now discuss pros and cons.

sanderjd 4y ago
Yeah agreed. But it's also ok to other one over the other on the full balance. But that also doesn't mean that either preference is the one true way.

ridiculous_fish 4y ago

How do monorepos work with something like gitlab or github, which expect a CI file in the repo root? My workplace has dozens of repos, and they all have separate CI needs and test suites. Trying to shoehorn them all into a single set of CI pipelines would be very difficult, even with the include and parent-child pipeline features.

shagie 4y ago
The CI file can specify what triggers different workflows. https://docs.github.com/en/actions/using-workflows/triggering-a-workflow#accessing-and-using-event-properties https://docs.github.com/en/actions/using-workflows/triggerin... on: pull_request: types: - opened paths: - '.github/workflows/**' - '.github/CODEOWNERS' - 'package*.json' That triggers when those files change. The "get them all to run in a single repo" is the challenge of the monorepo. The trade off is "if you change a common library, how do you trigger the test suites in all of the downstream repos to run with the new common library and verify that they still work?" With a monorepo, that second part is solved for you... at the cost of needing to solve the problem that you're asking about.
atomicfiredoll 4y ago
I have the most expense with Nx and can only speak to that. Nx projects are tied together with a dependency graph. The workflow/pipeline replaces something like "npm build" with with "nx affected --target=build --base=main". Nx compares the code to the base branch, identifies the leaves of the tree (projects) affected by the code change, and runs the specified task for each project. Each project can specify what that task (technically the "target") means in it's own context by directing it to run a specific "executor." For some projects, "build" may run a Weback executor, others may map it to run a Vite executor. Edit: To summarize, if you change something deep down in the tree like a design token, you can run the "test" and "build" tasks for everything affected by that change to make sure none of your apps break. If those projects are SPAs built in different frameworks, each can map the task to run the framework specific "test" or "build" executor. Now you have automatic assurance that all your apps still work. If that sounds like a lot for the CI to do, that's where distributed caching the of task results come in.

ohCh6zos 4y ago

I have worked at places that have micro-repos, places that have polyrepos, and places that have monorepos. From my observation monorepos are a sign of ossification and resistantce to new patterns and new technologies.

jsnell 4y ago

> Rolling out API changes concomitantly with downstream changes to the documentation or the OpenAPI spec. > Introducing feature-level changes and the blog post announcing those changes. These are horrible reasons to use a monorepo. Commits are not units of deployment. Even if you're pushing every system to prod on every commit, you'd still basically always want to make the changes incrementally, system by system, and with a sensibly sequenced rollout plan rather. To take one of the examples above, why would you ever have the code implementing a feature and an annoucement blog post in the same commit? The feature might not work correctly. You'd want to be able to test it in a staging environment first, right? Or if you don't have staging, be able to run it in prod behind a feature flag gated to only test users, or as a dark launch, or something to verify that the feature is working before letting real users at it and having it crash your systems, cause data corruption, or some other critical problem that would necessitate a rollback. But none of this pre-testing is possible if the code changes are really being done in the same commit as the public announcment. And talking of rolling back... When you revert the code changes that are misbehaving, what are you doing with the blog post? Unpublish it? Or do some kind of a dirty partial rollback that just reverts the code and leaves the blog post in place? The same goes for any kind of cross-project change[0], some of which appear more compelling on the surface than the "code and blog post in one" use case (e.g. refactoring an API by changing the interface and callers at the same time). Monorepos allow for making such changes atomically, but you'd quickly find out that it's a bad idea. There are great reasons to use monorepos, but this is not it. [0] I wrote more about this a couple of years back. https://www.snellman.net/blog/archive/2021-07-21-monorepo-atomic/ https://www.snellman.net/blog/archive/2021-07-21-monorepo-at...

sa46 4y ago
> you'd still basically always want to make the changes incrementally, system by system, and with a sensibly sequenced rollout plan rather. Depends. It's significantly faster to deploy everything at the same time and accept that unlucky requests might end up in a weird state than to safely sequence changes. In SRE phrasing, I'm choosing to spend our error budget to maximize change velocity by giving up on compatibility during deploys by skipping a multi-stage rollout plan. In return, I can condense a rollout to a single commit and deploy. A 99.9% availability target yields up to 86 seconds per day to pretend that deploys are "atomic".

voz_ 4y ago

Monorepos for life, don't conflate code storage and organization with deployment lifecycles tho.

snow_mac 4y ago

Just don't.

Vekz 4y ago

Anyone have resources on how to actually perform the migration of many repos to monorepo? Currently facing the same issue and want to make the consolidation move but now sure on how to merge the git history. This article and the previous one it links to are missing all the implementation details.

phphphphp 4y ago
Don’t, keep things simple. Maintain a separate archive of each individual repository and any history can be referenced in that repository when necessary.
j1elo 4y ago
I just finished a migration like that, and after some research ended up writing shell code to implement 3 methods and put them to the test. Maybe it is useful for you so I'll leave it here: https://gist.github.com/j1elo/6156437f26bfc580769613bb30419af8 https://gist.github.com/j1elo/6156437f26bfc580769613bb30419a... My first step was to learn the procedure that devs behind GStreamer used recently, when they went ahead and decided to merge all their disparate repos into a single monorepo. For my needs it is the less adequate of the three because it will break as soon as there is any collision between the desired destination and the already existing files of a repo. The second method is one proposed in the JDriven blog. It doesn't suffer collisions because the movement into destination subdir of each repo is done separately on the original repos, and then they are merged with everything already in place. Lastly, I found a third method which is the one I liked the most. It uses git subtrees, to perform the move and merge all in the target repo (the monorepo itself), without touching the original repos (unlike the JDriven proposal), so everything is tidy and left in place. Method 1 (GStreamer): * https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/474 https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/... * https://blogs.gnome.org/tsaunier/2021/09/29/gstreamer-one-repository-to-rule-them-all/ https://blogs.gnome.org/tsaunier/2021/09/29/gstreamer-one-re... * https://gitlab.freedesktop.org/thiblahute/gst-merger/blob/master/merge.py https://gitlab.freedesktop.org/thiblahute/gst-merger/blob/ma... Method 2 (JDriven): * https://blog.jdriven.com/2021/04/how-to-merge-multiple-git-repositories/ https://blog.jdriven.com/2021/04/how-to-merge-multiple-git-r... Method 3 (Subtrees): * https://mirrors.edge.kernel.org/pub/software/scm/git/docs/howto/using-merge-subtree.html https://mirrors.edge.kernel.org/pub/software/scm/git/docs/ho... * https://docs.github.com/en/get-started/using-git/about-git-subtree-merges https://docs.github.com/en/get-started/using-git/about-git-s...

ankitdce 4y ago

For everyone who complains about monorepos, remember some of the most forward thinking engineering companies like Google and FB also use monorepos. All the arguments that people make in favor of polyrepo are making so because of lack of strong tooling for monorepos. That's also why Google and FB would not have scaled if they were using GitHub / GitLab but had to build their own. Also Google's original source control was built on top of perforce!

fargo 4y ago
They way at least FB is using a monorepo is very different than any kind of monorepo most people imagine. It's not just about tooling, git itself could never handle it. I am all about using a monorepo but Google and FB having one is not the an argument for it.
- phyrex 4y ago
  how so?
threeseed 4y ago
a) There is no correlation between mono/poly repos and your ability to scale. There are many examples of successful companies using either approach. b) As a general rule people should be cautious about adopting approaches and technologies from Google, Meta without a clear understanding of why they need it. What works at their scale doesn't always apply to smaller teams.
afavour 4y ago
> remember some of the most forward thinking engineering companies like Google and FB also use monorepos As a counterpoint though I’d say that the issues Google and FB face, particularly in terms of the sheer scale of the work they’re doing, is pretty unique. Google literally invents programming languages for domains it feels needs them, I’m not about to blindly follow that practise either.
scarmig 4y ago
The issue with building a custom monorepo system that can handle Google's and Facebook's scale is that it fails to scale down, even to moderately large project and organizational size. It's expensive (think at least 7 figures opex) and not what most people should be doing. git, for all its issues (and I'm a git-hater), scales down to an individual coder and scales up (with a lot of hacks, the hacks being used varying depending on whether you're taking a poly or mono approach) to companies that employ thousands of developers.

jshen 4y ago

Does anyone have a pointer to a tutorial or example of what a successful monorepo and process looks like?

0xbadcafebee 4y ago

Do not listen to pithy blog posts. Seriously. There's too many factors to consider that aren't in the blog post. They get upvoted on HN here quickly, and most often when they are written from ignorance, which seems super confident, and leaves out all the detail and nuance that could make such an upvote questionable. I'm pretty sure the upvotes are based on an emotional reaction to the title alone. (The "reasonable drawbacks" part of this post should have been like 5 pages; and the same for multirepo) If you really want to know whether you should use a monorepo, first go buy a book about them. Then talk to senior leaders in organizations that have gone through an entire growth cycle and migrated from multirepo to monorepo or vice versa. The random dude on the random blog post doesn't have the perspective to inform you properly about the implications. Hell, I've done the switch twice, and I still wouldn't call myself an expert.

nonethewiser 4y ago

And trpc. If you're using typescript.

jalapenos 4y ago

I'd like to add my own PSA in concurrence with this. Just use a monorepo. Use tooling to work around its limitations if you reach that point. I work in a SaaS that's polyrepo based, having split from its original monorepo as part of a microservices push (which never succeeded, leaving us stuck half way in the worst of both worlds). Nothing has been more destructive to productivity than the polyrepos and their consequences. We're talking a 20% engineering spend dead weight loss. This is stark obvious to every single engineer, but trying to get people to accept that fact and sign off on a project to coalesce them back into a monorepo is just insurmountable. Polyrepos are irreversible damage, stay away from them. Hold the line on your monorepo. One organisation, one repo.

eclipticplane 4y ago
Either I work with you, or the _failed microservice push leaving us in the worst of all worlds_ is unfortunately common.
quickthrower2 4y ago
As a special case: polyrepos are fine for when you have a genuine plugin architecture. To be a genuine one: you have a public facing end to end documentation on how to use it. I would use chrome extensions vs. chrome itself as an example. Or same for VSCode. Then vendor plugins can be in their own repos. I think otherwise for simple module A and module B that interact privately, in an adhoc way, the separation of concerns is not there and monorepo is better.
lloydatkinson 4y ago
Absolutely this. I am experiencing a polyrepo setup of 254 repositories. The level of productivity loss is at a scale I don't think can be adequately describe in words.

wankle 4y ago

"Maybe it's some abstract sense of purity" Someone gets a few lines of code under their belt and now they overwhelmingly know what is best for all. It's simple, if it gets released together, put it in the same repo (even if it's a big repo like the Linux kernel). If not, separate it out so you can stay sane. If it's not released together then it's a monorepo, also defined as a big ball of spaghetti.

yellowapple 4y ago

Another big disadvantage of monorepos is the size. If you're having to clone the repo in its entirety on a routine basis (think CI/CD pipelines, or there's some other repo that needs to maintain multiple work trees referencing different commits), that repo being some massive monstrosity can be a real pain in the ass.

hansvm 4y ago

You have a monorepo anyway, and the choice is between dealing with it via wrappers automated or manual wrappers around tools like git (e.g., manually cloning relevant repositories and constructing a build environment referencing all the right versions appropriately) or whether you'll use dedicated monorepo tooling. That's still maybe an interesting question, but when appropriately framed it highlights that the real issue is that monorepo tooling may or may not fit your use case well, and you might be better served by hacking together more primitive solutions or making your own tooling. IMO, the major point of the article still stands; git and whatnot are pretty good, and the build/deploy tools interacting with your code are probably terrible, so absent major outside pressures you'll go a long ways figuring out how to leverage git (in a monorepo configuration) against jenkinbuckethubernetes, and that's often much less painful than inevitably figuring out how to marry two ancestrally disjoint codebases.

humanrebar 4y ago

Most opinions about monorepos conflate monocultures in source control with monocultures in build systems or workflows. Those two considerations are technically orthogonal. You can have one big repo with N build systems (like a node app and a python app in the same repo with disjoint workflows). And you can have N repos that have a common build system (each can combine into one big build with cloning and/or submoduling). I find most of these conversations are more interesting when people decide to make that distinction.

berjin 4y ago

I'm going to start a new term: goldilocks repo. Too small and you end up needing to deploy 5 apps to get your feature out and tests are difficult to coordinate. Too large and the huge CI suite takes forever, people step on each others toes and it can be difficult to get things done. Access control and permissions is also difficult if you ever want to transfer ownership to another team. The happy middle ground is to follow Conways law a little and have a few related apps/modules belonging to the same team in one repo and an a CI that has a simple integration test for all of them. It's fairly natural to achieve this if you don't have people following cargo cult memes.

exabrial 4y ago

I mean you can do whatever you want, but versions, tags, releases seem like a lot less work.

v3ss0n 4y ago

How it works with a team More than one person? How many merge conflicts it gonna introduce with just a team of 4? Mono repo also means that merging both frontend and backend into one, even with totally different languages? Should be fine with traditional web framework but how it will become for API+ SPAs?

throwaway892238 4y ago

The whole thing really comes down to this difference: If you want to change one piece of software, it's faster and easier with multirepo; If you want to change many pieces of software, it's faster and easier with monorepo. All a repository really is, is a collection of files in one "bucket". No magic involved, no secret sauce, no woo-woo philosophical jargon to intone, no technical mastery to obtain. There are other features of a repository, like branches, tags, subrepos, LFS, etc. But you don't even really have to use any of those things. At its core it's just files in buckets. The complexity comes in once you have to start working with these buckets of files. How do I change a lot of files in one bucket? How do I change a lot of files in a lot of buckets? When I want to perform an action with the files ("build", "test", "deploy", etc), in one or many buckets, how do I do that? How do I do these things with multiple people simultaneously changing the same files? Or files that depend on files that depend on files? How do I coordinate the subsequent actions of the first actions on the buckets on the files with multiple people testing multiple changes simultaneously? No matter what option you choose, there will be choices you have to make which require compromise. You will eventually have to work around the complicated consequences of these compromises. No matter what you choose, there is always a way to make things easier, but it always requires work to get there. So really the question isn't monorepo or multirepo. It's what actions do you want to make easy first, and what consequences you want to spend engineering effort to fix first. But at the end of the line, assuming your product exists long enough, everybody ends up with basically the same system, because all of them, when people use them in complex ways, require complex solutions.

jaxn 4y ago
In my experience is it not about how many projects we want to change at once as much as it is about how many we want to test/build at once.
- throwawaaarrgh 4y ago
  Aren't you test/building them because you changed them?
  - 4y ago

LarsDu88 4y ago

Meta, Google, Microsoft, Twitter -- all of these companies use the monorepo. I wonder if taking the monorepo (and managing it successfully) actually has something to do with company's becoming successful.

coding123 4y ago

This bugs me about Java software. Instead of having a single library, the library maintainers create some interface and then 30 more jars that implement it - xwidget-dropbox, xwidget-google-drive, xwidget-onedrive, etc... It's just really stupid. Everyone just pull down xwidget and move forward. Same goes for npm packages and front end things, rely on your treeshaker and ES6

hendry 4y ago

Thanks, inspired to create a video because of this post https://youtu.be/pKOF2DQnfno https://youtu.be/pKOF2DQnfno

LarsDu88 4y ago

To all the folks thinking that this project is too small to be representative of the benefits of monorepo -- both real world success stories and theory point towards monorepo being superior. I've seen this at several orgs. A polyrepo runs into dependency hell, so they go to microservice. Now you have dependency hell and microservice hell. Furthermore, your microservice have APIs that need to be updated, so each change needs to be done in multiple microservice repos. So to solve one problem you now created three problems. Monorepo reduce dependency hell to source control. The vast majority of monorepo issues people encounter are due to using 'git' which is why Google and Meta do not use 'git' at all.

hbrn 4y ago
All of your dependencies are individual repositories. The whole world runs on polyrepo. If that's not a success story, I don't know what is.
- LarsDu88 4y ago
  Polyrepo is literally the only way you can run open source projects. Particularly in academia. People constantly confuse the open source world for how they should run their private companies. The evidence from the most successful software companies run directly contrary to this model.
  - hbrn 4y ago
    You're confusing correlation with causation. The evidence suggests that buying expensive cars makes you rich.

TeeMassive 4y ago

Where I work we just package everything (nugets, python packages, npm) on our Artifactory. Contracts dependencies (DLLs, protobufs) are also distributed as packages. We made it easy to fetch and test the source and allow developers to develop, debug and test those dependencies with their own project if needed. Every time we try to assemble repositories in macro-repos we always end up regretting it. Multiple dedicated repositories allow autonomy for teams and enforce modularity and coding as a library. Monorepos have a tendency of becoming huge merge trains easily and often derailed and with lots of fear of being blamed on stepping on someone else's toes. We update often all our projects knowing full well that not doing so is just borrowing development time at high interest rate. As a side-note when we do have to do an assembly of different code base, we use git-subrepo: https://github.com/ingydotnet/git-subrepo https://github.com/ingydotnet/git-subrepo which provide the best of both submodules and subtree.

erik_seaberg 4y ago

People keep raving about google3, but it had recurring problems with changelists (far outside your own project) that break the world and stop you from testing or deploying anything new for hours. Eventually they had to give up on using everything at head, and instead use known-good versions of major shared dependencies that were a few hours or days behind head. I want to bump our dependencies when we have the slack to handle the risks. Alpha testing every commit of every library is not really how our team is supposed to be spending our time.

dima_vm 4y ago

You cannot open-source parts if it's a monorepo.

28304283409234 4y ago

Does this discussion not also apply to most mono vs poly scenarios? Monolith vs microservices comes to mind. The infamous monolith kernel vs micro kernel discussion between Tannenbaum and Torvalds. Even small companies vs large ones. We shift the complexity and pain to where we are most comfortable with it. Or where we see the best benefits from the trade off.

sakex 4y ago

> I am here to tell you: if you are running a software business and you aren't at, like, Google-tier scale, just throw it all in a monorepo. Funny because Google is famous for having a monorepo at scale.

Dave3of5 4y ago

I mean the only ways you can work with a monorepo are either submodules or a package manager. Do devs really think these are better than a folder ? Having used both they both introduce unneeded complexity and get in the way of actual development. If you use a sensible and modern CI they all support a monorepo flow.

deleted 4y ago

[deleted]

dajonker 4y ago

I couldn't agree more with the author, there's nothing worse than the self-flagellation of having a single change in one repository triggering a snowball effect where you have to update a dozen of other repositories, open another dozen pull requests to be approved, and carefully time releases so you don't break anything.

akoncius 4y ago
then you are doing at least one thing wrong.. software versioning was invented long time ago and we survived. people implement distributed monoliths and then complain that polyrepo sucks :)

maxyurk 4y ago

IMO it's not really a monorepo. There's a single codebase really. The author didn't meet the real monorepo pain points yet. The author didn't mention git operations slowing down, CI not handling the scale of the commit events from a single source and so on, instability of codebase due to many parallel changes, painful merges etc. The difference between 150K LOC and Google's largest codebase in the world is huge and AFAIK google has a monorepo. From my experience real monorepo requires a lot of tooling and maintenance.

fergie 4y ago

But doesn't Google actually have a monorepo?

jrmcauliffe 4y ago

Once you reach a certain scale in a particular type of company the repo just becomes another type of org unit. Your giant fish swallows a minnow, it looks good on paper, someone does a half arse integration, then that repo becomes the one. The one that your manager says, under no circumstances no-one commits to, because then it's OUR repo (with whatever baggage that brings)... The org depends on the code, but the org actively resists it like a virus. This kills any attempt at the mono repo at any largish non FAANG org.

jay-barronville 4y ago

This might be an unpopular opinion, but to me, this is just an unsolved problem. Between polyrepos and monorepos, I’m not convinced either of them is a clear winner. I’ve used both structures and as a project grows, I dislike them both for different reasons.

Shinchy 4y ago

Having worked for a company that had a monorepo approach and seeing what happens when it hits its apex of issues I don't think I would ever advocate for one again. I'm sure it works for some companies, but when it goes wrong it can literally destroy an entire company.

capableweb 4y ago
Mind expanding on exactly what went wrong? How was the choice of monorepo related to the destruction of the company?

ozim 4y ago

So author seems to have problem with premature optimization. It is not that you should not use monorepo for everything - only when starting out maybe you will be better off starting with single repo and then if you have the need to make a separate one.

etherealG 4y ago

For your 2nd drawback, I can recommend checking out git subtree. It allows 2 way split and merge of any subtree of a repo. You can easily open source part of your repo in a “fork” and keep it up to date with your mainline as well as incoming pull requests with very little effort.

patrick451 4y ago

One problem he didn't mention with the poly-repo approach is that repositories are like rabbits: as soon as you have more than one they begin breeding. Repositories, plural, beget more repositories. The problem of course is that the only way to share code between repos is to create another repo. As the saying goes, once you have two things, you will soon want three. "Oh, repo B needs to use some code in repo A. We'll just split that out into a separate repo." Be very wary of this impulse. Soon, you end up in the situation I'm in now, where you have hundreds of tightly coupled repos, some with a single file! Worse, these all compose at runtime into only one application running on one host. Consider instead that perhaps repo A and repo B should be joined. Or, (yes I dare say it), that perhaps the code should be duplicated.