30 ms·

Advantages of Monorepos (2015)

183 points by Naac 4y ago

pbiggar 4y ago
Monorepos are also great for small monorepos with just a few projects. The darklang monorepo [1] has a devcontainer that installs all the build tools for 4 projects which create 21 different services, using 6 languages, and building everything is one step. In fact, it makes it so easy to add new stuff that I didn't even realize we had 21 services til I counted. My first guess was 12. [1] https://github.com/darklang/dark https://github.com/darklang/dark
88913527 4y ago
In my experience, the developer experience for juniors is too much. Yarn + Lerna is just too much of a learning curve. However, having one repo and on CICD pipeline is convenient. But we've decided to divest from them. Your situation may not match mine, and that's okay.
- pcmaffey 4y ago
  Lerna is only needed if you're publishing multiple packages from the monorepo. If you're consuming your packages only within the monorepo for your various services, Yarn Workspaces is generally all that's needed.
- dimgl 4y ago
  Just don't use Yarn + Lerna :) pnpm is amazing
  - 88913527 4y ago
    Doesn't work older Angular projects. My monorepo is a hodge podge of technologies.
lisper 4y ago
It's very simple: with a monorepo you always have access to everything you need, together with a ton of stuff you don't. Whether or not this is advantageous boils down to whether the cost of not having access to something you need is greater than the cost of having access to a bunch of stuff you don't. As long as your system is reasonably efficient at letting you select small subsets of everything you could potentially have access to, the cost of having access to a bunch of stuff you don't need is essentially zero. Perforce is good at that. Git isn't. So people who use Perforce tend to think that monorepos are good and people who use git don't. And they're both right.
- mikepurvis 4y ago
  Years ago, Google had a gcheckout tool that would trace the dependency information for whatever project you were working on, and then selectively grab the portions of the monorepo that you were going to need for it. Maybe they still have that or it's evolved into something else; I dunno, I haven't been there in a really long time. Anyway, it seemed like such an obvious complement to the the perforce/monorepo style of working that I came away surprised that perforce wouldn't have hoisted such a thing into their product as a first-class feature. Tracing dependencies across a lot of different build systems is obviously not trivial, but it's not intractable, particularly if the tool is pluggable so that orgs can provide modules to handle their own particular approach.
  - jeffbee 4y ago
    What you are describing is an artifact of the old Perforce which copied everything in your client to local storage. After the conversion to srcfs and piper, which was more than a decade ago, this became unnecessary. http://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-source-code.html?m=1 http://google-engtools.blogspot.com/2011/06/build-in-cloud-a...
- throwaway894345 4y ago
  Honestly I would 100% do a monorepo every single time if there was good tooling for incrementally building and testing libraries. Having to rebuild every image from scratch for every single change scales miserably. Things like Bazel exist, but you basically have to have a team dedicated to operating it (maybe the difficulty varies by language, but it was a major pain when I tried to use it to build some relatively simple Python projects a few years ago).
  - AtlasBarfed 4y ago
    Is there, say, IntelliJ support for Bazel? Do you need a central server? I've heard bazel is a bear... But... all mature build systems are, because they become essentially enterprise workflow engines, process execution engines, internal systems integration hubs, and schedulers. Why? Because that's what an enterprise/mature build system is, it only differs from other software packages with the same capabilities in that it concentrates on build / deploy / CI "business tasks". My current employer uses Jenkins (which has workflows/pipelines, daemons) and then feeds into Spinnaker (which has a full DAG workflow engine and interface) and likely this is pretty close to a "standard" or "best of breed" cloud build CI system. Of course there is a dedicated team. Oh and of course the gradle code build in github has its own pipelines and general Turing machine to do whatever you want.
liminal 4y ago
Any suggestions for how to go from multiple Git repos to a monorepo? Preserving history would be really nice. I've looked at submodules and subtrees and both seem to have huge downsides and don't deliver the same benefits of a true monorepo.
- rkangel 4y ago
  Git as a system has no objection to having more than one 'initial commit'. It will happily take that branching history and merge it together. With a bit of branch renaming you can add extra remotes to your repo so both 'masters' are present. Commit to both to make the resulting directory structure not overlap, and then just merge. You'll end up with full history of both. I did a quick google and these instructions seem about right (without the delete step): https://gist.github.com/msrose/2feacb303035d11d2d05 https://gist.github.com/msrose/2feacb303035d11d2d05
- moojd 4y ago
  Yes! I once had to merge a dozen or so repos into a mono repo. I don't have my script handy but git allows merging repos with unrelated histories into one repo while preserving the history. If I remember correctly, this is how you do it: 1. Create a new empty repo for the monorepo 2. For each repo, 'git mv' all of the contents into a new directory with the repo's name 3. Add the repos to the mono repos as remotes 4. Run 'git merge --allow-unrelated-histories' for each repo You will now have a monorepo with preserved history with each old repo existing inside of a sub-directory in the new monorepo
- rubyist5eva 4y ago
  With Git I’m pretty sure you can literally just add a new remote to a completely unrelated repo, fetch it and then “merge” any branch from the other two into the new mono repo (ie. git checkout master; git merge other-remote/master). If all your projects are otherwise in a top level directory inside the monorepo this should merge cleanly and then can just live beside eachother in the checkout.
  - WorldMaker 4y ago
Naac 4y ago
I think its worth calling out that there are different types of monorepos. For example, I've worked in a monorepo that was one giant binary, but I've also worked in a monorepo that was a single repo that contained 4 ish independent services ( but were all in a single git repo ).
codenesium 4y ago
Having been down the route of repos for every service I would always choose monorepo in the future. I could see separate repos for libraries. There is just too much overhead trying to manage multiple repos. With a single repo it's possible to build a package that represents all of your software vs being forced to version everything. Tasks almost always touch multiple services unless you are so big you have a team per service.
- Yeroc 4y ago
  I don't agree with having libraries in their own repositories. For us one of the biggest costs to fixing bugs in libraries particularly was having to then update all dependent projects with the new version. By moving our libraries into a monorepo alongside all the consuming projects we got away from that busywork. It really streamlined things for us.
no_wizard 4y ago
I'm a big fan of monorepos. If they get too unwieldy or you need VCS granular permissions, you should use Perforce over git, but using either git or Perforce generally speaking I think works fine for monorepos. The tool has come such a long way from even 10 years ago, especially for front end codebases, but even for things like Rust the story is really strong. It comes down to how efficient you can be with tooling. Thats the one thing that monorepos really do require, is a good upfront investment in tooling, and long term maintenance. However I've found the initial setup "cost" of setting up a complex monorepo with correct tooling is far outweighed by the simplified operative overhead of working inside it.
benreesman 4y ago
Dan is diplomatic to a fault. Splitting repos on boundaries that aren’t necessary because of access control, legal obligation, or infrastructure constraint is for people who have nothing better to do. All the big shops have multiple repositories. They all broke each one out grudgingly and under some kind of pressure.
- exitheone 4y ago
  That's factually untrue. Google, Microsoft, Facebook, Twitter, Airbnb all have a huge monorepo. Splitting it out is obviously not necessary with the right tools.
  - williamsmj 4y ago
    No. Twitter absolutely has multiple repos. Some of them are very big, but they certainly have more than one. Why they have > 1 repo is a long story, but the post you're responding to is in the ballpark.
  - jeffbee 4y ago
    Google at least has separate repos for the linux kernel and android, and probably others (chrome/os/ium?). The hugeness of google3 is not in doubt, but the mono-ness may be.
  - disgruntledphd2 4y ago
    And FB have multiple, really, really large monorepos.
    - benreesman 4y ago
      So there was a (I hope flattering) joke behind Simpkins’ back that anyone of his broad-spectrum superiority, as he’s both a world-class hacker and conventionally handsome and consistently articulate that he must be carrying a “lizard”. His nickname was “the Iguana”. He spent a lot of time optimizing ‘hg’, and one assumes that the Iguana did what could be done.
  - 4y ago
jsnell 4y ago
(2015)-ish. Significant previous discussions: https://news.ycombinator.com/item?id=9562923 https://news.ycombinator.com/item?id=9562923 https://news.ycombinator.com/item?id=16362345 https://news.ycombinator.com/item?id=16362345
jkaptur 4y ago
> the downsides are already widely discussed. Does anyone have any useful pointers? I'm in such total agreement with the article that I actually don't know the counterarguments.
- hkt 4y ago
  Principle of least privilege springs to mind but I'm not familiar with the other issues.
- jeffbee 4y ago
  The main downside that people always mention is it takes a long time to clone or pull a large repo. This is actually a flaw of git, not a flaw of the monorepo as a concept.
  - yboris 4y ago
    I think you can clone just the last commit: > Provide an argument of -- depth 1 to the git clone command to copy only the latest revision of a repo: git clone -–depth [depth] [remote-url]
    - jeffbee 4y ago
      OK but it still takes a long time with millions of source files.
    - WorldMaker 4y ago
      Git has some strange behaviors with shallow clones (trying to manage volume using --depth). Shallow clones are great for CI builds, but not so great with working copies used actively by developers. At this point in git the better tool is called "sparse clones" (using the --sparse keyword and some other associated tools). A lot of interesting engineering work has been put into making "sparse clones" very performant on "conical sections" ("sparse cones") of a repository at a time. (In the way of checking out a single sub-directory of a monorepo and just its history type of thing.)
  - dundarious 4y ago
oceanplexian 4y ago
I know some of the FAANGs do monorepo (Google being the biggest) but AWS does not. A monorepo is an organizational mess when trying to manage and transfer ownership across thousands of teams, contain the blast radius of changes, unless you invest a ton of resources into proprietary tooling that requires a bunch of maintenance, since all the open source solutions are terrible at this and the whole data model is built around splitting out individual project repositories. And then after all that effort, why wouldn’t you just use tooling the way it was intended, and the way it’s used in the open source model, so you can partition your CI/CD without a bunch of hacks, and don’t run into bizarre scaling issues with your VCS. It perplexes me people advocate for this strategy. All I can think is it’s another one of those cargo-cult ideas that everyone is doing because Google did it (So it must be good).
- jvolkman 4y ago
  Let's not pretend that Amazon hasn't spent significant effort over decades building and maintaining their own (non-monorepo) build systems and tooling.
- radicality 4y ago
  * Why is ownership management a mess in a monorepo? Can just decide “this team owns this folder hierarchy” etc. * ‘Contain blast radius of changes’ - is that actually difficult? Isn’t there tooling that figures out what changed and what dependencies need rebuilding? (eg Facebook buck) For context, I was for a very long time at FB so am definitely used to the monorepo way, and recently switched to place which uses github + many repos, and it feels so much worse. Honest question - how do you actually effectively share code between many repos? Example: How do I know that me changing my backend app’s API doesn’t break any other project in the company potentially calling it? It should be a compile/buildtime error for the other project, but how does that work if everything is in its own little repository?
  - giaour 4y ago
    > Honest question - how do you actually effectively share code between many repos? Locally, you can use an Amazon-internal tool to check out multiple repos and make changes to all of them. The tooling calls this a "workspace," but it feels very much like working in a monorepo since building and testing can happen at the workspace level. > How do I know that me changing my backend app’s API doesn’t break any other project in the company potentially calling it? In terms of change management, Amazon dependency graphs are managed as "version sets." Changes have to be built into a given version set, and that build will also rebuild any package in the version set that consumes the repository whose changes are being built in. (Usually, repositories are configured to build into one of the owning team's version sets on each commit to the primary branch.)
bob1029 4y ago
We've been doing this for a few years now. Biggest non-intentional thing that came out of it was that the entire team started speaking in terms of commit hashes. Once a non-technical person learns that the entire state of a product/project/organization can be described by a hash, they will begin to abuse it for literally everything. And, I totally endorse this. Its incredible to watch unfold. An employee passively noting the current commit hash like its the time of day puts a bit of joy into my brain every time. Everyone can speak this language. The semantics are ridiculously simple.
- a9h74j 4y ago
  > Everyone can speak this language. The semantics are ridiculously simple. Similar to the engineering/BOM-oriented semantics of everything is a drawing with a matching part number?
- chubot 4y ago
  Hm can you give an example of that? Are they wondering if the features they care about are deployed? The linear version numbers could have an advantage in that regard. If you want to know if CL 12345 is deployed, and you know the current deployment is running as of CL 12350, then it should be in there. Conversely if it's less than that number, it's definitely not in there. git hashes also have good properties but I'm wondering how non-technical employees use them. Do they know how to dig through the git history?
denimnerd42 4y ago
git seems like the wrong tool for monorepos so what is used instead if you can't immediately just build your own tools
- tom_ 4y ago
  Perforce is ok. It has roughly the right model, it scales pretty well, and the tools are... good enough.
- WorldMaker 4y ago
  Microsoft and Twitter, at least, have very publicly invested a lot of engineering work in making git a tool for monorepos.
exfascist 4y ago
I'd argue that the optimal configuration is really a compromise; use sub modules with a dvcs tool like git. You get the organizational benefits of monorepos with the isolation benefits of individual repos. Your branches go stale in weeks rather than days, cloning even with full history can be very fast, and you don't need to learn new tools when you change organizations.
hardwaregeek 4y ago
I agree that monorepos are great if you're using version control systems in their current state. But I can't help but wonder if it's a question of monorepos being good, or version control/tooling inhibiting other options. If you had a VC tool that could compose repositories with ease, that could understand multiple histories and allow for atomic commits across repos, perhaps monorepos wouldn't be the best? Or you could keep the monorepo, but allow a "lens" into a specific subsection. Even with Dan's point about monorepos making tooling easier, if a VC tool had a good API, perhaps this point would be moot. Why is it hard to query files and repository dependencies? Should there be some way to model dependencies in your version control system? It'd be interesting to see someone tackle these problems in version control.
- throwaway894345 4y ago
  > I agree that monorepos are great if you're using version control systems in their current state. But I can't help but wonder if it's a question of monorepos being good, or version control/tooling inhibiting other options. Ultimately the problem is that we need tooling which is aware of dependencies, and the repo abstraction isn't. Whether that code lives in a single repo or in many repos is fairly irrelevant, but keeping the code in a single repo is usually a fair bit easier for many things (especially when you're working in a single language since the language's build tools are usually well-suited for this basic case) and you don't need to manually update dependency manifest files, test how a given upstream change affects every downstream package, or coordinate half a dozen PRs for every change.
  - WorldMaker 4y ago
    If you are using a dependency manager the repo abstraction, with multiple repos, does start to align as a useful node abstraction at the dependency graph level. For instance, if you are working in JS/TS, every repo has a top-level package.json file that is very easily consumed by tooling to discover dependencies. Github has a dependency graph that's pretty comprehensive for public packages as dependended on by public repos. For instance, the repos that depend on Typescript: https://github.com/microsoft/TypeScript/network/dependents?package_id=UGFja2FnZS01MTE3ODUxNjg%3D https://github.com/microsoft/TypeScript/network/dependents?p... There's often less tooling available for private repository hosts and private package feeds, but dependency management from a per-repo standpoint is if not a solved problem in practice, an easily solvable problem. (Github has some tools for private repos if you pay for them. Other systems can borrow from the same playbooks.) (Other languages have similar dependency manifest files, most of which are similarly slurpable by easily automated tooling given the need/chance. Dependency discovery doesn't have to be a problem in multi-repository environments.) > test how a given upstream change affects every downstream package, or coordinate half a dozen PRs for every change Some of this is push versus pull questions. One developer needing to push a lot of changes is a lot of work for that one developer. Downstream "owners" needing to pull changes at a frequency is in some cases much tinier slices of work spread out/delegated over a larger team of people, many of whom may be closer to downstream projects to better fix and/or troubleshoot secondary impacts. Monorepos make push easier, definitely. Sometimes pull is a better workflow. (Especially if you are using the same dependency tooling for third-party components. These days given CVEs and such you want a regular update cadence on third-party components anyway, using the same tools for first-party updates keeps more reason to keep that cadence regular. Lots of small changes over time rather than big upgrade processes all at once.)
captainmuon 4y ago
One upside of smaller repos that I rarely hear about is that it forces you to think about versioning. If you have a monorepo, you often don't version individual components, you just have master that always builds. If your product is a user facing website, that is fine. But if you make releases, and have multiple components in different versions that have a stable API, and are expected to work in different combinations, then it is a real hassle. Of course you can tag individual library versions in a monorepo, but that is not the way of least resistance. One place I've worked at migrated to a monorepo, the ATLAS experiment at CERN. It was not bad, although there were the usual problems with long checkout time. But it worked because we tended to version every single piece of software together in a big "release" anyway (to make scientific results reproducable).
- WorldMaker 4y ago
  Right, smaller repos add more friction to dependencies, that is certain, but flipside view of that is that it enforces API boundaries and thinking about systems building as SOLID components in their own right. That friction sometimes helps: If it is painful to update Dependency A because it usually means upstreaming changes to A's Dependency B first, for instance, that can often indicate a tight-coupling problem that in a class diagram someone might easily discover and refactor over lunch but in a systems diagram was non-obvious without that "update hell" pain. Solving such tight-coupling problems is hard, and it may mean living with the pain for some time, and while monorepos make that pain go away they never solve those coupling problems (and arguably make it far easier to strongly couple systems that you likely don't want coupled). It's a lot like turning off all the Warnings in your compiler; it makes the immediate dev experience a lot nicer, but it risks missing things that while not problems now may be problems in the future. I think there are also some benefits to using the same dependency managers for first-party components/libraries as for third-party components. The auto-updating of first-party versions is seen as a benefit to monorepos, but if recent and current CVEs have taught us anything you need to audit and update your third-party components quite regularly. Needing to also update first-party components/libraries with the same dependency managers has some benefits in terms of forcing a regular dependency update cadence, that then also benefits additional developer eyes on third party update rhythms. (Especially as increasingly more dependency managers pick up auto-auditing/security and CVE awareness tooling that runs on each update. There's more likely developer eyeballs on those audit reports if frequently run for first-party components and third-party components.) Dependency managers are their own friction in the process, but necessary friction for third-party components, and there are benefits to first-party components needing the same friction. As with most software development practices there is no objectively "right" answer here. Monorepos have less friction in a large org. Friction and pain are sometimes useful tools, despite few people "want" them in their developer experience. Systems design is hard and tight-coupling is often an easy solution. Looser coupling is often better, more resilient design that is easier to work with at the boundaries and the "I can trust this other team's repo to be a black box and they let me file bug reports as if they were a second-party vendor" level, which can be its own tool for avoidance of mental fatigue.
pbalau 4y ago
For multi repo I will need to build automation to manage all the repos and enforce a consistent experience across them, including syncing the repos, if we end up using stuff like submodules. And I need to do this now. We tried to "trust" every repo owner to do the right thing, but it was a cluster fuck. With monorepo, I had to set up things once and go on my merry way. And I will be able to kick the monorepo-is-too-slow-can down the road for a few years from now.
wjmao88 4y ago
Its Conway's Law, Your code organization is a reflection of your engineering teams organization. The number of repos you have should roughly be equal to how many autonomous engineering "groups" you can divide into that work largely independent of other groups. Anything a group touches should probably be in the same repo as everything else that the same group touches.
trollied 4y ago
> With a monorepo, projects can be organized and grouped together in whatever way you find to be most logically consistent, and not just because your version control system forces you to organize things in a particular way. Using a single repo also reduces overhead from managing dependencies. I don't actually understand this. You can do this with git submodules. It's just a directory structure. Can somebody please explain? If the problem is committing to multiple things at the same time for a point-in-time release, then the answer is tags. Rather than terabytes of git history for a gigantic organisation that has many unrelated projects. A good example for you: Google releases the Google Ad Manager API externally periodically, with dated releases. How does having that in a huge monorepo make sense?
- nickdandakis 4y ago
  Have you used git submodules before? I've only used them once and vowed to never use them again. It's effectively just a pointer to a hash, and ends up being useless for versioning + a really nice footgun for tracking upstream updates. The monorepo vs manyrepo tradeoff boils down to this: Do you want more complicated build + deploy tooling or do you want more complicated dependency management? If the former, pick monorepo. If the latter, pick manyrepo.
  - perrygeo 4y ago
    This is the best summary of the topic I've read. Having used all three (submodules, monorepo, manyrepo) the only thing I can say with any certainty is - don't use submodules. The mono/manyrepo decision is not as clear cut but your description nails it. Edit: submodules IS a viable solution for truly third-party repos over which you have no control and don't expect to ever edit.
    - slavik81 4y ago
      Have you ever tried contributing to Qt? I rather liked their use of submodules. https://github.com/qt/qt5 https://github.com/qt/qt5
akshayshah 4y ago
I hope that Amazon open-sources Brazil and the surrounding version set ecosystem someday. They're the only large company I know of that uses individual project repos at scale, and they've built tools that solve many of these problems. (I've never worked there, so I don't know how loved those tools are internally.) Edit: I worked at Microsoft, which also uses tons of tiny repos (at least within Azure). I didn't encounter any good cross-repo management tools, though; apart from having a Jira-like ticketing system built in, Azure DevOps seemed quite a bit worse than GitHub.
- qznc 4y ago
  At least, I would like to read more about it. A while ago, I collected some information [0] but I don't know it first-hand. One Amazon developer told me that Nix is the closest thing in the Open Source world. [0] http://beza1e1.tuxen.de/amazon_manyrepo_builds.html http://beza1e1.tuxen.de/amazon_manyrepo_builds.html
paulvnickerson 4y ago
How do you address the blast radius problem with monorepos? For instance, I want to have a single gitlab repo for postgresql clusters. Using jsonnet, I deploy and configure a cluster for each customer, and adding a new cluster is as easy as adding a config file. However, my colleague explained that it's a bad idea because any config changes or accidental button presses on gitlab's ci/cd page can bring down or wipe out everybody's cluster. How can that problem be mitigated? It seems intrinsic to monorepo style.
- pbalau 4y ago
  Not sure why you got downvoted. The problem is with your deploy system. You can consider each of the clusters to be a service. Thus, a change in Service A (cluster A), should not trigger a deployment of Service B (cluster B). My pipeline is split in 2: 1. on bitbucket, we run a pipeline that builds "build artefacts", docker images and "packaged" cloudformation templates. Each of these artefacts has a list of triggers, either base docker images or source code. I'm building the relevant docker image or cf package based on the triggers (it's quite a naïve glob() use). 2. On aws side, I have something I call AWS Apps, in short a Stack Name, along with a set of triggers (the above build artefacts). On merge to main, I only deploy the AWS Apps affected by new build artefacts.
Thaxll 4y ago
How do you manage versions / tags with monorepo? If you need to tag something ( a lib ) everyone gets the same, the entire repo now has a tag v0.0.1 eventhough only your library changed.
- jvolkman 4y ago
  At least at Google: you don't. There is just one current version of your library. There are exceptions, but for the vast majority of things there is no concept of versioning.
- jefftk 4y ago
  I think the most common option is you don't. The organizations that Dan gives as examples mostly don't produce public facing libraries, and when they do it's a separate process that lives on GitHub. Instead, everything is based around the idea that you check out the state of the world at some commit, do your build, whatever validation you need, and send it to production. You do this pretty often, ideally multiple times a week. Very occasionally you have an emergency where you want prod + cherrypick, and you generally build tooling that allows saying "build at this commit, but also with these later specific commits merged in".
- pvarangot 4y ago
  I worked on a team that was doing embedded software inside a monorepo with Linux and cloud stuff, so we needed to version our stuff because we were not doing "continuous deployment" flashing the uCs all the time a change on master happened. We just had big "feature branches" and those got rebased and merged weekly. For the cloud stuff, there was just one version and that version was what other team was striving to continuously deploy into staging and then production. It's not ideal but it was handy to have access to all the cloud and application code on the embedded side for stuff like interface definitions for communication protocols and stuff like that. On the same company I worked on another project where the definition files for the cloud interface where on a different repo we had to use a submodule and I preferred the monorepo. As other commenter said, we used bazel and there was indeed a smallish team that gave build support. Ramping up new hires on the build system was one of the more painful processes, I had to give support myself to teammates that had like a six month tenure only that they hadn't been there from day one.
trasz 4y ago
Monorepo is one of the features I really like in FreeBSD. It makes adding functionality that goes across layers - eg adding a syscall implementation, its manual page, libc stub, and making use of it in some userspace component - trivial, compared to the hurdles necessary in the Linux world, where you'd need to interact with kernel folks, libc folks, some random userspace project folks, and then wait until it goes into distributions.
honkycat 4y ago
The thing about monorepos is similar to the the thing about micro-services: they require a lot of tooling and discipline and documentation that most organizations do not have. On our multi-repos I have consistently seen dozens, if not hundreds, of stale pull requests and branches and issues piling up never to be merged. This compounds with a monorepo. Additionally, how do you avoid doing pointless builds when new features are pushed? I can only imagine what the `.github` folder in a monorepo looks like. For me it is similar to the "one large file" argument, and why I don't agree: obfuscation is bad, but information hiding is GOOD. When I open a file, I want the information relevant to the current domain I am working in, not all of the information all at once. Similarly, when I open a github page, I want its issues, pull requests, branches, and wiki to represent the state of a single project. The one I am currently interested in. You lose this with a monorepo. You can argue "well tooling can..." yes tooling that does not exist and that I do not want to implement. Similar to the "one large file" argument, editors are set up to manage many different files with tabs. You COULD just compile the code and navigate symbols, but that isn't the world we currently live in.
- Orphis 4y ago
  > Additionally, how do you avoid doing pointless builds when new features are pushed? I can only imagine what the `.github` folder in a monorepo looks like. It's simple, with proper tooling, you know exactly the dependencies, so you know which test depend on the affected files and can run those tests, the rest shouldn't be impacted. And that tooling exists. It's not the one you may be using, but it exists, and not just in FAANG.
deleted 4y ago
[deleted]
atx42 4y ago
Our team is unique at our company, having a "monorepo" with 9 components versus the standard 1 component / 1 repo that other teams use. With maven, we can use one command to build any one or all components. If we split, we'd tell Jenkins how to build everything, but would say goodbye to simple local builds. Without introducing some more technology or complexity and likely specifying how the build works in two different places, I didn't see a good solution to this. I mention this here, as maybe I'm missing some obvious solution.
MichaelMoser123 4y ago
one problem with multiple repos: you may end up with multiple binary components, like shared libraries, static libraries, etc, where each binary is produced from the sources of a separate repository. Now it may turn out to be a bit tricky to track a given binary found in a deployment to its sources. (on the JVM you could partially get by without the sources, as you have good decompilers) I have never worked with mono repos, but I guess that this task would be somewhat easier, given that all sources are under a single repository.
Maksadbek 4y ago
We use git with monorepos. The codebase is so large that git status command takes about 3-6 secs. Do you also use git with monorepos ?
lliamander 4y ago
All these advantages really come down to making it easier to manage tightly coupled systems. That's great that the monorepo approach used by large tech companies with whole departments devoted to developer tooling can make that work. However, I think the "polyrepo" response to most of these advantages would be to focus on decoupling your systems instead. Take for instance: > With a monorepo, you just refactor the API and all of its callers in one commit. That's not always trivial, but it's much easier than it would be with lots of small repos. I've seen APIs with thousands of usages across hundreds of projects get refactored and with a monorepo setup it's so easy that it's no one even thinks twice. Like, that's really cool you can do that. But why are doing that?! Why are you breaking your API contract and forcing all of your clients to change all at once? Of course, proper decoupling also requires good engineering. A polyrepo environment can still get horribly tangled, but the natural response to all of these tangling problems in a polyrepo is to move in a direction of looser coupling.
- jvolkman 4y ago
  What is proper decoupling? In a properly-tooled monorepo, project A can't take a dependency on project B unless B is a public library or explicitly gives access to A [1]. Authors have full control over coupling. Sure, avoid changing the API contract. But when the time comes to change the API, you can 1) make the change backwards compatible and maintain both methods forever; 2) release a new major version and maintain both versions forever; or 3) just migrate the callers and immediately be free of all technical debt that would've been accrued in 1 and 2. This assumes internal clients whom you presumably can't break. 1: https://bazel.build/concepts/visibility https://bazel.build/concepts/visibility
  - lliamander 4y ago
    Visibility is somewhat orthogonal. I mean yes, you can avoid coupling issues by preventing the dependency altogether, but really the interesting problems occur once you do have a dependency. Once you have the dependency, the loosely coupled approach means that (when possible) you avoid making changes to your API contract that would break your clients. I see the appeal of approach #3 that you suggest, but here's the problem I see with that (and maybe you have an answer): For any change that breaks the contract (outside of trivial things like renaming an identifier) you are necessarily either adding a requirement your clients may not be able to satisfy, or removing a capability they depend upon. Migrating your clients in that case is more than just a simple refactor; the client may need to re-architect so that it can adapt to the change in contract or even move to a new dependency altogether. If you're not the owner of that client, that means you are either interrupting the other team while they are forced to help you with the migration, or you are blocked waiting for them to have the time. In general, I would say the best approach to making breaking changes to an API is to use a deprecation process. That allows clients to migrate at their own pace. You can of course do that in either a monorepo or a polyrepo approach, but my expectation would be that the monorepo doesn't really provide you with any advantages in that case.
switch33 4y ago
Large repos make sense or don't make sense based on companies that work with large data or not based on predicate calculus and derivatives usually dealing with repos as well as stories and have more problems with ssds too. There is lots of problems associated with ssds as well as large monorepos. There are more complicated than people realize but if you did google code jam it teaches them somewhat but needs to be explained too. There problem is stories sort of intersect with programming too. Clockwork with ssds needs to be reworked for google code jams. The problem is elixir sort of works with stories and programming. Predicate calculus and proof theories sort of are the only way programming will really make sense in a world full of ssds. Leveldb could be a more interesting problem for google code jams if it has some newer features too. Conflict resolution is tower of hanoi and that has problems with consensus algorithms and concat too.SSDs need to do derivatives for pieceing and parting software too and that is more interesting too.
dqpb 4y ago
Use a monorepo, but organize your code as if it will someday be split into many repos.

183 points by Naac 4y ago

Monorepos are also great for small monorepos with just a few projects. The darklang monorepo [1] has a devcontainer that installs all the build tools for 4 projects which create 21 different services, using 6 languages, and building everything is one step. In fact, it makes it so easy to add new stuff that I didn't even realize we had 21 services til I counted. My first guess was 12. [1] https://github.com/darklang/dark https://github.com/darklang/dark

88913527 4y ago

In my experience, the developer experience for juniors is too much. Yarn + Lerna is just too much of a learning curve. However, having one repo and on CICD pipeline is convenient. But we've decided to divest from them. Your situation may not match mine, and that's okay.

pcmaffey 4y ago
Lerna is only needed if you're publishing multiple packages from the monorepo. If you're consuming your packages only within the monorepo for your various services, Yarn Workspaces is generally all that's needed.
dimgl 4y ago
Just don't use Yarn + Lerna :) pnpm is amazing
- 88913527 4y ago
  Doesn't work older Angular projects. My monorepo is a hodge podge of technologies.

lisper 4y ago

It's very simple: with a monorepo you always have access to everything you need, together with a ton of stuff you don't. Whether or not this is advantageous boils down to whether the cost of not having access to something you need is greater than the cost of having access to a bunch of stuff you don't. As long as your system is reasonably efficient at letting you select small subsets of everything you could potentially have access to, the cost of having access to a bunch of stuff you don't need is essentially zero. Perforce is good at that. Git isn't. So people who use Perforce tend to think that monorepos are good and people who use git don't. And they're both right.

mikepurvis 4y ago
Years ago, Google had a gcheckout tool that would trace the dependency information for whatever project you were working on, and then selectively grab the portions of the monorepo that you were going to need for it. Maybe they still have that or it's evolved into something else; I dunno, I haven't been there in a really long time. Anyway, it seemed like such an obvious complement to the the perforce/monorepo style of working that I came away surprised that perforce wouldn't have hoisted such a thing into their product as a first-class feature. Tracing dependencies across a lot of different build systems is obviously not trivial, but it's not intractable, particularly if the tool is pluggable so that orgs can provide modules to handle their own particular approach.
- jeffbee 4y ago
  What you are describing is an artifact of the old Perforce which copied everything in your client to local storage. After the conversion to srcfs and piper, which was more than a decade ago, this became unnecessary. http://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-source-code.html?m=1 http://google-engtools.blogspot.com/2011/06/build-in-cloud-a...
throwaway894345 4y ago
Honestly I would 100% do a monorepo every single time if there was good tooling for incrementally building and testing libraries. Having to rebuild every image from scratch for every single change scales miserably. Things like Bazel exist, but you basically have to have a team dedicated to operating it (maybe the difficulty varies by language, but it was a major pain when I tried to use it to build some relatively simple Python projects a few years ago).
- AtlasBarfed 4y ago
  Is there, say, IntelliJ support for Bazel? Do you need a central server? I've heard bazel is a bear... But... all mature build systems are, because they become essentially enterprise workflow engines, process execution engines, internal systems integration hubs, and schedulers. Why? Because that's what an enterprise/mature build system is, it only differs from other software packages with the same capabilities in that it concentrates on build / deploy / CI "business tasks". My current employer uses Jenkins (which has workflows/pipelines, daemons) and then feeds into Spinnaker (which has a full DAG workflow engine and interface) and likely this is pretty close to a "standard" or "best of breed" cloud build CI system. Of course there is a dedicated team. Oh and of course the gradle code build in github has its own pipelines and general Turing machine to do whatever you want.

liminal 4y ago

Any suggestions for how to go from multiple Git repos to a monorepo? Preserving history would be really nice. I've looked at submodules and subtrees and both seem to have huge downsides and don't deliver the same benefits of a true monorepo.

rkangel 4y ago
Git as a system has no objection to having more than one 'initial commit'. It will happily take that branching history and merge it together. With a bit of branch renaming you can add extra remotes to your repo so both 'masters' are present. Commit to both to make the resulting directory structure not overlap, and then just merge. You'll end up with full history of both. I did a quick google and these instructions seem about right (without the delete step): https://gist.github.com/msrose/2feacb303035d11d2d05 https://gist.github.com/msrose/2feacb303035d11d2d05
moojd 4y ago
Yes! I once had to merge a dozen or so repos into a mono repo. I don't have my script handy but git allows merging repos with unrelated histories into one repo while preserving the history. If I remember correctly, this is how you do it: 1. Create a new empty repo for the monorepo 2. For each repo, 'git mv' all of the contents into a new directory with the repo's name 3. Add the repos to the mono repos as remotes 4. Run 'git merge --allow-unrelated-histories' for each repo You will now have a monorepo with preserved history with each old repo existing inside of a sub-directory in the new monorepo
rubyist5eva 4y ago
With Git I’m pretty sure you can literally just add a new remote to a completely unrelated repo, fetch it and then “merge” any branch from the other two into the new mono repo (ie. git checkout master; git merge other-remote/master). If all your projects are otherwise in a top level directory inside the monorepo this should merge cleanly and then can just live beside eachother in the checkout.
- WorldMaker 4y ago

Naac 4y ago

I think its worth calling out that there are different types of monorepos. For example, I've worked in a monorepo that was one giant binary, but I've also worked in a monorepo that was a single repo that contained 4 ish independent services ( but were all in a single git repo ).

codenesium 4y ago

Having been down the route of repos for every service I would always choose monorepo in the future. I could see separate repos for libraries. There is just too much overhead trying to manage multiple repos. With a single repo it's possible to build a package that represents all of your software vs being forced to version everything. Tasks almost always touch multiple services unless you are so big you have a team per service.

Yeroc 4y ago
I don't agree with having libraries in their own repositories. For us one of the biggest costs to fixing bugs in libraries particularly was having to then update all dependent projects with the new version. By moving our libraries into a monorepo alongside all the consuming projects we got away from that busywork. It really streamlined things for us.

no_wizard 4y ago

I'm a big fan of monorepos. If they get too unwieldy or you need VCS granular permissions, you should use Perforce over git, but using either git or Perforce generally speaking I think works fine for monorepos. The tool has come such a long way from even 10 years ago, especially for front end codebases, but even for things like Rust the story is really strong. It comes down to how efficient you can be with tooling. Thats the one thing that monorepos really do require, is a good upfront investment in tooling, and long term maintenance. However I've found the initial setup "cost" of setting up a complex monorepo with correct tooling is far outweighed by the simplified operative overhead of working inside it.

benreesman 4y ago

Dan is diplomatic to a fault. Splitting repos on boundaries that aren’t necessary because of access control, legal obligation, or infrastructure constraint is for people who have nothing better to do. All the big shops have multiple repositories. They all broke each one out grudgingly and under some kind of pressure.

exitheone 4y ago
That's factually untrue. Google, Microsoft, Facebook, Twitter, Airbnb all have a huge monorepo. Splitting it out is obviously not necessary with the right tools.
- williamsmj 4y ago
  No. Twitter absolutely has multiple repos. Some of them are very big, but they certainly have more than one. Why they have > 1 repo is a long story, but the post you're responding to is in the ballpark.
- jeffbee 4y ago
  Google at least has separate repos for the linux kernel and android, and probably others (chrome/os/ium?). The hugeness of google3 is not in doubt, but the mono-ness may be.
- disgruntledphd2 4y ago
  And FB have multiple, really, really large monorepos.
  - benreesman 4y ago
    So there was a (I hope flattering) joke behind Simpkins’ back that anyone of his broad-spectrum superiority, as he’s both a world-class hacker and conventionally handsome and consistently articulate that he must be carrying a “lizard”. His nickname was “the Iguana”. He spent a lot of time optimizing ‘hg’, and one assumes that the Iguana did what could be done.
- 4y ago

jsnell 4y ago

(2015)-ish. Significant previous discussions: https://news.ycombinator.com/item?id=9562923 https://news.ycombinator.com/item?id=9562923 https://news.ycombinator.com/item?id=16362345 https://news.ycombinator.com/item?id=16362345

jkaptur 4y ago

> the downsides are already widely discussed. Does anyone have any useful pointers? I'm in such total agreement with the article that I actually don't know the counterarguments.

hkt 4y ago
Principle of least privilege springs to mind but I'm not familiar with the other issues.
jeffbee 4y ago
The main downside that people always mention is it takes a long time to clone or pull a large repo. This is actually a flaw of git, not a flaw of the monorepo as a concept.
- yboris 4y ago
  I think you can clone just the last commit: > Provide an argument of -- depth 1 to the git clone command to copy only the latest revision of a repo: git clone -–depth [depth] [remote-url]
  - jeffbee 4y ago
    OK but it still takes a long time with millions of source files.
  - WorldMaker 4y ago
    Git has some strange behaviors with shallow clones (trying to manage volume using --depth). Shallow clones are great for CI builds, but not so great with working copies used actively by developers. At this point in git the better tool is called "sparse clones" (using the --sparse keyword and some other associated tools). A lot of interesting engineering work has been put into making "sparse clones" very performant on "conical sections" ("sparse cones") of a repository at a time. (In the way of checking out a single sub-directory of a monorepo and just its history type of thing.)
- dundarious 4y ago

oceanplexian 4y ago

I know some of the FAANGs do monorepo (Google being the biggest) but AWS does not. A monorepo is an organizational mess when trying to manage and transfer ownership across thousands of teams, contain the blast radius of changes, unless you invest a ton of resources into proprietary tooling that requires a bunch of maintenance, since all the open source solutions are terrible at this and the whole data model is built around splitting out individual project repositories. And then after all that effort, why wouldn’t you just use tooling the way it was intended, and the way it’s used in the open source model, so you can partition your CI/CD without a bunch of hacks, and don’t run into bizarre scaling issues with your VCS. It perplexes me people advocate for this strategy. All I can think is it’s another one of those cargo-cult ideas that everyone is doing because Google did it (So it must be good).

jvolkman 4y ago
Let's not pretend that Amazon hasn't spent significant effort over decades building and maintaining their own (non-monorepo) build systems and tooling.
radicality 4y ago
* Why is ownership management a mess in a monorepo? Can just decide “this team owns this folder hierarchy” etc. * ‘Contain blast radius of changes’ - is that actually difficult? Isn’t there tooling that figures out what changed and what dependencies need rebuilding? (eg Facebook buck) For context, I was for a very long time at FB so am definitely used to the monorepo way, and recently switched to place which uses github + many repos, and it feels so much worse. Honest question - how do you actually effectively share code between many repos? Example: How do I know that me changing my backend app’s API doesn’t break any other project in the company potentially calling it? It should be a compile/buildtime error for the other project, but how does that work if everything is in its own little repository?
- giaour 4y ago
  > Honest question - how do you actually effectively share code between many repos? Locally, you can use an Amazon-internal tool to check out multiple repos and make changes to all of them. The tooling calls this a "workspace," but it feels very much like working in a monorepo since building and testing can happen at the workspace level. > How do I know that me changing my backend app’s API doesn’t break any other project in the company potentially calling it? In terms of change management, Amazon dependency graphs are managed as "version sets." Changes have to be built into a given version set, and that build will also rebuild any package in the version set that consumes the repository whose changes are being built in. (Usually, repositories are configured to build into one of the owning team's version sets on each commit to the primary branch.)

bob1029 4y ago

We've been doing this for a few years now. Biggest non-intentional thing that came out of it was that the entire team started speaking in terms of commit hashes. Once a non-technical person learns that the entire state of a product/project/organization can be described by a hash, they will begin to abuse it for literally everything. And, I totally endorse this. Its incredible to watch unfold. An employee passively noting the current commit hash like its the time of day puts a bit of joy into my brain every time. Everyone can speak this language. The semantics are ridiculously simple.

a9h74j 4y ago
> Everyone can speak this language. The semantics are ridiculously simple. Similar to the engineering/BOM-oriented semantics of everything is a drawing with a matching part number?
chubot 4y ago
Hm can you give an example of that? Are they wondering if the features they care about are deployed? The linear version numbers could have an advantage in that regard. If you want to know if CL 12345 is deployed, and you know the current deployment is running as of CL 12350, then it should be in there. Conversely if it's less than that number, it's definitely not in there. git hashes also have good properties but I'm wondering how non-technical employees use them. Do they know how to dig through the git history?

denimnerd42 4y ago

git seems like the wrong tool for monorepos so what is used instead if you can't immediately just build your own tools

tom_ 4y ago
Perforce is ok. It has roughly the right model, it scales pretty well, and the tools are... good enough.
WorldMaker 4y ago
Microsoft and Twitter, at least, have very publicly invested a lot of engineering work in making git a tool for monorepos.

exfascist 4y ago

I'd argue that the optimal configuration is really a compromise; use sub modules with a dvcs tool like git. You get the organizational benefits of monorepos with the isolation benefits of individual repos. Your branches go stale in weeks rather than days, cloning even with full history can be very fast, and you don't need to learn new tools when you change organizations.

hardwaregeek 4y ago

I agree that monorepos are great if you're using version control systems in their current state. But I can't help but wonder if it's a question of monorepos being good, or version control/tooling inhibiting other options. If you had a VC tool that could compose repositories with ease, that could understand multiple histories and allow for atomic commits across repos, perhaps monorepos wouldn't be the best? Or you could keep the monorepo, but allow a "lens" into a specific subsection. Even with Dan's point about monorepos making tooling easier, if a VC tool had a good API, perhaps this point would be moot. Why is it hard to query files and repository dependencies? Should there be some way to model dependencies in your version control system? It'd be interesting to see someone tackle these problems in version control.

throwaway894345 4y ago
> I agree that monorepos are great if you're using version control systems in their current state. But I can't help but wonder if it's a question of monorepos being good, or version control/tooling inhibiting other options. Ultimately the problem is that we need tooling which is aware of dependencies, and the repo abstraction isn't. Whether that code lives in a single repo or in many repos is fairly irrelevant, but keeping the code in a single repo is usually a fair bit easier for many things (especially when you're working in a single language since the language's build tools are usually well-suited for this basic case) and you don't need to manually update dependency manifest files, test how a given upstream change affects every downstream package, or coordinate half a dozen PRs for every change.
- WorldMaker 4y ago
  If you are using a dependency manager the repo abstraction, with multiple repos, does start to align as a useful node abstraction at the dependency graph level. For instance, if you are working in JS/TS, every repo has a top-level package.json file that is very easily consumed by tooling to discover dependencies. Github has a dependency graph that's pretty comprehensive for public packages as dependended on by public repos. For instance, the repos that depend on Typescript: https://github.com/microsoft/TypeScript/network/dependents?package_id=UGFja2FnZS01MTE3ODUxNjg%3D https://github.com/microsoft/TypeScript/network/dependents?p... There's often less tooling available for private repository hosts and private package feeds, but dependency management from a per-repo standpoint is if not a solved problem in practice, an easily solvable problem. (Github has some tools for private repos if you pay for them. Other systems can borrow from the same playbooks.) (Other languages have similar dependency manifest files, most of which are similarly slurpable by easily automated tooling given the need/chance. Dependency discovery doesn't have to be a problem in multi-repository environments.) > test how a given upstream change affects every downstream package, or coordinate half a dozen PRs for every change Some of this is push versus pull questions. One developer needing to push a lot of changes is a lot of work for that one developer. Downstream "owners" needing to pull changes at a frequency is in some cases much tinier slices of work spread out/delegated over a larger team of people, many of whom may be closer to downstream projects to better fix and/or troubleshoot secondary impacts. Monorepos make push easier, definitely. Sometimes pull is a better workflow. (Especially if you are using the same dependency tooling for third-party components. These days given CVEs and such you want a regular update cadence on third-party components anyway, using the same tools for first-party updates keeps more reason to keep that cadence regular. Lots of small changes over time rather than big upgrade processes all at once.)

captainmuon 4y ago

One upside of smaller repos that I rarely hear about is that it forces you to think about versioning. If you have a monorepo, you often don't version individual components, you just have master that always builds. If your product is a user facing website, that is fine. But if you make releases, and have multiple components in different versions that have a stable API, and are expected to work in different combinations, then it is a real hassle. Of course you can tag individual library versions in a monorepo, but that is not the way of least resistance. One place I've worked at migrated to a monorepo, the ATLAS experiment at CERN. It was not bad, although there were the usual problems with long checkout time. But it worked because we tended to version every single piece of software together in a big "release" anyway (to make scientific results reproducable).

WorldMaker 4y ago
Right, smaller repos add more friction to dependencies, that is certain, but flipside view of that is that it enforces API boundaries and thinking about systems building as SOLID components in their own right. That friction sometimes helps: If it is painful to update Dependency A because it usually means upstreaming changes to A's Dependency B first, for instance, that can often indicate a tight-coupling problem that in a class diagram someone might easily discover and refactor over lunch but in a systems diagram was non-obvious without that "update hell" pain. Solving such tight-coupling problems is hard, and it may mean living with the pain for some time, and while monorepos make that pain go away they never solve those coupling problems (and arguably make it far easier to strongly couple systems that you likely don't want coupled). It's a lot like turning off all the Warnings in your compiler; it makes the immediate dev experience a lot nicer, but it risks missing things that while not problems now may be problems in the future. I think there are also some benefits to using the same dependency managers for first-party components/libraries as for third-party components. The auto-updating of first-party versions is seen as a benefit to monorepos, but if recent and current CVEs have taught us anything you need to audit and update your third-party components quite regularly. Needing to also update first-party components/libraries with the same dependency managers has some benefits in terms of forcing a regular dependency update cadence, that then also benefits additional developer eyes on third party update rhythms. (Especially as increasingly more dependency managers pick up auto-auditing/security and CVE awareness tooling that runs on each update. There's more likely developer eyeballs on those audit reports if frequently run for first-party components and third-party components.) Dependency managers are their own friction in the process, but necessary friction for third-party components, and there are benefits to first-party components needing the same friction. As with most software development practices there is no objectively "right" answer here. Monorepos have less friction in a large org. Friction and pain are sometimes useful tools, despite few people "want" them in their developer experience. Systems design is hard and tight-coupling is often an easy solution. Looser coupling is often better, more resilient design that is easier to work with at the boundaries and the "I can trust this other team's repo to be a black box and they let me file bug reports as if they were a second-party vendor" level, which can be its own tool for avoidance of mental fatigue.

pbalau 4y ago

For multi repo I will need to build automation to manage all the repos and enforce a consistent experience across them, including syncing the repos, if we end up using stuff like submodules. And I need to do this now. We tried to "trust" every repo owner to do the right thing, but it was a cluster fuck. With monorepo, I had to set up things once and go on my merry way. And I will be able to kick the monorepo-is-too-slow-can down the road for a few years from now.

wjmao88 4y ago

Its Conway's Law, Your code organization is a reflection of your engineering teams organization. The number of repos you have should roughly be equal to how many autonomous engineering "groups" you can divide into that work largely independent of other groups. Anything a group touches should probably be in the same repo as everything else that the same group touches.

trollied 4y ago

> With a monorepo, projects can be organized and grouped together in whatever way you find to be most logically consistent, and not just because your version control system forces you to organize things in a particular way. Using a single repo also reduces overhead from managing dependencies. I don't actually understand this. You can do this with git submodules. It's just a directory structure. Can somebody please explain? If the problem is committing to multiple things at the same time for a point-in-time release, then the answer is tags. Rather than terabytes of git history for a gigantic organisation that has many unrelated projects. A good example for you: Google releases the Google Ad Manager API externally periodically, with dated releases. How does having that in a huge monorepo make sense?

nickdandakis 4y ago
Have you used git submodules before? I've only used them once and vowed to never use them again. It's effectively just a pointer to a hash, and ends up being useless for versioning + a really nice footgun for tracking upstream updates. The monorepo vs manyrepo tradeoff boils down to this: Do you want more complicated build + deploy tooling or do you want more complicated dependency management? If the former, pick monorepo. If the latter, pick manyrepo.
- perrygeo 4y ago
  This is the best summary of the topic I've read. Having used all three (submodules, monorepo, manyrepo) the only thing I can say with any certainty is - don't use submodules. The mono/manyrepo decision is not as clear cut but your description nails it. Edit: submodules IS a viable solution for truly third-party repos over which you have no control and don't expect to ever edit.
  - slavik81 4y ago
    Have you ever tried contributing to Qt? I rather liked their use of submodules. https://github.com/qt/qt5 https://github.com/qt/qt5

akshayshah 4y ago

I hope that Amazon open-sources Brazil and the surrounding version set ecosystem someday. They're the only large company I know of that uses individual project repos at scale, and they've built tools that solve many of these problems. (I've never worked there, so I don't know how loved those tools are internally.) Edit: I worked at Microsoft, which also uses tons of tiny repos (at least within Azure). I didn't encounter any good cross-repo management tools, though; apart from having a Jira-like ticketing system built in, Azure DevOps seemed quite a bit worse than GitHub.

qznc 4y ago
At least, I would like to read more about it. A while ago, I collected some information [0] but I don't know it first-hand. One Amazon developer told me that Nix is the closest thing in the Open Source world. [0] http://beza1e1.tuxen.de/amazon_manyrepo_builds.html http://beza1e1.tuxen.de/amazon_manyrepo_builds.html

paulvnickerson 4y ago

How do you address the blast radius problem with monorepos? For instance, I want to have a single gitlab repo for postgresql clusters. Using jsonnet, I deploy and configure a cluster for each customer, and adding a new cluster is as easy as adding a config file. However, my colleague explained that it's a bad idea because any config changes or accidental button presses on gitlab's ci/cd page can bring down or wipe out everybody's cluster. How can that problem be mitigated? It seems intrinsic to monorepo style.

pbalau 4y ago
Not sure why you got downvoted. The problem is with your deploy system. You can consider each of the clusters to be a service. Thus, a change in Service A (cluster A), should not trigger a deployment of Service B (cluster B). My pipeline is split in 2: 1. on bitbucket, we run a pipeline that builds "build artefacts", docker images and "packaged" cloudformation templates. Each of these artefacts has a list of triggers, either base docker images or source code. I'm building the relevant docker image or cf package based on the triggers (it's quite a naïve glob() use). 2. On aws side, I have something I call AWS Apps, in short a Stack Name, along with a set of triggers (the above build artefacts). On merge to main, I only deploy the AWS Apps affected by new build artefacts.

Thaxll 4y ago

How do you manage versions / tags with monorepo? If you need to tag something ( a lib ) everyone gets the same, the entire repo now has a tag v0.0.1 eventhough only your library changed.

jvolkman 4y ago
At least at Google: you don't. There is just one current version of your library. There are exceptions, but for the vast majority of things there is no concept of versioning.
jefftk 4y ago
I think the most common option is you don't. The organizations that Dan gives as examples mostly don't produce public facing libraries, and when they do it's a separate process that lives on GitHub. Instead, everything is based around the idea that you check out the state of the world at some commit, do your build, whatever validation you need, and send it to production. You do this pretty often, ideally multiple times a week. Very occasionally you have an emergency where you want prod + cherrypick, and you generally build tooling that allows saying "build at this commit, but also with these later specific commits merged in".
pvarangot 4y ago
I worked on a team that was doing embedded software inside a monorepo with Linux and cloud stuff, so we needed to version our stuff because we were not doing "continuous deployment" flashing the uCs all the time a change on master happened. We just had big "feature branches" and those got rebased and merged weekly. For the cloud stuff, there was just one version and that version was what other team was striving to continuously deploy into staging and then production. It's not ideal but it was handy to have access to all the cloud and application code on the embedded side for stuff like interface definitions for communication protocols and stuff like that. On the same company I worked on another project where the definition files for the cloud interface where on a different repo we had to use a submodule and I preferred the monorepo. As other commenter said, we used bazel and there was indeed a smallish team that gave build support. Ramping up new hires on the build system was one of the more painful processes, I had to give support myself to teammates that had like a six month tenure only that they hadn't been there from day one.

trasz 4y ago

Monorepo is one of the features I really like in FreeBSD. It makes adding functionality that goes across layers - eg adding a syscall implementation, its manual page, libc stub, and making use of it in some userspace component - trivial, compared to the hurdles necessary in the Linux world, where you'd need to interact with kernel folks, libc folks, some random userspace project folks, and then wait until it goes into distributions.

honkycat 4y ago

The thing about monorepos is similar to the the thing about micro-services: they require a lot of tooling and discipline and documentation that most organizations do not have. On our multi-repos I have consistently seen dozens, if not hundreds, of stale pull requests and branches and issues piling up never to be merged. This compounds with a monorepo. Additionally, how do you avoid doing pointless builds when new features are pushed? I can only imagine what the `.github` folder in a monorepo looks like. For me it is similar to the "one large file" argument, and why I don't agree: obfuscation is bad, but information hiding is GOOD. When I open a file, I want the information relevant to the current domain I am working in, not all of the information all at once. Similarly, when I open a github page, I want its issues, pull requests, branches, and wiki to represent the state of a single project. The one I am currently interested in. You lose this with a monorepo. You can argue "well tooling can..." yes tooling that does not exist and that I do not want to implement. Similar to the "one large file" argument, editors are set up to manage many different files with tabs. You COULD just compile the code and navigate symbols, but that isn't the world we currently live in.

Orphis 4y ago
> Additionally, how do you avoid doing pointless builds when new features are pushed? I can only imagine what the `.github` folder in a monorepo looks like. It's simple, with proper tooling, you know exactly the dependencies, so you know which test depend on the affected files and can run those tests, the rest shouldn't be impacted. And that tooling exists. It's not the one you may be using, but it exists, and not just in FAANG.

deleted 4y ago

[deleted]

atx42 4y ago

Our team is unique at our company, having a "monorepo" with 9 components versus the standard 1 component / 1 repo that other teams use. With maven, we can use one command to build any one or all components. If we split, we'd tell Jenkins how to build everything, but would say goodbye to simple local builds. Without introducing some more technology or complexity and likely specifying how the build works in two different places, I didn't see a good solution to this. I mention this here, as maybe I'm missing some obvious solution.

MichaelMoser123 4y ago

one problem with multiple repos: you may end up with multiple binary components, like shared libraries, static libraries, etc, where each binary is produced from the sources of a separate repository. Now it may turn out to be a bit tricky to track a given binary found in a deployment to its sources. (on the JVM you could partially get by without the sources, as you have good decompilers) I have never worked with mono repos, but I guess that this task would be somewhat easier, given that all sources are under a single repository.

Maksadbek 4y ago

We use git with monorepos. The codebase is so large that git status command takes about 3-6 secs. Do you also use git with monorepos ?

lliamander 4y ago

All these advantages really come down to making it easier to manage tightly coupled systems. That's great that the monorepo approach used by large tech companies with whole departments devoted to developer tooling can make that work. However, I think the "polyrepo" response to most of these advantages would be to focus on decoupling your systems instead. Take for instance: > With a monorepo, you just refactor the API and all of its callers in one commit. That's not always trivial, but it's much easier than it would be with lots of small repos. I've seen APIs with thousands of usages across hundreds of projects get refactored and with a monorepo setup it's so easy that it's no one even thinks twice. Like, that's really cool you can do that. But why are doing that?! Why are you breaking your API contract and forcing all of your clients to change all at once? Of course, proper decoupling also requires good engineering. A polyrepo environment can still get horribly tangled, but the natural response to all of these tangling problems in a polyrepo is to move in a direction of looser coupling.

jvolkman 4y ago
What is proper decoupling? In a properly-tooled monorepo, project A can't take a dependency on project B unless B is a public library or explicitly gives access to A [1]. Authors have full control over coupling. Sure, avoid changing the API contract. But when the time comes to change the API, you can 1) make the change backwards compatible and maintain both methods forever; 2) release a new major version and maintain both versions forever; or 3) just migrate the callers and immediately be free of all technical debt that would've been accrued in 1 and 2. This assumes internal clients whom you presumably can't break. 1: https://bazel.build/concepts/visibility https://bazel.build/concepts/visibility
- lliamander 4y ago
  Visibility is somewhat orthogonal. I mean yes, you can avoid coupling issues by preventing the dependency altogether, but really the interesting problems occur once you do have a dependency. Once you have the dependency, the loosely coupled approach means that (when possible) you avoid making changes to your API contract that would break your clients. I see the appeal of approach #3 that you suggest, but here's the problem I see with that (and maybe you have an answer): For any change that breaks the contract (outside of trivial things like renaming an identifier) you are necessarily either adding a requirement your clients may not be able to satisfy, or removing a capability they depend upon. Migrating your clients in that case is more than just a simple refactor; the client may need to re-architect so that it can adapt to the change in contract or even move to a new dependency altogether. If you're not the owner of that client, that means you are either interrupting the other team while they are forced to help you with the migration, or you are blocked waiting for them to have the time. In general, I would say the best approach to making breaking changes to an API is to use a deprecation process. That allows clients to migrate at their own pace. You can of course do that in either a monorepo or a polyrepo approach, but my expectation would be that the monorepo doesn't really provide you with any advantages in that case.

switch33 4y ago

Large repos make sense or don't make sense based on companies that work with large data or not based on predicate calculus and derivatives usually dealing with repos as well as stories and have more problems with ssds too. There is lots of problems associated with ssds as well as large monorepos. There are more complicated than people realize but if you did google code jam it teaches them somewhat but needs to be explained too. There problem is stories sort of intersect with programming too. Clockwork with ssds needs to be reworked for google code jams. The problem is elixir sort of works with stories and programming. Predicate calculus and proof theories sort of are the only way programming will really make sense in a world full of ssds. Leveldb could be a more interesting problem for google code jams if it has some newer features too. Conflict resolution is tower of hanoi and that has problems with consensus algorithms and concat too.SSDs need to do derivatives for pieceing and parting software too and that is more interesting too.

dqpb 4y ago

Use a monorepo, but organize your code as if it will someday be split into many repos.