11 ms·
Benefits of a Monorepo
- royjacobs 8y agoHmm, if you're working on a single product (like the author is), then of course it makes sense to stick them in the same repository. But isn't this just common sense? I mean, if you have a single product split over many repositories then you will run into the issues mentioned by the author: A change will invariably involve multiple repositories. Again, to me this seems to be just an indication that you are, in fact, working on a single product.
- TickleSteve 8y agoThis is obvious to me (a monorepo advocate) but most people these days tend to organise their repos based on module/implementation lines rather than on product boundaries. My motto is: "What you release together, you should version-control together".
- Conan_Kudo 8y agoI'm a multirepo advocate, but I suggest splits similarly. Maybe it's my old-timey way of releasing software, but because I do releases as RPMs, it bothers me less if you do things like have your base library, your backend services, and your frontend in the same repo. If they need to be able to be deployed independently, I just split them up as subpackages. I guess most language package managers don't really have such a concept, so people are optimizing for that...
- ctroein89 8y agoI've learnt that for "What you release together, you should version-control together", the phrase "released together" needs to be defined as "what gets globally updated at once". If you provide anything for customers to implement, you cannot update those globally at once (because you need the customer to update), and should think about using a multi-repo set-up. My experience is that testing the interactions of historic versions of different components in a mono-repo against each other is difficult. With a multi-repo set-up, one can checkout to whatever historic version one needs to test. That makes it trivially easy to set-up testing that allows one to test historic versions of components against each.
- adrianN 8y agoI like having one repo for the project ("What you release together") that tracks subprojects as submodules.
- lukasLansky 8y agoIt's a common sense, but it's really hard to argue against super-fine splitting. Having repo per module seems like stronger separations of concerns from a distance. Of course, what in the end really decides about the strength of your concern separation is the shape of your CI, but alas, repository granuality is just much more visible then CI checks.
- royjacobs 8y agoI would argue that if you have clear separation of concerns you wouldn't need to make changes in multiple repositories all the time.
- lukasLansky 8y agoWell let's take frontend/backend separation in a classic webapp -- let's not think about React for now. This makes good concern separation as rendering HTML is something else than computing prices. If you put frontend and backend into different repositories, you need to make two PRs for every new feature and -- for much worse -- you make integration testing much harder as you don't have atomic product merges to run automation against. Don't laugh about this example, my corporation is doing stuff like this (backend only repo tightly coupled to frontend repo) all the time. So maybe we can steelman the argument into "good vertical concern separation doesn't force you into doing multiple PR for single feature, as opposed to horizontal frontend/backend/repository splits."? But I'm still not sure. Wide integration testing is necesity for reliable microservices. It's difficult to do that with dependencies being out of the scope of the change.
- royjacobs 8y agoIntegrating with other products is something that goes beyond the scope of SCM repository layout, though. If you change your API then clearly someone on the other side also needs to adjust, but that's not simply sending a pull request. That's also making sure you deploy your products in the right order, ensure backwards compatibility and so on. Re integration testing: That's why things like contract testing are becoming more popular. You make sure that you do as much testing as you can without relying on an entire ecosystem of depdendencies being available, so once you DO release something you can be fairly confident that it's already working as expected.
- dalore 8y agoSounds like you need/could split it up into core/engine functionality, and then implementation of that core in different environments (i.e. a ios and android one). That makes sense to me, to have 3 repositories. Common code, perhaps even in your own dsl. And then implementations done in the various platforms.
- enraged_camel 8y agoWhat about a single product with multiple components though? Let's say you have an API and a browser client (SPA). Should you keep them in the same repo, or separate them? So far the only good reason I've had for separating them has been related to deployment concerns (e.g. your API is deployed to AWS and your browser client is on Cloudflare).
- royjacobs 8y agoIf you have an API and a SPA that's tightly coupled then, yes, same repo. If your API is a separate product with a separate release cycle, proper testing and backwards compatibility checks, then different repo.
- viraptor 8y ago> then of course it makes sense to stick them in the same repository. But isn't this just common sense? It depends. Are you planning more projects which may share functionality? Are you going to release within a company with other teams that may want to integrate? Are some components managed by dedicated teams who may have their own cycles of work? Are you planning to split off a specific service soon? Or to quote risky business: what is the problem we're trying to solve?
- dkersten 8y ago> ... may ... If something only may be the case, then I'd say its premature effort to plan for something that may or may not happen. Better to do the simplest thing now and then refactor when the need arises.
- Ensorceled 8y agoI decided to read this expecting to see an explanation as to why my DRF backend should be in the same repository as my React Native frontend, and instead found a discussion on why my React Native code shouldn't be broken up into iOS and Android. Not sure this is what people mean by a monorepo ...
- shibel 8y agoI tried deploying my DRF backend[1] + Nuxt.js[2] demo-app on Heroku[3] as one repo but it was a NIGHTMARE. Yes, it sucks to have to open both projects in my IDE in development, but it's nowhere near the difficulty I had with the mono-repo. Plus, I can find things much faster within each project when they are split. [1]: backend → https://github.com/SHxKM/django-vue-ssr https://github.com/SHxKM/django-vue-ssr [2]: frontend → https://github.com/SHxKM/django-nuxt-ssr-front https://github.com/SHxKM/django-nuxt-ssr-front [3]: live-app → https://django-nuxt-ssr.herokuapp.com/ https://django-nuxt-ssr.herokuapp.com/ (on a free dyno, could crash on first load)
- nicoburns 8y agoI've settled on a many-repos-in-same-parent-folder pattern, which means that I can open them all in the same editor window if I want to. I've also forgone proper IDE "project" in favour of opening the directory in my editor on an ad-hoc basis from the command line (e.g. `cd frontend && subl .`). It's not perfect, but it avoids the issue of having too many windows, or having to fumble through IDE project switching dialogs.
- deleted 8y ago[deleted]
- shibel 8y agoYeah, the "issue" for me is spinning up the backend-server from the frontend project, or vice-versa, but I should probably stop being lazy and write a simple script to spin up both at the same time.
- amelius 8y agoWhat tools are people using to manage multiple repos? A search reveals many, but what is the latest and greatest?
- pjmlp 8y agoNuGET, Maven, whatever is the package manager for the language in question. The outcome of multiple repos should be modular libraries.
- Cthulhu_ 8y ago> The outcome of multiple repos should be modular libraries. Yes; also they should all be treated as separate, independent products. Where multi-repos go wrong is when a whole product spans multiple repos, and a feature of the product depends on features being added or altered in multiple projects. Ideally, the split should be so that nobody has to work across multiple repos to finish a feature. The github / NPM ecosystem is IMO a good way to look at it; libraries that are all independent of one another who have a (usually pretty good) adherence to semver.
- amelius 8y ago> Ideally, the split should be so that nobody has to work across multiple repos to finish a feature. That does not sound realistic. Often I have to add a new function, and I think: where (in what library) does this function belong, and very often it's not the main library I'm working in at that moment.
- lmm 8y agoAdding a new function anywhere other than the leaf project where it's used is premature. Organize around user-facing functionality. Pull out common libraries as and when you have something common enough to factor out (and, crucially, test independently), not before.
- 8y ago
- is0tope 8y agoI've been an avid user of docker-compose for web app type projects. In this case, I've found putting everything in one repo perfectly manageable. Usually this means the backend, frontend, nginx and any other random microservices are just in separate folders in the base repo along with a compose yaml file. I gather that if you are using more complex CI/CD tools this becomes more complicated...
- robbrit 8y agoOne advantage that isn't mentioned in the article: code examples. As a monorepo grows, it's more likely that someone else using the repo somewhere has attempted to do the same thing that you want to do, and you can use tools like searchcode to easily find usages and see how they did it. Often this works better than documentation, which for internal projects tends to be non-existent or out-of-date. One disadvantage: a lot of IDEs don't cope well with monorepos as they try to index the entire thing.
- jacques_chester 8y ago> This makes CI test runs slightly longer, as they first need to do a fresh checkout. -depth=1 ameliorates this somewhat (though plays poorly with tag-heavy workflows).
- bostonpete 8y agoI read this, and I still don't know what "many benefits" they're referring to. I basically read one in the article -- a monorepo avoids the pain associated with making changes to multiple repo at once, which was happening frequently. Does the author make reference to some other benefits that I just completely missed...?
- mark_l_watson 8y agoWhen I was a contractor at Google I was surprised by the mono repo, but really liked it. Scaling issues aside, it makes sense having all dependencies in one repo. I thought of having a mono repo at home because many of my projects are inter related but instead I spent some time with a library setup. For Common Lisp, I used a pattern in one of Zach’s blogs to have one central repo for all of my personal Common Lisp code that is configured as a root for Quicklisp. Any of my code can be quick loaded no matter where I am working. Really simple but it makes it possible to write small utilities/libraries and reuse them for whatever I am doing. I sort of do the same for Python, writing a setup.py for each small project or library to install globally, including appropriate executable scripts. This also allows me to combine small things easily. I am not yet set up to do this yet for my Haskell projects but it is on my todo list.
- dkersten 8y agoThat's what I do too. I have a git repo with submodules for each library and service that's related to the overall thing. So, while I'm working on it, locally, it almost looks like a monorepo, but the way things are built, run, deployed and stored in remote git is as separate repos and artifacts. If changes regularly involve multiple submodules, then I know that the boundaries are wrong and refactoring is needed.
- petters 8y agoI merged all my personal C++ projects into one monorepo and I really liked it. Especially useful with C++ since setting up the build system is so cumbersome.
- baybal2 8y agoIt could well read like "The Many Benefits of Using Repo Modules" To me it feels like a misguided attempts to work around the need of having a package manager Why not simply build your SRPM or DEB package repo for code? Worked like magic for me on four of my past workplaces. This also works towards building your culture of stable releases instead of non-stop development and hotfixes.
- satya71 8y agoIt sounds like this person would have benefited from using Subversion or Perforce. Subversion 1.12 has alpha support for a limited local branch (called Checkpoints).
- notacoward 8y agoThese benefits do exist, but drawbacks still exist too. The "constantly updating, constantly recompiling" problem can get pretty bad when there are many developers. Developers either lose time recompiling more than they should need to, or keep working in out-of-date trees and dealing with trickier merges. Multiply by the number of developers in a large org, and that can be a lot of lost productivity. The thing I don't see discussed often enough, in all of the "monorepos are good/bad" debates, is that they require a certain discipline. When one commit can span multiple components, it's easy to add or ignore bad coupling between those components. That contributes to the rebuild treadmill for everyone, and is bad design in a bunch of other ways as well. IDEs make it worse, because they obscure the boundaries between components, but everyone ends up using an IDE because their predecessors made it very difficult to follow the flow of control across components otherwise. So you end up with one big ball of mud. With separate repos this is discouraged, because violating the rules of good design causes immediate pain. With a monorepo the pain must come in the form of stronger reviews, to discourage bad coupling even though it's easy. Unfortunately, the very same companies that have embraced monorepos have also abandoned separation of concerns as a design principle. :( ETA: please read bunderbunder's comment (https://news.ycombinator.com/item?id=19796960 https://news.ycombinator.com/item?id=19796960) as well. It's an insightful take on some of the same issues.
- Waterluvian 8y agoI like your comment about commits spanning components. Having many repos forces my team to think about how to stage changes that span many repos.
- jsight 8y agoTBH, I'm not convinced that a great effort to think about "how to stage changes that span many repos" is always (or maybe even often) a benefit. It seems like the simpler approach of just making it easy to make multimodule changes is a benefit of the monorepo.
- notacoward 8y ago
- bluGill 8y agoSummary for those who can't be bothered to read: we had two repos and copy/pasted code that needed to be in both instead of adding more repos. That copy/paste had pain and we solved it by going to one repo. They could have solved the pain by adding more repos as well. They never considered this as far as I can tell.
- underwater 8y agoThey specifically mentioned breaking code outbid to more repos: > We had three choices: ... Extract Core out of PSPDFKit-Android and again deal with fragmented pull requests.
- jorams 8y agoThat is a very uncharitable summary. The article discusses codebases being developed together or growing closer together over time, causing feature changes in one to need changes in another. It doesn't mention copy/pasting anything. They solved it by moving everything into a single repo. Adding more repos would once again split up pull requests for features that touch multiple codebases, which is the reason they merged two codebases into one repo in the first place.
- bluGill 8y agoI will admit to being uncharitable. I am assuming they did copy/paste coding to get features into both. It is possible that they actually did write everything twice, but that is generally such a stupid idea I can't believe it (except for UI - it makes perfect sense to write a the UI for android and iOS completely separate - the logic behind the UI should still be common, but the UI elements/APIs are different enough that common probably isn't worth it) Adding more repos should not split up pull requests. If it does that is a function of bad architecture.
- rkangel 8y agoSide note: blog posts and news articles that don't display the date are very irritating. There has been a lot of discussions about monorepos, I want to know if this is something new adding to the debate or something old. If it's a news article, I want to know how up to date it is. All I get from this is the slug saying '2019'.
- bunderbunder 8y agoFor my part, I've worked with two long-lived monorepos so far. Both were quite small, all told, but, nonetheless, in both cases I saw the monorepo largely serving as an incubator for bit rot. All those direct interdependencies, combined with the impossibility of just using package versioning to allow a library's consumers to upgrade at their own pace, meant that developers had a strong disincentive to do any cleanup that might result in a breaking change. They'd tend to prefer patches instead. So the long-term trend was for everything to eventually become a tangled mass of patchwork. Google has entire teams devoted to combatting this problem. I'm not sure most companies can (or want to) afford that. This is one of those spots where I think it's wise to be cautious about blog-driven development. By analogy, I've seen experienced woodworkers have fun with mocking the DIY furniture projects that come up in places like Pinterest. One common criticism was that, since wood expands and contracts with changing humidity, but only in certain directions, if you put it together in certain ways, you end up with a product that will literally tear itself apart over time. Of course, this isn't apparent from the Instructable article, because it's a slow process, and all those pictures were taken when the piece was still brand spanking new. It's also not something you're likely to see written in the comments section or anything like that - the people I've known who know this sort of thing aren't generally the type to write Instructables articles or maintain their own DIY blogs. I suspect something like this also happens with any decent sized codebase: Code is a living thing that expands and contracts with changing business requirements. Over time, it will twist itself out of shape. Weak coupling and fragmented codebases are two different (though related) ways of trying to limit the damage from this effect. They both bring their own challenges, though, and aren't perfect. Monorepos don't have the same drawbacks, but that doesn't mean they don't have any. One, I think, is that you're left directly exposed to the full strength of the forces that cause bit rot. If you're disciplined enough to recognize this and treat it as another incentive to attack bit rot head-on, then you'll probably end up in a better place for it. If you think monorepos are a panacea, though, you're setting yourself up for the kinds of failure that people generally don't like to blog about.
- notacoward 8y ago> an incubator for bit rot Genius phrase. Overall, that might be one of the best comments I've ever read on HN. The DIY furniture analogy is a good one. There's so much focus on what seems to work today, and so little awareness of what affects long-term maintainability. I see this at work too, everyone chasing "impact" for the half even when it screws us over any longer time scale. Understanding how code evolves over time is an essential part of being a competent developer. It's much more important than any in-the-moment coding test.
- solarengineer 8y agoApproaches such as Fan In supported by GoCD are of great help when working with monorepos. See https://docs.gocd.org/current/advanced_usage/fan_in.html https://docs.gocd.org/current/advanced_usage/fan_in.html
- tracker1 8y agoOne of the projects I'm on at work just switched several repos into a mono repo. The issue was that there were often related changes to the Database, UI, Configuration and API projects... Those four in particular were almost always released for 2-3 of them for a given feature. Coordinating releases to dev/qa servers even was a bit of a pain. Merging them removed most of this. There are several smaller/micro services that are still separate, however. They're generally less effected by the coordination of db/api/ui changes.
- mikewhy 8y agoThe CI story has been my biggest gripe with monorepos. I'm surprised that no separate tool has emerged, like the "custom Jenkins scripts" mentioned in the article. Maybe I'm missing something, but couldn't you get pretty far with something like: # This tells the tool where to look for changes packages_folder: ./packages # List dependencies. In this instance, if any files in `shared_api` change, frontend and backend need to build. dependencies: - frontend: - shared_api - backend: - shared_api # Definite build order build_order: - shared_api - backend - frontend # Somehow find these scripts in each package subfolder and try to run them. scripts: - build - test - deploy The tool itself would see which files have changed between two git revisions, figure out what packages need to be built, and run the scripts. But I'm probably missing a million and one things.
- kevinsimper 8y agoI really wish there existed more tools, it seems like there is a gap for some tooling. It would be a simple one that could check the git log and see what files has changed since last time. Or check the hash of each folder and compare it to previous commit. If you use Docker, you can leverage that docker will take a hash of the context and if the images already exist on the machine use the cache if nothing has changed for those files. That is also quite effective +1