9 ms·

From a Single Repo, to Multi-Repos, to Monorepo, to Multi-Monorepo

122 points by leoloso 5y ago

gpsx 5y ago
I have a project, a programming environment/runtime, that contains a number of overlapping applications. I started with a small number of repos, including one main one. I had a number of frustrations with that. One of them is I didn't like having shared and unshared code from different applications in the same repo, for the sake of managing versioning. I separated into more repos, maybe 20. After I split it up I discovered a number of new issues, much like the OP here. One of which is complications from working with several repos at the same time, such as when refactoring. I'll think about the next steps taken by the OP. Does anyone have any other practices they can recommend for managing these type of projects?
- dgnemo 5y ago
  > I had a number of frustrations with that. Could you elaborate more? The OP had to split the monorepo due to https://getcomposer.org/ https://getcomposer.org/ limitations (each package needs it's own repo it seems). What other issues have you found when working with a monorepo? Thanks!
  - Master_Odin 5y ago
    While composer does have this limitation in that packages are published by making new tags within the repo, frameworks like symfony and cakephp have workarounds where they have one monorepo where all packages are worked on, and then automation to push changes to read only repos of each component. So there's https://github.com/symfony/symfony https://github.com/symfony/symfony pushing to https://github.com/symfony/event-dispatcher https://github.com/symfony/event-dispatcher which gets published to packagist.
  - gpsx 5y ago
    For some background, I have a web version and two electron versions of the programming environment. Then there is a web runtime and a web server which run on the output of the programming environment. Originally the main repo contained code for all the applications, but also some code specific to one or more of the applications. I had trouble versioning a subset of the repository. I ended up versioning the main repo and bumping the version of all apps if anything in the main repo changed, because I didn't have a systematic way of knowing that only certain parts of the repo changed. Perhaps that is the piece I would need? I said there were a number of frustrations, but that was the driver. In the new repo format each repo produces a single library/module, and can be versioned as a whole. When I run an application for dev, I can run a given library from a release or from live code (with a dev server serving ES modules from a configured path). This works well when one of the repos is not changing. But I end up developing in a significant number of repos at the same time, so I just configure it to run from live code rather than release for all those libraries i'm working on. This feels like working with a monorepo except I have the hassle of managing commits to all thse repos. Long story short, right now I am working out of the master branch because I screwed up commits. You may have figured out by now my strength is not in dev ops.
    - dgnemo 5y ago
      Thanks for sharing your experience! I can see how you were struggling trying to handle different "versions" in subsets of the repo. In my experience the "monorepo" workflow implies a "working on master/head" mindset, but this is not always the best approach depending on your specific needs.
      - FranksTV 5y ago
        Lerna let's you version things independently, and if you use something like conventional commits / commitizen it can auto detect whether versions are major or minor. When you make a version with lerna it auto-tags the commit with the version numbers of the components, so consumers can still depend on a specific version of a component. I'm setting it up now at a new company and its pretty amazing.
- reportt 5y ago
  > Does anyone have any other practices they can recommend for managing these type of projects? Honestly, the only way around these sorts of issues is to utilize automation in some form. I've found that setting up repositories (like devpi[0], Artifactory[1], or Docker Registry[2]) on a shared network location (for your project, could be local if you work alone) and using CI/CD tools (like Jenkins[3]) are the key. The goal is that you end up working on one portion of the code base at a time, and those need to go through the standard validation processes so that you can pull in the updated package version when you work on something down-stream. You making sure that the CI/CD environment _doesn't_ have access to other packages's non-versioned code is key for making sure things actually work as expected. For example, if you have FooLib, and you need an update in that for BarApp, then even if you branch FooLib 1.2.3 to 1.2.3-1-gabc1234d (the `git describe` of the commit) on `feat/new-thingy` , then even if BarApp v2.3.4-1-gaf901234 depends on that new branch, it shouldn't be in any way able to reference that branch on the CI/CD build process. How do you get around this? Good development -- finish the FooLib branch, get that working, merge it in with the updated version, and push the package (with the new version) to the CI/CD-accessible repository. At that point, when you push your BarApp change, it can actually build and not die. But until FooLib has got a versioned update, BarApp's branch _shouldn't_ be able to build. The statement of "But I want to work on the changes locally, in parallel" is valid. That's what local development is for -- giving you space to work on related things that don't impact the upstream codebase. You should have the option to utilize FooLib's branch code in your BarApp code locally, and you can often do that via things like `pip install` or `maven install` or whatever the relevant local install command is. At this point, the package still probably has the same version number, so the local build doesn't trigger issues. You can work on the two and tweak and twist as you want, but refrain from actually trying to push BarApp referencing FooLib's branch until it's actually in the repo. This all takes a great deal of restraint and patience. The goal here is make it just a tad harder to introduce problems somewhere since you can't depend on something that hasn't been given the go-ahead. While there might be a lot of "Updated FooLib requirement to v1.2.4" throughout your codebase, why are you doing that just off-hand? If you are doing it because of a security issue or bug, let that be known in the commit message. If you are doing it because you can utilize a new feature/whatever, your commit message won't be just "Updated FooLib", you likely are doing "Added Feature X2Y, updated FooLib to 1.2.4". PHP I try not to touch much, simply because I've always had bad experiences. I know for a fact that there are decent ways to do it with build tools like Maven[4], setuptools[5], and Docker[6]. Hell, I have used Docker as a way to introduce versioned dependency packaging, only needing to use Docker Registry (each dependent project does a multi-stage build, pulling in the dependencies via the versioned package images). --- [0]: https://devpi.net/docs/devpi/devpi/latest/%2Bd/index.html https://devpi.net/docs/devpi/devpi/latest/%2Bd/index.html [1]: https://jfrog.com/artifactory/ https://jfrog.com/artifactory/ [2]: https://docs.docker.com/registry/ https://docs.docker.com/registry/ [3]: https://www.jenkins.io/ https://www.jenkins.io/ [4]: https://maven.apache.org/ https://maven.apache.org/ [5]: https://setuptools.readthedocs.io/en/latest/ https://setuptools.readthedocs.io/en/latest/ [6]: https://www.docker.com/ https://www.docker.com/
- lloydatkinson 5y ago
  A monorepo that contains a directory per thing is the way forward IMO.
lhorie 5y ago
Nice write-up. I have explored different repo strategies quite a bit myself in the course of a few efforts that I've been involved with. On one, we originally had a monolythic framework and everything the article said about cons is pretty spot on. However, I'll qualify by saying that I think the problems come less because of the nature of monolyths in general and more because of lack of experience with modular design. We wrote a new framework from scratch using a monorepo approach, with separate packages via Lerna. The problem here was tooling. Dependent builds were not supported and I've had to delete node_modules more times than I'd ever cared to count. The article talks about some github specific problems (namely, the issues list being a hodge-podge of every disparate package). We tried zenhub, it works ok, but it's a hack and it kinda shows. I've seen other projects organize things via tags. Ultimately it comes down to what the team is willing to put up with. We eventually broke the monorepo out into multi-repos, and while that solved the problem of managing issues, now the problem was that publishing packages + cross-package dependencies meant that development was slower (especially with code reviews, blocking CI tests, etc). Back to a monorepo using Rush.js (and later Bazel). Rush had similar limitations as Lerna (in particular, no support for dependent tests) and we ditched it soon afterwards. Bazel has a lot of features, but it takes some investment to get the most out of it. I wrote a tool to wrap over it[0] and setup things to meet our requirements. We tried the "multi-monorepo" approach at one point (really, this is just git submodules), and didn't get very good results. The commands that you need to run are draconian and having to remember to sync things manually all the time is prone to errors. What's worse is that since you're dealing with physically separate repos, you're back to not having good ways to do atomic integration tests across package boundaries. To be fair, I've seen projects use the submodules approach[1] and it could work depending on how stable your APIs are, but for corporate requirements, where things are always in flux, it didn't work out well. Which brings me to another effort I was involved with more recently: moving all our multi-repo services into a monorepo. The main rationale here is somewhat related to another reason submodules don't really fly: there's a ton of packages being used, a lot of stakeholders with various degrees of commit frequency, and reconciling security updates with version drift is a b*tch. For this effort we also invested into using Bazel. One of the strengths of this tool is how you can specify dependent tasks, for example "if I touch X file, only run the tests that are relevant". This is a big deal, because at 600+ packages, a full CI run consumes dozens of hours worth of compute time and we see several dozens commits a day. The problem with monorepos comes largely from the sheer scale: bumping something to the next major version requires codemods, and there's always someone doing some crazy thing you never anticipated. With that said, monorepos are not a panacea. A project from a sibling team is a components library and it uses a single repo approach. This means a single version to manage for the entire set of components. You may object that things are getting bumped even when they don't need to, but it turns out this is actually very well received by consumers, because it's far easier to upgrade than having to figure out the changelog of dozens of separate packages. I used a single repo monolyth-but-actually-modular setup for my OSS project[2] and that has worked well for me, for similar reasons: people appreciate curation, and since we want to avoid willy-nilly breaking changes, a single all-emcompassing version scheme encourages development to work towards stability rather than features-for-features-sake. My takeaway is that multi-repos cause a lot of headaches both for framework authorship and for service development, that single repos can be a great poor-mans choice for framework authors, and monorepos - with the appropriate amount of investment in tooling - have good multiplicative potential for complex project clusters. YMMV. [0] https://github.com/uber-web/jazelle https://github.com/uber-web/jazelle [1] https://github.com/sebbekarlsson/fjb/tree/master/external https://github.com/sebbekarlsson/fjb/tree/master/external [2] https://mithril.js.org/ https://mithril.js.org/
cryptica 5y ago
How far does this have to go before people realize how ridiculous this is. The Multi-mono-multi-monorepo stage? You don't need a monorepo. It's an anti-pattern that people resort to when their code is a tangled, tightly coupled mess. If the modules are loosely coupled and high cohesion, multiple repos is the ideal approach. The solution to the problem of having to 'constantly update dependencies' is not to bring them all into a single monorepo. The solution is to ensure that these dependencies handle separate concerns and have simple interfaces which allow them to be loosely coupled with the main project logic in such a way that they can be updated independently of each other. If different module dependencies often need to be updated together whenever you add a new feature or fix a bug, it almost certainly means that you have a problem with coupling and/or cohesion. You don't need tools that make it easier to work with a tangled mess. The correct solution is to untangle the mess. Otherwise the mess will keep getting worse.
- TechBro8615 5y ago
  What if you want to publish some of your modules and keep the rest private but depending on the published ones?
- choeger 5y ago
  Tightly coupled code is, and should be, the norm. Loose coupling is a pipe dream as soon as domain-specific data structures are involved. Loose coupling over domain-specific data structures (and their related procedures) just means that errors that should be caught by a run-of-the-mill type checker are now found in production. An example of this pattern can be seen pretty much whenever you open the developer console in a non-chrome browser on any site using some JavaScript framework. Downloading some code from the website and executing in the browser is an extreme form of loose coupling and by my experience it is pretty much never free of errors, except for the very specific combinations of browser/platform that the developer could test.
  - ragebol 5y ago
    You can have the procedural/functional code only tightly coupled to the datastructures but loose coupling between the other bits.
malkia 5y ago
I saw this recently https://chromium.googlesource.com/chromiumos/platform2 https://chromium.googlesource.com/chromiumos/platform2 " We moved from multiple separate repos in platform/ to a single repo in platform2/ for a number of reasons: - Make it easier to work across multiple projects simultaneously - Increase code re-use (via common libs) rather than duplicate utility functions multiple items over - Share the same build system " for "multi"-repo (not sure) what really, fuchsia uses something called jiri, and there is still gsync
- Cthulhu_ 5y ago
  One thing to note is that Chromium and other Google projects are HUGE codebases; most codebases of people thinking about multi- or monorepos, microservices or monoliths, etc only WISH they had the scale of code and developers that companies like Google have. All I'm saying is, beware of cargo cult thinking. Do something because you need it, it's practical, it's faster, not because someone else does it. I've had a few projects that were intentionally difficult because Someone decided to make it a microservices architecture, but in a monorepo to encourage the distributed monolith idea.
  - throwaways885 5y ago
    To be fair, Google, Facebook, et al chose to have monorepos when they were startups, so it's not necessarily cargo culting to look at what they did. I'd argue part of the success was cultural - with everything in one repo there isn't generally specific folders which are restricted in these large codebases, and that helps with a bunch of things.
alajmo 5y ago
I recently built mani[0], a cli tool used to manage multiple repositories, which can solve some of the same issues with many-repo solutions, though I have a lot less packages than the author. It's similar to the package meta mentioned in the article, but comes with improved filtering of projects when running commands, a way to organize your projects from a single config, create scripts that target certain projects/groups only, auto-completion and some other helpful features. [0] https://github.com/alajmo/mani https://github.com/alajmo/mani
lmilcin 5y ago
I have worked with all of these models. I have figured out that monorepo mainly covers for faults in maintaining separation between components (like ensuring only single direction of dependencies and no cycles, ensuring backwards compatibility, self-service, et.c) and to some extend to cover for out of control microservice craze. Kinda like giving unrestricted access to your PROD databases does help improve efficiency, but at the cost of additional risks and deterioration of separation between applications, lack of APIs for users to self service themselves, etc. Especially with microservices, these tend to fare poorly if you don't solve static costs of maintaining a service. So rather than invest in proper tooling let's just plop it into single repository. It will cover some problems a little bit. Now, I don't want to say monorepo is bad by itself. The problem is when it stunts people's ability to maintain proper APIs and development process.
- zarzavat 5y ago
  That's the whole point of a monorepo is it not? You get rid of all the ceremony about APIs and just say "Whomever modifies an API is responsible for updating all the API consumers as well as the API itself" (and the reverse too). This means that developers are free to write code that is correct now rather than be limited by technical decisions made in the past, that may no longer be valid. It also means that when you do update an API you are confronted with the technical reality of how the API consumers actually work. This may cause you to reevaluate and improve your mental model of the API. If your API is siloed away in its own repo then you can easily get in a position of taking it in a direction that is not actually in line with what API consumers actually need.
  - rahkiin 5y ago
    Whenever you do code updates of separate services, you still need to update all running instances at the same time. Being able to change the code in 1 commit does not necessarily mean the services also update at the same time, which will cause api errors.
    - jayd16 5y ago
lugged 5y ago
Maybe it's time to wonder if it's not how you arrange the files?
- NavinF 5y ago
  Monorepos have nothing to do with arranging files. It's all about making atomic commits to multiple projects simultaneously.
  - dboreham 5y ago
    I'd argue that it's not. The atomic commit thing is cited because it's about the only argument in favor of monorepo that has technical merit. So a smart debater will latch onto that argument knowing it is hard to defeat. imho it is all about inertia and laziness: an organization begins with little code and one repo. They grow. They write more code. They acquire more code. Nobody is interested in stepping forward to do the hard work of managing all this code. So the repo gets bigger and bigger with more and more discrete projects inside it, until the situation becomes largely unworkable (e.g. clone the tree takes 20 min, CI builds of different projects don't work because tools used can't deal with multiple projects in the same repo). Then people justify their mess with arguments about sharing code and atomic commits. Also I'd argue it is to some extent about lack of knowledge -- people may just not be familiar modularity mechanisms that will solve their problems, because they've always depended on code being in the same repo.
    - lhorie 5y ago
      The atomic commit thing is just a means to some ends. Some ends are indeed easier to achieve with monorepos. Security audits/patching is one example I've dealt with in the past (dealing w/ supply chain vulns in a multi-repo world is a goddamn nightmare). Cross-package integration testing is another example. The thing about projects getting bigger is orthogonal to whether people pick monorepos or not. Many project feature sets are simply complex in a way that you can't refactor your way out of. When I hear this argument that projects could somehow be made smaller, typically it's from someone who's never had to deal with the regressions of such a decision or someone who's never had to be accountable for their estimates. I've seen firsthand projects that got rewritten with the benefit of hindsight, proper staffing, management blessing and all the jazz, and still struggle to meet feature parity of their older counterparts. It's clearly not a question of being interested in putting in the work, or even having the budget for it. From what I've seen, messiness is largely a function of experience. A lot of people simply don't have experience in writing libraries and/or architecting systems. I find that once they acquire the skills, quality of encapsulation improves a lot. FWIW, I think monorepos help with that transition because it provides that fast feedback loop of single repos while a developer is learning the ropes of how to librarize, instead of getting bogged down waiting on slow code review/CI/publish feedback loops.
sebyx07 5y ago
The core problem is that code is always changing, every single day from Monday to Friday. And it gets worse when other code/projects depend on it
dboreham 5y ago
Whatever the problem, git submodules are rarely the solution.
- jayd16 5y ago
  Now that most guis support them, submodules actually work pretty well. The only major issue I face with git novices is making sure everyone on the team sets their machines to pull recursively.
Dm_Linov 5y ago
It looks like the last step in missing: after discovering all disadvantages of submodules (they are in fact quite dangerous when working in a team), the author should have had switched to Git X-Modules[0] and use the same multi-monorepo approach without extra hassle :-) [0] https://gitmodules.com https://gitmodules.com

122 points by leoloso 5y ago

gpsx 5y ago

I have a project, a programming environment/runtime, that contains a number of overlapping applications. I started with a small number of repos, including one main one. I had a number of frustrations with that. One of them is I didn't like having shared and unshared code from different applications in the same repo, for the sake of managing versioning. I separated into more repos, maybe 20. After I split it up I discovered a number of new issues, much like the OP here. One of which is complications from working with several repos at the same time, such as when refactoring. I'll think about the next steps taken by the OP. Does anyone have any other practices they can recommend for managing these type of projects?

dgnemo 5y ago
> I had a number of frustrations with that. Could you elaborate more? The OP had to split the monorepo due to https://getcomposer.org/ https://getcomposer.org/ limitations (each package needs it's own repo it seems). What other issues have you found when working with a monorepo? Thanks!
- Master_Odin 5y ago
  While composer does have this limitation in that packages are published by making new tags within the repo, frameworks like symfony and cakephp have workarounds where they have one monorepo where all packages are worked on, and then automation to push changes to read only repos of each component. So there's https://github.com/symfony/symfony https://github.com/symfony/symfony pushing to https://github.com/symfony/event-dispatcher https://github.com/symfony/event-dispatcher which gets published to packagist.
- gpsx 5y ago
  For some background, I have a web version and two electron versions of the programming environment. Then there is a web runtime and a web server which run on the output of the programming environment. Originally the main repo contained code for all the applications, but also some code specific to one or more of the applications. I had trouble versioning a subset of the repository. I ended up versioning the main repo and bumping the version of all apps if anything in the main repo changed, because I didn't have a systematic way of knowing that only certain parts of the repo changed. Perhaps that is the piece I would need? I said there were a number of frustrations, but that was the driver. In the new repo format each repo produces a single library/module, and can be versioned as a whole. When I run an application for dev, I can run a given library from a release or from live code (with a dev server serving ES modules from a configured path). This works well when one of the repos is not changing. But I end up developing in a significant number of repos at the same time, so I just configure it to run from live code rather than release for all those libraries i'm working on. This feels like working with a monorepo except I have the hassle of managing commits to all thse repos. Long story short, right now I am working out of the master branch because I screwed up commits. You may have figured out by now my strength is not in dev ops.
  - dgnemo 5y ago
    Thanks for sharing your experience! I can see how you were struggling trying to handle different "versions" in subsets of the repo. In my experience the "monorepo" workflow implies a "working on master/head" mindset, but this is not always the best approach depending on your specific needs.
    - FranksTV 5y ago
      Lerna let's you version things independently, and if you use something like conventional commits / commitizen it can auto detect whether versions are major or minor. When you make a version with lerna it auto-tags the commit with the version numbers of the components, so consumers can still depend on a specific version of a component. I'm setting it up now at a new company and its pretty amazing.
reportt 5y ago
> Does anyone have any other practices they can recommend for managing these type of projects? Honestly, the only way around these sorts of issues is to utilize automation in some form. I've found that setting up repositories (like devpi[0], Artifactory[1], or Docker Registry[2]) on a shared network location (for your project, could be local if you work alone) and using CI/CD tools (like Jenkins[3]) are the key. The goal is that you end up working on one portion of the code base at a time, and those need to go through the standard validation processes so that you can pull in the updated package version when you work on something down-stream. You making sure that the CI/CD environment _doesn't_ have access to other packages's non-versioned code is key for making sure things actually work as expected. For example, if you have FooLib, and you need an update in that for BarApp, then even if you branch FooLib 1.2.3 to 1.2.3-1-gabc1234d (the `git describe` of the commit) on `feat/new-thingy` , then even if BarApp v2.3.4-1-gaf901234 depends on that new branch, it shouldn't be in any way able to reference that branch on the CI/CD build process. How do you get around this? Good development -- finish the FooLib branch, get that working, merge it in with the updated version, and push the package (with the new version) to the CI/CD-accessible repository. At that point, when you push your BarApp change, it can actually build and not die. But until FooLib has got a versioned update, BarApp's branch _shouldn't_ be able to build. The statement of "But I want to work on the changes locally, in parallel" is valid. That's what local development is for -- giving you space to work on related things that don't impact the upstream codebase. You should have the option to utilize FooLib's branch code in your BarApp code locally, and you can often do that via things like `pip install` or `maven install` or whatever the relevant local install command is. At this point, the package still probably has the same version number, so the local build doesn't trigger issues. You can work on the two and tweak and twist as you want, but refrain from actually trying to push BarApp referencing FooLib's branch until it's actually in the repo. This all takes a great deal of restraint and patience. The goal here is make it just a tad harder to introduce problems somewhere since you can't depend on something that hasn't been given the go-ahead. While there might be a lot of "Updated FooLib requirement to v1.2.4" throughout your codebase, why are you doing that just off-hand? If you are doing it because of a security issue or bug, let that be known in the commit message. If you are doing it because you can utilize a new feature/whatever, your commit message won't be just "Updated FooLib", you likely are doing "Added Feature X2Y, updated FooLib to 1.2.4". PHP I try not to touch much, simply because I've always had bad experiences. I know for a fact that there are decent ways to do it with build tools like Maven[4], setuptools[5], and Docker[6]. Hell, I have used Docker as a way to introduce versioned dependency packaging, only needing to use Docker Registry (each dependent project does a multi-stage build, pulling in the dependencies via the versioned package images). --- [0]: https://devpi.net/docs/devpi/devpi/latest/%2Bd/index.html https://devpi.net/docs/devpi/devpi/latest/%2Bd/index.html [1]: https://jfrog.com/artifactory/ https://jfrog.com/artifactory/ [2]: https://docs.docker.com/registry/ https://docs.docker.com/registry/ [3]: https://www.jenkins.io/ https://www.jenkins.io/ [4]: https://maven.apache.org/ https://maven.apache.org/ [5]: https://setuptools.readthedocs.io/en/latest/ https://setuptools.readthedocs.io/en/latest/ [6]: https://www.docker.com/ https://www.docker.com/
lloydatkinson 5y ago
A monorepo that contains a directory per thing is the way forward IMO.

lhorie 5y ago

Nice write-up. I have explored different repo strategies quite a bit myself in the course of a few efforts that I've been involved with. On one, we originally had a monolythic framework and everything the article said about cons is pretty spot on. However, I'll qualify by saying that I think the problems come less because of the nature of monolyths in general and more because of lack of experience with modular design. We wrote a new framework from scratch using a monorepo approach, with separate packages via Lerna. The problem here was tooling. Dependent builds were not supported and I've had to delete node_modules more times than I'd ever cared to count. The article talks about some github specific problems (namely, the issues list being a hodge-podge of every disparate package). We tried zenhub, it works ok, but it's a hack and it kinda shows. I've seen other projects organize things via tags. Ultimately it comes down to what the team is willing to put up with. We eventually broke the monorepo out into multi-repos, and while that solved the problem of managing issues, now the problem was that publishing packages + cross-package dependencies meant that development was slower (especially with code reviews, blocking CI tests, etc). Back to a monorepo using Rush.js (and later Bazel). Rush had similar limitations as Lerna (in particular, no support for dependent tests) and we ditched it soon afterwards. Bazel has a lot of features, but it takes some investment to get the most out of it. I wrote a tool to wrap over it[0] and setup things to meet our requirements. We tried the "multi-monorepo" approach at one point (really, this is just git submodules), and didn't get very good results. The commands that you need to run are draconian and having to remember to sync things manually all the time is prone to errors. What's worse is that since you're dealing with physically separate repos, you're back to not having good ways to do atomic integration tests across package boundaries. To be fair, I've seen projects use the submodules approach[1] and it could work depending on how stable your APIs are, but for corporate requirements, where things are always in flux, it didn't work out well. Which brings me to another effort I was involved with more recently: moving all our multi-repo services into a monorepo. The main rationale here is somewhat related to another reason submodules don't really fly: there's a ton of packages being used, a lot of stakeholders with various degrees of commit frequency, and reconciling security updates with version drift is a b*tch. For this effort we also invested into using Bazel. One of the strengths of this tool is how you can specify dependent tasks, for example "if I touch X file, only run the tests that are relevant". This is a big deal, because at 600+ packages, a full CI run consumes dozens of hours worth of compute time and we see several dozens commits a day. The problem with monorepos comes largely from the sheer scale: bumping something to the next major version requires codemods, and there's always someone doing some crazy thing you never anticipated. With that said, monorepos are not a panacea. A project from a sibling team is a components library and it uses a single repo approach. This means a single version to manage for the entire set of components. You may object that things are getting bumped even when they don't need to, but it turns out this is actually very well received by consumers, because it's far easier to upgrade than having to figure out the changelog of dozens of separate packages. I used a single repo monolyth-but-actually-modular setup for my OSS project[2] and that has worked well for me, for similar reasons: people appreciate curation, and since we want to avoid willy-nilly breaking changes, a single all-emcompassing version scheme encourages development to work towards stability rather than features-for-features-sake. My takeaway is that multi-repos cause a lot of headaches both for framework authorship and for service development, that single repos can be a great poor-mans choice for framework authors, and monorepos - with the appropriate amount of investment in tooling - have good multiplicative potential for complex project clusters. YMMV. [0] https://github.com/uber-web/jazelle https://github.com/uber-web/jazelle [1] https://github.com/sebbekarlsson/fjb/tree/master/external https://github.com/sebbekarlsson/fjb/tree/master/external [2] https://mithril.js.org/ https://mithril.js.org/

cryptica 5y ago

How far does this have to go before people realize how ridiculous this is. The Multi-mono-multi-monorepo stage? You don't need a monorepo. It's an anti-pattern that people resort to when their code is a tangled, tightly coupled mess. If the modules are loosely coupled and high cohesion, multiple repos is the ideal approach. The solution to the problem of having to 'constantly update dependencies' is not to bring them all into a single monorepo. The solution is to ensure that these dependencies handle separate concerns and have simple interfaces which allow them to be loosely coupled with the main project logic in such a way that they can be updated independently of each other. If different module dependencies often need to be updated together whenever you add a new feature or fix a bug, it almost certainly means that you have a problem with coupling and/or cohesion. You don't need tools that make it easier to work with a tangled mess. The correct solution is to untangle the mess. Otherwise the mess will keep getting worse.

TechBro8615 5y ago
What if you want to publish some of your modules and keep the rest private but depending on the published ones?
choeger 5y ago
Tightly coupled code is, and should be, the norm. Loose coupling is a pipe dream as soon as domain-specific data structures are involved. Loose coupling over domain-specific data structures (and their related procedures) just means that errors that should be caught by a run-of-the-mill type checker are now found in production. An example of this pattern can be seen pretty much whenever you open the developer console in a non-chrome browser on any site using some JavaScript framework. Downloading some code from the website and executing in the browser is an extreme form of loose coupling and by my experience it is pretty much never free of errors, except for the very specific combinations of browser/platform that the developer could test.
- ragebol 5y ago
  You can have the procedural/functional code only tightly coupled to the datastructures but loose coupling between the other bits.

malkia 5y ago

I saw this recently https://chromium.googlesource.com/chromiumos/platform2 https://chromium.googlesource.com/chromiumos/platform2 " We moved from multiple separate repos in platform/ to a single repo in platform2/ for a number of reasons: - Make it easier to work across multiple projects simultaneously - Increase code re-use (via common libs) rather than duplicate utility functions multiple items over - Share the same build system " for "multi"-repo (not sure) what really, fuchsia uses something called jiri, and there is still gsync

Cthulhu_ 5y ago
One thing to note is that Chromium and other Google projects are HUGE codebases; most codebases of people thinking about multi- or monorepos, microservices or monoliths, etc only WISH they had the scale of code and developers that companies like Google have. All I'm saying is, beware of cargo cult thinking. Do something because you need it, it's practical, it's faster, not because someone else does it. I've had a few projects that were intentionally difficult because Someone decided to make it a microservices architecture, but in a monorepo to encourage the distributed monolith idea.
- throwaways885 5y ago
  To be fair, Google, Facebook, et al chose to have monorepos when they were startups, so it's not necessarily cargo culting to look at what they did. I'd argue part of the success was cultural - with everything in one repo there isn't generally specific folders which are restricted in these large codebases, and that helps with a bunch of things.

alajmo 5y ago

I recently built mani[0], a cli tool used to manage multiple repositories, which can solve some of the same issues with many-repo solutions, though I have a lot less packages than the author. It's similar to the package meta mentioned in the article, but comes with improved filtering of projects when running commands, a way to organize your projects from a single config, create scripts that target certain projects/groups only, auto-completion and some other helpful features. [0] https://github.com/alajmo/mani https://github.com/alajmo/mani

lmilcin 5y ago

I have worked with all of these models. I have figured out that monorepo mainly covers for faults in maintaining separation between components (like ensuring only single direction of dependencies and no cycles, ensuring backwards compatibility, self-service, et.c) and to some extend to cover for out of control microservice craze. Kinda like giving unrestricted access to your PROD databases does help improve efficiency, but at the cost of additional risks and deterioration of separation between applications, lack of APIs for users to self service themselves, etc. Especially with microservices, these tend to fare poorly if you don't solve static costs of maintaining a service. So rather than invest in proper tooling let's just plop it into single repository. It will cover some problems a little bit. Now, I don't want to say monorepo is bad by itself. The problem is when it stunts people's ability to maintain proper APIs and development process.

zarzavat 5y ago
That's the whole point of a monorepo is it not? You get rid of all the ceremony about APIs and just say "Whomever modifies an API is responsible for updating all the API consumers as well as the API itself" (and the reverse too). This means that developers are free to write code that is correct now rather than be limited by technical decisions made in the past, that may no longer be valid. It also means that when you do update an API you are confronted with the technical reality of how the API consumers actually work. This may cause you to reevaluate and improve your mental model of the API. If your API is siloed away in its own repo then you can easily get in a position of taking it in a direction that is not actually in line with what API consumers actually need.
- rahkiin 5y ago
  Whenever you do code updates of separate services, you still need to update all running instances at the same time. Being able to change the code in 1 commit does not necessarily mean the services also update at the same time, which will cause api errors.
  - jayd16 5y ago

lugged 5y ago

Maybe it's time to wonder if it's not how you arrange the files?

NavinF 5y ago
Monorepos have nothing to do with arranging files. It's all about making atomic commits to multiple projects simultaneously.
- dboreham 5y ago
  I'd argue that it's not. The atomic commit thing is cited because it's about the only argument in favor of monorepo that has technical merit. So a smart debater will latch onto that argument knowing it is hard to defeat. imho it is all about inertia and laziness: an organization begins with little code and one repo. They grow. They write more code. They acquire more code. Nobody is interested in stepping forward to do the hard work of managing all this code. So the repo gets bigger and bigger with more and more discrete projects inside it, until the situation becomes largely unworkable (e.g. clone the tree takes 20 min, CI builds of different projects don't work because tools used can't deal with multiple projects in the same repo). Then people justify their mess with arguments about sharing code and atomic commits. Also I'd argue it is to some extent about lack of knowledge -- people may just not be familiar modularity mechanisms that will solve their problems, because they've always depended on code being in the same repo.
  - lhorie 5y ago
    The atomic commit thing is just a means to some ends. Some ends are indeed easier to achieve with monorepos. Security audits/patching is one example I've dealt with in the past (dealing w/ supply chain vulns in a multi-repo world is a goddamn nightmare). Cross-package integration testing is another example. The thing about projects getting bigger is orthogonal to whether people pick monorepos or not. Many project feature sets are simply complex in a way that you can't refactor your way out of. When I hear this argument that projects could somehow be made smaller, typically it's from someone who's never had to deal with the regressions of such a decision or someone who's never had to be accountable for their estimates. I've seen firsthand projects that got rewritten with the benefit of hindsight, proper staffing, management blessing and all the jazz, and still struggle to meet feature parity of their older counterparts. It's clearly not a question of being interested in putting in the work, or even having the budget for it. From what I've seen, messiness is largely a function of experience. A lot of people simply don't have experience in writing libraries and/or architecting systems. I find that once they acquire the skills, quality of encapsulation improves a lot. FWIW, I think monorepos help with that transition because it provides that fast feedback loop of single repos while a developer is learning the ropes of how to librarize, instead of getting bogged down waiting on slow code review/CI/publish feedback loops.

sebyx07 5y ago

The core problem is that code is always changing, every single day from Monday to Friday. And it gets worse when other code/projects depend on it

dboreham 5y ago

Whatever the problem, git submodules are rarely the solution.

jayd16 5y ago
Now that most guis support them, submodules actually work pretty well. The only major issue I face with git novices is making sure everyone on the team sets their machines to pull recursively.

Dm_Linov 5y ago

It looks like the last step in missing: after discovering all disadvantages of submodules (they are in fact quite dangerous when working in a team), the author should have had switched to Git X-Modules[0] and use the same multi-monorepo approach without extra hassle :-) [0] https://gitmodules.com https://gitmodules.com