5 ms·

In the last few years, I saw a lot of people using monorepos, and discouraging using submodules. Frankly, I don't know why. For small projects, sure - it it an

by stared 7y ago

In the last few years, I saw a lot of people using monorepos, and discouraging using submodules. Frankly, I don't know why. For small projects, sure - it it an overhead. For anything larger, I find it useful to encapsulate things - be it installable packages, or if it is not the case - at least other repositories. In the last months I split one project into 4 repos (https://github.com/Quantum-Game/ https://github.com/Quantum-Game/) and couldn't be happier about that move. It makes the code cleaner, Pull Requests more separated, etc.

edwintorok 7y ago
Once your code grows and you end up with too many repos you'll be yearning to get back to monorepos. It allows to make changes in one go instead of having 3-4 PRs that all have to be merged at same time or otherwise the build breaks. Having separate repos only makes sense if they are maintained by separate teams with a well defined API in-between that doesn't change often. If the same team maintains all repos then it is just overhead, especially if you end up with a 100 repos and you have significantly less than 100 engineers in your team. There are also potential build time savings when having a monorepo since you can do parallel builds across different subdirs without serialising at package boundaries.
- cryptica 7y ago
  I maintain such a project with many dependencies which I also maintain separately. Yes it can be a lot of extra work to publish but this argument is based on the assumption that dependencies need to be changed constantly and this assumption is wrong. There should be pressure to design dependencies in such a way that they are modular and don't need to be changed often - This unpleasantness associated with updating deep dependencies is good and necessary. These should rarely be updated, if they need to be updated often, then they were designed incorrectly. Many of the dependencies and sub-dependencies which I maintain are several years old. They are small enough and stable enough that they basically never need to change. If I need to update them (which is rare), I may need to do a cascading update of multiple dependents and this can take a while but if I have to do this big update only once every 6 months, it's totally worth having them in separate repos.
  - beagle3 7y ago
    That depends on the scale of the project, of course - if you have thousands of engineers as MicroFaceGoog does, then it’a a daily, even hourly, occurrence for many dependencies. You are not MicroFaceGood, of course, so you only do that every 6 months. However, I what way are separate repos nicer for you? Personally, I settled on a monorepo with a “$dep-devel” branch for every dependency, which I occasionally have on multiple working trees out at the same time, and which I feel gives me all the benefit a multirepo could have (whatever they may be, I can’t find them) while still making tracking, branching and merging across dependencies trivial.
- stepbeek 7y ago
  > Having separate repos only makes sense if they are maintained by separate teams with a well defined API Agree! I've found that a multirepo setup only works if there's an obvious boundary. Similar to SOA, distributed monoliths - or distributed monorepos - are problematic.
  - cryptica 7y ago
    > Having separate repos only makes sense if they are maintained by separate teams with a well defined API If some logic is modular then it should be in a separate repo. If it's not modular then it should be part of the same repo; just regular files inside directories... IMO, monorepos try to have it both ways but this doesn't make any sense; it encourages developers to write logic which is "somewhat modular". But "somewhat modular" is not modular at all. Either it's modular and it can easily be extracted out completely or it's not and then it should be embedded into the project's code. Code which is not modular should never need to be shared with other projects.
    - _se 7y ago
      This is a ridiculous oversimplification that doesn't scale at all to larger orgs. Every modular component in its own repo? That basically means creating a repo for every library. You could easily be managing thousands of repos with negative benefit. People seem to be afraid of using a single repo for some reason. It's really simple though. You should have one repo per "release cycle" that you have. Anything meant to be deployed together should be in the same repo. Things not meant to be deployed together should be in different repos. With the right tooling, that is what solves most problems.
      - bluGill 7y ago
        I have two things that are not deployed together. However there is common algorithms between the two that I'd rather not copy and paste maintain. Mono repo doesn't make sense because there is always something close to release and getting risk adverse (there are legal consequences to some bugs) so nothing can every be updated.
- stared 7y ago
  I saw some projects in which my instinct was "too many repos" (e.g. https://github.com/ActivityWatch/activitywatch https://github.com/ActivityWatch/activitywatch), exactly for the reasons you mentioned. However, in my case - as the project gets bigger, I appreciate it more. In the first place, it makes code reusable, and forces to encapsulated functionalities as much as possible. (Yes, having non-encapsulated code would be a nightmare). To get to the extreme, in the Node ecosystem code is pretty much structured as many, many small packages. (Bear in mind, that turning something into a package is a much stricter requirement that making it effectively a subfolder.)
- kdumont 7y ago
  > It allows to make changes in one go instead of having 3-4 PRs that all have to be merged at same time or otherwise the build breaks. Can't you solve this by pointing submodules in the parent repo to a specific commit SHA before the breaking change? Then update the pointer in parent repo when you've made the corresponding changes. I understand it's still added work compared to a monorepo, but seems like this could be a controlled way to handle submodules when necessary.
Joe8Bit 7y ago
The benefits of monorepos are (IME) all in people and organisation scale. For example, I've found it orders of magnitude easier to align tooling, quality, release and standards on a monorepo with 100's of 1000's of engineers working on it simultaneously. The point you've made is a good one though, the idea of monorepos at small scale has come up a lot recently and I'm not sure the arguments for it are as compelling (I don't see many compelling arguments _against_ it either mind you). What I _have_ seen though is the pain large orgs go through going from multi-to-mono repos, I think Twitter did it recently and it was pretty painful.
- koonsolo 7y ago
  Do they use git or some other versioning system?
  - Joe8Bit 7y ago
    Most orgs at that scale build their own tooling to make working with monorepos more ergonomic, in my experience. They're based on git (or other DVCS tools like Mercurial) but they allow you to easily do things like partial clones and company specific workflow tasks like submitting patches for review or tagging in weird and arcane ways. A small, illustrative example of this is the tooling that's built for Chrome's source code[0] with the depot_tools project [1]. [0]: https://chromium.googlesource.com/chromium/src/+/master/docs/mac_build_instructions.md#get-the-code https://chromium.googlesource.com/chromium/src/+/master/docs... [1]: https://chromium.googlesource.com/chromium/tools/depot_tools https://chromium.googlesource.com/chromium/tools/depot_tools
Plasmoid2000ad 7y ago
One size rarely fits all. Moving to mono repos has been great for my team for reducing build times and removing the overhead of understanding decades of cruft built up to make a huge mono repo manageable. But with turnover of both projects and people, and many projects not requiring active development, there's an awful lot of orphaned repos, with less than one person dedicated supporting them now. In my environment at least, you can just stop development. Packages need to be kept up to date, as vulnerabilities are discovered, using Azure Devops means we need to move along with changes to the build process. Infrastructure and Secrets policies changes come from outside, and require us to make changes. Making these kind of changes to a handful of repos is quite a bit of overhead. It's clear now, we went too small on the repo size, given the tools we have for the maintenance tasks we have to do. And people advocate for even smaller repos...
dan-robertson 7y ago
There’s a longer discussion here which may interest you: https://danluu.com/monorepo/ https://danluu.com/monorepo/ But if your code uses someone else’s libraries that don’t change or has super well defined interfaces between parts that never change or is going to be a mix of languages then maybe you won’t have issues or see and advantage of monorepos. If you have a library in a monorepo then it is much easier to improve the library in a breaking way and upgrade all its dependencies. If you have many repos then this work gets more distributed (so one person does it many times or lots of people who are less expert in the change to the library have to do it) or libraries are just not frequently updated and you get the same helper functions not being up streamed or you use different interfaces for the same library in different repos.

by stared 7y ago

edwintorok 7y ago

Once your code grows and you end up with too many repos you'll be yearning to get back to monorepos. It allows to make changes in one go instead of having 3-4 PRs that all have to be merged at same time or otherwise the build breaks. Having separate repos only makes sense if they are maintained by separate teams with a well defined API in-between that doesn't change often. If the same team maintains all repos then it is just overhead, especially if you end up with a 100 repos and you have significantly less than 100 engineers in your team. There are also potential build time savings when having a monorepo since you can do parallel builds across different subdirs without serialising at package boundaries.

cryptica 7y ago
I maintain such a project with many dependencies which I also maintain separately. Yes it can be a lot of extra work to publish but this argument is based on the assumption that dependencies need to be changed constantly and this assumption is wrong. There should be pressure to design dependencies in such a way that they are modular and don't need to be changed often - This unpleasantness associated with updating deep dependencies is good and necessary. These should rarely be updated, if they need to be updated often, then they were designed incorrectly. Many of the dependencies and sub-dependencies which I maintain are several years old. They are small enough and stable enough that they basically never need to change. If I need to update them (which is rare), I may need to do a cascading update of multiple dependents and this can take a while but if I have to do this big update only once every 6 months, it's totally worth having them in separate repos.
- beagle3 7y ago
  That depends on the scale of the project, of course - if you have thousands of engineers as MicroFaceGoog does, then it’a a daily, even hourly, occurrence for many dependencies. You are not MicroFaceGood, of course, so you only do that every 6 months. However, I what way are separate repos nicer for you? Personally, I settled on a monorepo with a “$dep-devel” branch for every dependency, which I occasionally have on multiple working trees out at the same time, and which I feel gives me all the benefit a multirepo could have (whatever they may be, I can’t find them) while still making tracking, branching and merging across dependencies trivial.
stepbeek 7y ago
> Having separate repos only makes sense if they are maintained by separate teams with a well defined API Agree! I've found that a multirepo setup only works if there's an obvious boundary. Similar to SOA, distributed monoliths - or distributed monorepos - are problematic.
- cryptica 7y ago
  > Having separate repos only makes sense if they are maintained by separate teams with a well defined API If some logic is modular then it should be in a separate repo. If it's not modular then it should be part of the same repo; just regular files inside directories... IMO, monorepos try to have it both ways but this doesn't make any sense; it encourages developers to write logic which is "somewhat modular". But "somewhat modular" is not modular at all. Either it's modular and it can easily be extracted out completely or it's not and then it should be embedded into the project's code. Code which is not modular should never need to be shared with other projects.
  - _se 7y ago
    This is a ridiculous oversimplification that doesn't scale at all to larger orgs. Every modular component in its own repo? That basically means creating a repo for every library. You could easily be managing thousands of repos with negative benefit. People seem to be afraid of using a single repo for some reason. It's really simple though. You should have one repo per "release cycle" that you have. Anything meant to be deployed together should be in the same repo. Things not meant to be deployed together should be in different repos. With the right tooling, that is what solves most problems.
    - bluGill 7y ago
      I have two things that are not deployed together. However there is common algorithms between the two that I'd rather not copy and paste maintain. Mono repo doesn't make sense because there is always something close to release and getting risk adverse (there are legal consequences to some bugs) so nothing can every be updated.
stared 7y ago
I saw some projects in which my instinct was "too many repos" (e.g. https://github.com/ActivityWatch/activitywatch https://github.com/ActivityWatch/activitywatch), exactly for the reasons you mentioned. However, in my case - as the project gets bigger, I appreciate it more. In the first place, it makes code reusable, and forces to encapsulated functionalities as much as possible. (Yes, having non-encapsulated code would be a nightmare). To get to the extreme, in the Node ecosystem code is pretty much structured as many, many small packages. (Bear in mind, that turning something into a package is a much stricter requirement that making it effectively a subfolder.)
kdumont 7y ago
> It allows to make changes in one go instead of having 3-4 PRs that all have to be merged at same time or otherwise the build breaks. Can't you solve this by pointing submodules in the parent repo to a specific commit SHA before the breaking change? Then update the pointer in parent repo when you've made the corresponding changes. I understand it's still added work compared to a monorepo, but seems like this could be a controlled way to handle submodules when necessary.

Joe8Bit 7y ago

The benefits of monorepos are (IME) all in people and organisation scale. For example, I've found it orders of magnitude easier to align tooling, quality, release and standards on a monorepo with 100's of 1000's of engineers working on it simultaneously. The point you've made is a good one though, the idea of monorepos at small scale has come up a lot recently and I'm not sure the arguments for it are as compelling (I don't see many compelling arguments _against_ it either mind you). What I _have_ seen though is the pain large orgs go through going from multi-to-mono repos, I think Twitter did it recently and it was pretty painful.

koonsolo 7y ago
Do they use git or some other versioning system?
- Joe8Bit 7y ago
  Most orgs at that scale build their own tooling to make working with monorepos more ergonomic, in my experience. They're based on git (or other DVCS tools like Mercurial) but they allow you to easily do things like partial clones and company specific workflow tasks like submitting patches for review or tagging in weird and arcane ways. A small, illustrative example of this is the tooling that's built for Chrome's source code[0] with the depot_tools project [1]. [0]: https://chromium.googlesource.com/chromium/src/+/master/docs/mac_build_instructions.md#get-the-code https://chromium.googlesource.com/chromium/src/+/master/docs... [1]: https://chromium.googlesource.com/chromium/tools/depot_tools https://chromium.googlesource.com/chromium/tools/depot_tools

Plasmoid2000ad 7y ago

One size rarely fits all. Moving to mono repos has been great for my team for reducing build times and removing the overhead of understanding decades of cruft built up to make a huge mono repo manageable. But with turnover of both projects and people, and many projects not requiring active development, there's an awful lot of orphaned repos, with less than one person dedicated supporting them now. In my environment at least, you can just stop development. Packages need to be kept up to date, as vulnerabilities are discovered, using Azure Devops means we need to move along with changes to the build process. Infrastructure and Secrets policies changes come from outside, and require us to make changes. Making these kind of changes to a handful of repos is quite a bit of overhead. It's clear now, we went too small on the repo size, given the tools we have for the maintenance tasks we have to do. And people advocate for even smaller repos...

dan-robertson 7y ago

There’s a longer discussion here which may interest you: https://danluu.com/monorepo/ https://danluu.com/monorepo/ But if your code uses someone else’s libraries that don’t change or has super well defined interfaces between parts that never change or is going to be a mix of languages then maybe you won’t have issues or see and advantage of monorepos. If you have a library in a monorepo then it is much easier to improve the library in a breaking way and upgrade all its dependencies. If you have many repos then this work gets more distributed (so one person does it many times or lots of people who are less expert in the change to the library have to do it) or libraries are just not frequently updated and you get the same helper functions not being up streamed or you use different interfaces for the same library in different repos.