7 ms·

There are real, tangible benefits to monorepos. And real, tangible downsides, too. Like everything else, it's a tradeoff. I'm sure people advocating for them h

There are real, tangible benefits to monorepos. And real, tangible downsides, too. Like everything else, it's a tradeoff. I'm sure people advocating for them have brought up some benefits. But in case they haven't, here's an example I recently encountered at work: for an application we are developing, we needed functionality X. We knew that we'd need X in other projects, too, so we made it a library. I published it to our internal package registry and referenced it in the project. So far, so good. Turns out, feature X wasn't as well understood as we thought it was and we are still changing things around all the time. That meant: * changing the library * cutting a new release and publishing it * updating dependencies in the project and using the new feature Since this was happening much too often for what was often a one or two line change, I moved the library into the project repo and build it as part of the project pipeline. This removes so much friction around these changes, I find it makes a significant difference. Eventually, I'll move the library back into its own repository, but for now the monorepo is the right choice in my eyes.

Archelaos 3y ago
> Turns out, feature X wasn't as well understood as we thought it was and we are still changing things around all the time. If you publish something as a library for someone else, you should generally commit to supporting the public interface of that library as if it were a long-term support release. Exactly how long that should be depends on many factors, of course. But as a rule of thumb, I would recommend supporting this library version for at least two years. If that doesn't make sense, don't release it or look for a subset, where it makes sense.
- The_Colonel 3y ago
  This is exactly the friction/overhead the other person means and what monorepo works around. (but also keep in mind this is an internal library for likely a couple of internal apps, supporting BC for 2 years is in most cases unwarranted)
  - Repulsion9513 3y ago
    Monorepo doesn't "work around" it, it means you have to support everything (including migrating/BC) on the current version, instead of having to support old stuff on the old version.
    - actionfromafar 3y ago
      You can “fork” inside the monorepo too, just take a copy.
    - sjducb 3y ago
      You can see where the “library” is used. Often the bit you want to change is only used by your project so you can change it freely. If someone else is using it you know who to talk to about changing it.
- surajrmal 3y ago
  This guidance is far too specific to make sense in the general case. Not every scenario has the same constraints. If I had to pay a 2 year maintenance cost for every library I write and wanted to reuse between projects I'd never end up writing it. If you can migrate all library clients over to the new version, there is no reason to maintain old versions.
  - Archelaos 3y ago
    I would not call this a "library", but a set of interrelated project with "shared moduls" instead. With a "library" I mean a piece of code where the publisher does know who is using it for what exact purpose. Of couse we could now fight over a proper general definition for "library", but this is how I would use the term in this context, because thee long term aspect is what really makes outsourcing such code meaningful -- in contrast to the parents observation that it does not make sense to incorporate foreign code that constantly changes.
    - mewpmewp2 3y ago
      The argument then is also that monorepo allows you to use this as "shared modules" instead of published "libraries". The point is you have some code that you want to reuse, and either within a monorepo or between multiple repositories.
      - Archelaos 3y ago
        The question is, when it is wise to do so. Changing the public interface of a "shared module" or published "library" comes with a liability. Code reuse is almost a no brainer in the case of a standard library that rarely or never changes. Using a permanently changing modul whose modifications provide no real benefits for your project makes no economic sense. The sweetspot is somewhere in between. What the parent actually did when copying the library/module code was creating a sort of a a long-term support version for himself maintained by himself. His cost-benefit analysis told him that this is better than always trying to keep up with the changes of the library/modul. My comment was aimed at setting the bar rather high for when to share reusable libraries. And even for my own modules I rather prefer copying code instead of reusing it in different contexts. There is a reason why the contexts differ and it is typically easier and more economic to adress this differences in its particular place instead of preparing for them in shared code. The more up-stream a library/module is, the more general it must be, but the less liable to change it should be.
- ownagefool 3y ago
  It depends. If you drop a library for internal usage only and you want to change the contract, and the tests, library, and api are all in the same repo, you just change them in a single PR and that's done. This works because the single commit hash contains all information. It requires people put their code in the monorepo and hookup their tests, but assuming they do, you can build a reasonable degree of confidence on a green build. Once you separate the library different repo, and let people consume that as something in their own repos, you need to do the versioning dance as you have no idea if their still is still working. A lot of people go for the latter because it contextually allows them to ignore the rest of the stack, and there are some pros to doing this, but testing, deploying, versioning, etc, all become more difficult, and that's something struggle with. Thus, unless you have a crap load of code / commits, monorepos arguably have more advantages than disadvantages.
  - bluGill 3y ago
    If the same problem doesn't exist in a monorepo in a different form then your project is not large enough to be in this discussion in the first place.
    - ownagefool 3y ago
      It's actually just trade offs that we should be able to have professional discussions about, otherwise your advocating for "always use a monorepo" unless your library is for public consumption. The reality is, at some point, having repos that are hundreds of GB with hundreds of active PRs also has its own downsides that requires tooling and workflows to combat. Meanwhile, splitting it all up introduces integration problems. It's definitely pick your poison, though specific requirements and circumstances make specific paths more or less potent.
      - bluGill 3y ago
        The problem is many people arguing for a mono repo feel like the size of their project means breaking into multiple repos mean put each function in a separate repository - which is obviously absurd. It is good if your project is that small, but you also are not facing the same pains as large projects are.
- actionfromafar 3y ago
  This is orthogonal to mono-repo vs not mono-repo and indeed distribution mechanism.
- Cu3PO42 3y ago
  This isn't wrong, but somewhat orthogonal. Yes, I "published" it on our package registry, but it wasn't public. Access was still scoped to the team developing the application I was speaking about.
- JackSlateur 3y ago
  This is exactly why monorepo is advised : you do not need to keep and maintain useless code for years. Instead, you can "git grep", then write and submit MR to quickly change all users. I have never encounter this use case, tho
rrr_oh_man 3y ago
> Since this was happening much too often for what was often a one or two line change The famous 5 minute project. Happens in coding, car repair, and interstellar travel.
Arch-TK 3y ago
I don't think this is really advocacy for monorepo but rather advocacy against premature solidification of abstractions. Whenever I encounter a situation like this, the first step I take is to write the code without any API or interface boundaries to worry about. Then if I still feel it would be a good idea to separate it, I will separate it while keeping it in the source tree (this isn't really vendoring or monorepo, it's just abstraction into a module). Once this thing has survived a few rounds of revisions and you're ready to use it in another project, copy paste it first. Then once two projects have both had a chance to exercise the abstraction, then you factor it out into something totally separate This process requires that you are able to work on rewrites and refactorings without having to beg. But it's truly the best way to get from an idea of a library to an actually well designed library. In some situations it might be you already know what you want and can separate it out from the beginning, but if you are coming up with anything remotely novel, then you want to follow this process. A monorepo will still contain libraries like these. It will also contain vendored code (although monorepos help reduce the amount of duplicate vendored bits of code). Making such changes will be easier in a monorepo once your abstraction starts to solidify. But the process you should go through should still be similar. The reason for not having two things depend on your wobbly abstraction too early is simply that it will inevitability lead to you prematurely solidifying uncertain abstractions.
- bluGill 3y ago
  Most people arguing one way or the other are in such a small code base that they don't face the problems the argument is really about in the first place. If you only have a a million lines of code then by all means use a monorepo as you probably aren't facing the downsides that make people reach for a multi-repo solution anyway.
efxhoy 3y ago
I find this too at work all the time. People are very quick to branch off some “independent” feature into its own service or lib and repo. Great for them when building, tiny repos do have that greenfield clean slate feeling, but awful when you realize the integration points aren’t as simple and well understood as they first thought.
Repulsion9513 3y ago
That's not a monorepo.
BlackFly 3y ago
The real problems with monorepos are that most of the benefits vanish as the scale increases unless you invest into building more monorepo tooling. In your particular case, if your library becomes too popular, your one or two line implementation detail changes ripple out and trigger rebuilds of too many downstreams, many of which will have flakey tests and fail your MR. If most users are not actually depending on that functionality, or if you simply are doing semver properly, then you could avoid rebuilding those dependencies. Eventually the builds take too much time and CI rejects the pipelines or you continuously bump the build timeout but wait longer and longer for your changes to go live. You can solve these problems, if you invest in more monorepo tooling. Similarly, once you are too popular a library in a monorepo, you will never do any atomic breaking API changes since it would require updating too many downstreams. Instead you will fake version it: add a new API, migrate users to it in multiple commits, delete the old version. Some of these migrations run out of steam midway in the biggest phase: phase 2. This approach does have the benefit of forcing the upstream author to make the two versions of the API co-exist. Of course I am talking about scales where real limits start to break down. When your codebase is larger than your ram, an index won't fit into RAM anymore and every code search requires disk or network I/O. Eventually your repository doesn't fit on disk anymore and you interact with it with a special IDE and only download things you start editing or perform sparse checkouts in the first place so discoverability is again a problem. Edit: of course some problems crop up sooner than hard limits are reached, like the flakey test issue I mentioned as well as visibility and control of changes to actual maintainers.
- xign 3y ago
  I think in a centralized environment (workplace), it could be argued that immediately triggering all the build failures and having good hygiene in cleaning them up is actually not a bad thing. It really depends on how that's set up. And how is sparse checkout worse for discoverability? With multiple repos it's even harder to find what you want sometimes if you are talking about 100's of random repos that aren't organized well.
  - bluGill 3y ago
    > I think in a centralized environment (workplace), it could be argued that immediately triggering all the build failures and having good hygiene in cleaning them up is actually not a bad thing. In abstract I agree. However when I'm trying to get my code working having test failures in code that isn't even related to the problem I'm working on is annoying and I can't switch tasks to work on this new failure when the current code isn't working either.
hk__2 3y ago
> for an application we are developing, we needed functionality X. We knew that we'd need X in other projects, too, so we made it a library This is the mistake: you should make it a library when you do need the functionality, not when you "know" you "would need" it.
- bluGill 3y ago
  I disagree. If you will really need it, then just make it a library. This is not 1960 when we were still figuring out software. You are not writing something new anymore, the problem has been solved enough time that you can look around and figure out what will really be needed in other projects and what will not. This is something your architects should be doing as getting it right is important and you will regret not doing so (when you have 10 different implementations of the same thing all slightly different) Now I will agree that developers are often wrong. We often think something we write is important than it really is and so make it reusable when it never will be. We often think something won't be reused when it really would be. This is a hard problem, but that doesn't mean you should not think about it and work hard to get things right.
wongarsu 3y ago
Obviously that depends on your language's tooling and your company's processes, but directly depending on (a specific branch or commit of) a git repository rather than a release is a great in-between solution that allows you to have separate repositories while still allowing fast iteration. Obviously you want to switch back to a release model once the feature stabilizes, to have all the advantages your internal package registry brings you.

Archelaos 3y ago

> Turns out, feature X wasn't as well understood as we thought it was and we are still changing things around all the time. If you publish something as a library for someone else, you should generally commit to supporting the public interface of that library as if it were a long-term support release. Exactly how long that should be depends on many factors, of course. But as a rule of thumb, I would recommend supporting this library version for at least two years. If that doesn't make sense, don't release it or look for a subset, where it makes sense.

The_Colonel 3y ago
This is exactly the friction/overhead the other person means and what monorepo works around. (but also keep in mind this is an internal library for likely a couple of internal apps, supporting BC for 2 years is in most cases unwarranted)
- Repulsion9513 3y ago
  Monorepo doesn't "work around" it, it means you have to support everything (including migrating/BC) on the current version, instead of having to support old stuff on the old version.
  - actionfromafar 3y ago
    You can “fork” inside the monorepo too, just take a copy.
  - sjducb 3y ago
    You can see where the “library” is used. Often the bit you want to change is only used by your project so you can change it freely. If someone else is using it you know who to talk to about changing it.
surajrmal 3y ago
This guidance is far too specific to make sense in the general case. Not every scenario has the same constraints. If I had to pay a 2 year maintenance cost for every library I write and wanted to reuse between projects I'd never end up writing it. If you can migrate all library clients over to the new version, there is no reason to maintain old versions.
- Archelaos 3y ago
  I would not call this a "library", but a set of interrelated project with "shared moduls" instead. With a "library" I mean a piece of code where the publisher does know who is using it for what exact purpose. Of couse we could now fight over a proper general definition for "library", but this is how I would use the term in this context, because thee long term aspect is what really makes outsourcing such code meaningful -- in contrast to the parents observation that it does not make sense to incorporate foreign code that constantly changes.
  - mewpmewp2 3y ago
    The argument then is also that monorepo allows you to use this as "shared modules" instead of published "libraries". The point is you have some code that you want to reuse, and either within a monorepo or between multiple repositories.
    - Archelaos 3y ago
      The question is, when it is wise to do so. Changing the public interface of a "shared module" or published "library" comes with a liability. Code reuse is almost a no brainer in the case of a standard library that rarely or never changes. Using a permanently changing modul whose modifications provide no real benefits for your project makes no economic sense. The sweetspot is somewhere in between. What the parent actually did when copying the library/module code was creating a sort of a a long-term support version for himself maintained by himself. His cost-benefit analysis told him that this is better than always trying to keep up with the changes of the library/modul. My comment was aimed at setting the bar rather high for when to share reusable libraries. And even for my own modules I rather prefer copying code instead of reusing it in different contexts. There is a reason why the contexts differ and it is typically easier and more economic to adress this differences in its particular place instead of preparing for them in shared code. The more up-stream a library/module is, the more general it must be, but the less liable to change it should be.
ownagefool 3y ago
It depends. If you drop a library for internal usage only and you want to change the contract, and the tests, library, and api are all in the same repo, you just change them in a single PR and that's done. This works because the single commit hash contains all information. It requires people put their code in the monorepo and hookup their tests, but assuming they do, you can build a reasonable degree of confidence on a green build. Once you separate the library different repo, and let people consume that as something in their own repos, you need to do the versioning dance as you have no idea if their still is still working. A lot of people go for the latter because it contextually allows them to ignore the rest of the stack, and there are some pros to doing this, but testing, deploying, versioning, etc, all become more difficult, and that's something struggle with. Thus, unless you have a crap load of code / commits, monorepos arguably have more advantages than disadvantages.
- bluGill 3y ago
  If the same problem doesn't exist in a monorepo in a different form then your project is not large enough to be in this discussion in the first place.
  - ownagefool 3y ago
    It's actually just trade offs that we should be able to have professional discussions about, otherwise your advocating for "always use a monorepo" unless your library is for public consumption. The reality is, at some point, having repos that are hundreds of GB with hundreds of active PRs also has its own downsides that requires tooling and workflows to combat. Meanwhile, splitting it all up introduces integration problems. It's definitely pick your poison, though specific requirements and circumstances make specific paths more or less potent.
    - bluGill 3y ago
      The problem is many people arguing for a mono repo feel like the size of their project means breaking into multiple repos mean put each function in a separate repository - which is obviously absurd. It is good if your project is that small, but you also are not facing the same pains as large projects are.
actionfromafar 3y ago
This is orthogonal to mono-repo vs not mono-repo and indeed distribution mechanism.
Cu3PO42 3y ago
This isn't wrong, but somewhat orthogonal. Yes, I "published" it on our package registry, but it wasn't public. Access was still scoped to the team developing the application I was speaking about.
JackSlateur 3y ago
This is exactly why monorepo is advised : you do not need to keep and maintain useless code for years. Instead, you can "git grep", then write and submit MR to quickly change all users. I have never encounter this use case, tho

rrr_oh_man 3y ago

> Since this was happening much too often for what was often a one or two line change The famous 5 minute project. Happens in coding, car repair, and interstellar travel.

Arch-TK 3y ago

I don't think this is really advocacy for monorepo but rather advocacy against premature solidification of abstractions. Whenever I encounter a situation like this, the first step I take is to write the code without any API or interface boundaries to worry about. Then if I still feel it would be a good idea to separate it, I will separate it while keeping it in the source tree (this isn't really vendoring or monorepo, it's just abstraction into a module). Once this thing has survived a few rounds of revisions and you're ready to use it in another project, copy paste it first. Then once two projects have both had a chance to exercise the abstraction, then you factor it out into something totally separate This process requires that you are able to work on rewrites and refactorings without having to beg. But it's truly the best way to get from an idea of a library to an actually well designed library. In some situations it might be you already know what you want and can separate it out from the beginning, but if you are coming up with anything remotely novel, then you want to follow this process. A monorepo will still contain libraries like these. It will also contain vendored code (although monorepos help reduce the amount of duplicate vendored bits of code). Making such changes will be easier in a monorepo once your abstraction starts to solidify. But the process you should go through should still be similar. The reason for not having two things depend on your wobbly abstraction too early is simply that it will inevitability lead to you prematurely solidifying uncertain abstractions.

bluGill 3y ago
Most people arguing one way or the other are in such a small code base that they don't face the problems the argument is really about in the first place. If you only have a a million lines of code then by all means use a monorepo as you probably aren't facing the downsides that make people reach for a multi-repo solution anyway.

efxhoy 3y ago

I find this too at work all the time. People are very quick to branch off some “independent” feature into its own service or lib and repo. Great for them when building, tiny repos do have that greenfield clean slate feeling, but awful when you realize the integration points aren’t as simple and well understood as they first thought.

Repulsion9513 3y ago

That's not a monorepo.

BlackFly 3y ago

The real problems with monorepos are that most of the benefits vanish as the scale increases unless you invest into building more monorepo tooling. In your particular case, if your library becomes too popular, your one or two line implementation detail changes ripple out and trigger rebuilds of too many downstreams, many of which will have flakey tests and fail your MR. If most users are not actually depending on that functionality, or if you simply are doing semver properly, then you could avoid rebuilding those dependencies. Eventually the builds take too much time and CI rejects the pipelines or you continuously bump the build timeout but wait longer and longer for your changes to go live. You can solve these problems, if you invest in more monorepo tooling. Similarly, once you are too popular a library in a monorepo, you will never do any atomic breaking API changes since it would require updating too many downstreams. Instead you will fake version it: add a new API, migrate users to it in multiple commits, delete the old version. Some of these migrations run out of steam midway in the biggest phase: phase 2. This approach does have the benefit of forcing the upstream author to make the two versions of the API co-exist. Of course I am talking about scales where real limits start to break down. When your codebase is larger than your ram, an index won't fit into RAM anymore and every code search requires disk or network I/O. Eventually your repository doesn't fit on disk anymore and you interact with it with a special IDE and only download things you start editing or perform sparse checkouts in the first place so discoverability is again a problem. Edit: of course some problems crop up sooner than hard limits are reached, like the flakey test issue I mentioned as well as visibility and control of changes to actual maintainers.

xign 3y ago
I think in a centralized environment (workplace), it could be argued that immediately triggering all the build failures and having good hygiene in cleaning them up is actually not a bad thing. It really depends on how that's set up. And how is sparse checkout worse for discoverability? With multiple repos it's even harder to find what you want sometimes if you are talking about 100's of random repos that aren't organized well.
- bluGill 3y ago
  > I think in a centralized environment (workplace), it could be argued that immediately triggering all the build failures and having good hygiene in cleaning them up is actually not a bad thing. In abstract I agree. However when I'm trying to get my code working having test failures in code that isn't even related to the problem I'm working on is annoying and I can't switch tasks to work on this new failure when the current code isn't working either.

hk__2 3y ago

> for an application we are developing, we needed functionality X. We knew that we'd need X in other projects, too, so we made it a library This is the mistake: you should make it a library when you do need the functionality, not when you "know" you "would need" it.

bluGill 3y ago
I disagree. If you will really need it, then just make it a library. This is not 1960 when we were still figuring out software. You are not writing something new anymore, the problem has been solved enough time that you can look around and figure out what will really be needed in other projects and what will not. This is something your architects should be doing as getting it right is important and you will regret not doing so (when you have 10 different implementations of the same thing all slightly different) Now I will agree that developers are often wrong. We often think something we write is important than it really is and so make it reusable when it never will be. We often think something won't be reused when it really would be. This is a hard problem, but that doesn't mean you should not think about it and work hard to get things right.

wongarsu 3y ago

Obviously that depends on your language's tooling and your company's processes, but directly depending on (a specific branch or commit of) a git repository rather than a release is a great in-between solution that allows you to have separate repositories while still allowing fast iteration. Obviously you want to switch back to a release model once the feature stabilizes, to have all the advantages your internal package registry brings you.