4 ms·

This thread is reminding me of a prior one about complexity merchants. I am seeing a lot of sentiment that there is somehow a technical sacrifice by moving to a

This thread is reminding me of a prior one about complexity merchants. I am seeing a lot of sentiment that there is somehow a technical sacrifice by moving to a monorepo. This is absolutely ludicrous unless you fail to grasp the power of a hierarchical file system. I don't see how a big mess like CI/CD is made easier by spreading it out to more points of configuration. To me the whole point of a monorepo is atomic commits for the whole org. The power of this is really hard to overstate when you are trying to orchestrate the efforts of lots of developers - contrary to many claims. Rebasing in one repo and having one big meeting is a hell of a lot easier than doing it N times. Even if the people on the team hate each other and refuse to directly collaborate. I still don't see the reason to not monorepo. In this scenario, the monorepo becomes a useful management and HR tool.

cmrdporcupine 1y ago
The push to fragmentation and atomism is so strong with this generation of devs. The obsession with microservices, dozens of small repositories, splitting everything up from fear of "monoliths." What they're doing is creating a mass of complexity that is turning org-chart problems into future technical ones and at the same time not recognizing the intrinsic internal dependencies of the software systems they're building. Luckily my current job is not like this, but the last one was, and I couldn't believe the wasted hours spent doing things as simple as updating the fields in a protobuf schema file.
- bluGill 1y ago
  That push to fragmentation is in large part because of hard lessons learned from the problems of a monolith. The answer is IMO somewhere in between. Microservices can get too tiny and thus the system becomes impossible to understand. However a monolith is impossible to understand as well. The real problem is you need good upfront architecture to figure out how the whole system fits together. However that is really hard to get right (and Agile discourages it - which is right for small projects where those architects add complex things to mitigate problems you will never have)
  - jyounker 1y ago
    Monolith != Monorepo. They're independent concepts.
    - MiscCompFacts 1y ago
      I work at a company that has several micro services and a backend all in one monorepo. For some anecdata.
    - datadrivenangel 1y ago
      The tooling defaults around Github encourage having one thing per repo.
      - layer8 1y ago
        A monolith can (and often does) consist of separately versioned libraries.
  - cmrdporcupine 1y ago
    https://askastaffengineer.com/ https://askastaffengineer.com/
- mherkender 1y ago
  Large software projects cycle back and forth between fragmentation and defragmentation. There is no right answer, only what's right for each project at the time. Relevant xkcd: https://xkcd.com/2044/ https://xkcd.com/2044/
- ecoffey 1y ago
  In my experience microservices are easier to manage and understand when organized in a monorepo.
  - layer8 1y ago
    That indicates a strong coupling between those microservices.
    - ecoffey 1y ago
      Even loose coupling is still coupling. For the things that have to be coupled having the code organized in the same place, being able to easily read the source for “the other side”, make a change and verify that dependees test still pass, etc is immensely powerful.
    - folkrav 1y ago
      I like using the term “distributed monolith” for those systems with very tightly coupled microservices.
gengstrand 1y ago
It is true that there are significant benefits to monorepo but it comes at a cost. Managing a monorepo is more expensive than polyrepo. For the details behind that claim, check out https://www.exploravention.com/blogs/monorepo/ https://www.exploravention.com/blogs/monorepo/ The question is this. Do the costs of monorepo justify the benefits for your situation? The answer is not always yes.
- codethief 1y ago
  Note that the costs depend on the scale and size of the monorepo, and a polyrepo is not without costs either (which people often like to forget or ignore because they are less visible). I spent some years at small to mid-sized companies (~30-100 devs) that would have profited from a monorepo. However, people were in the habit of splitting repositories every other month. Sometimes a single team would own a given repository, but more often than not several teams would contribute to each repository. I have serious PTSD from that time. Every single pipeline in every repo worked differently, tools were different, setup, scripts and commands were different. In some repositories you could trust the CI pipeline, in others you absolutely couldn't. CI performance gains in one repo wouldn't translate to another. And of course you often still had some tech debt lying around from when repositories got split and people forgot to clean up thoroughly. Uggh. Now, to be fair, people did put quite a bit of effort into their setups and pipelines and all that, and it wasn't that the outcome of that effort in and by itself was totally bad. Not at all. But everyone did things differently. And often teams just re-did the work other people had already done & solved – there was a ton of overhead. Finally, the worst part were inter-repository dependencies. People argued coupling would be loose and all that, so we could easily split repos, but in reality there were so many implicit dependencies because ultimately all repositories made up one big application. Merge requests had to be coordinated. One repo imported files from another. Later a CI pipeline in one repo triggered a pipeline in another repo… This brings me to another problem with polyrepos: They harm discoverability and cause certain dependencies not to be encoded in code. In a monorepo, in contrast, when people wonder where they can find X, or whether anyone uses or depends on Y, the answer is usually only a `ripgrep` or `git grep` away. In a polyrepo, however, they need to know where to look first. Born from all these experiences, my mantra has been: Split repos only if you really have to and have a very good reason! (Entirely different applications or ecosystems; different parts of the code owned by different & independent teams; …)
yencabulator 1y ago
> To me the whole point of a monorepo is atomic commits for the whole org. The belief that a monorepo makes a change somehow more atomic is one of the traps. From the article: > The greatest power and biggest lie of the monorepo is that it is possible to make atomic commits across your entire codebase. [...] > Your monorepo now contains many different deployable artifacts that deploy at different times. It is also technically possible to make, for example, a breaking change to a service’s interface, a service’s implementation, and the service’s clients all in one PR. However, this PR will break when you deploy it because you do not deploy your service and all of its clients atomically. While this is also possible in a world with many repositories, the requirement to do this change in multiple pull requests is often enough to remind engineers that breaking changes to a service contract are not safe to make. > Your users must understand that your deployment system operates asynchronously with respect to what happens in the monorepo. Its primary interaction with the monorepo is to go and pick up the “latest” build artifacts for a particular service; everything else happens on timetables that are potentially not under your control and can happen arbitrarily far in the future. > A common CI job in a monorepo is to validate service contracts and make sure that they are not broken unless the author deliberately intended to do so, and they are required to provide a justification as to why such a change is OK.
- crazygringo 1y ago
  That's confusing two different things, though. A monorepo does make changes atomic in the code. There's no trap there. You're talking about deployment, and yes when deployment is staggered, then obviously all atomic changes need to be backward-compatible, or else be very carefully orchestrated. But that doesn't have anything to do with monorepo vs polyrepo. That's just staggered deployment. You have to deal with backwards compatibility in both cases. But at least with the monorepo you can see and track and merge all the changes related to a feature in one place, and be able to roll them back in one place.
  - yencabulator 1y ago
    There's no such thing as when deployment is staggered. It's a distributed problem, so by definition it is not synchronous. (Or you turn services off for the duration of the deploy. Most companies do not want that these days.) Also, you're missing this part of the article: > While this is also possible in a world with many repositories, the requirement to do this change in multiple pull requests is often enough to remind engineers that breaking changes to a service contract are not safe to make.
marcosdumay 1y ago
> The power of this is really hard to overstate A really useful heuristics when you are designing programming environments is: the more power you give to a team of developers, the more problems you will have. Technically, atomic commits are not more power, they are less. But it does empower the team to work with bad interfaces. And that's a power that creates problems.
jayd16 1y ago
Here's the issue... Being able to track commits across multiple projects is just a nice to have. Does it even significantly increase your ability to track dependencies or trigger down stream tests? You can track that stuff with multi-repo automations. Maybe it helps a bit but it's not free and its not complete. Your deploys don't become atomic. Your builds don't become atomic. At best you get to tell yourself you can version a bit more loosely. Mono-repos themselves do not scale easily. Inherently its a harder technical problem to solve. You need to toss git and find something better, which is not easy. Its work. Its so much work that it is incredibly clear that you've never experienced it yourself.
- gugagore 1y ago
  Do you have go-to multi repo automations and other tools for operating across multiple repos?
nobodywillobsrv 1y ago
While things like git submodules have problems, are they not sort of the best of both worlds camp? I never quite got the arguments/grumblings about this stuff but perhaps I do not understand what friction people are hitting up agains. It feels to me the problems of multiple repositories is the same problem of having to manage versions of dependencies like in pip or something that you do not even own. Perhaps people are using tools that mean they are not able to search/click through their multi repo setup even though the code connects?
- Too 1y ago
  Yes and No. One top level "superrepo" is the best of both worlds. You don't break the limits of git, you can include third party repos, you can seal off access to some secret repos. All while still giving benefit of everything checked out in one place. Tooling for flattening the git log exists and is relatively cheap to build yourself, It's easy to design CI that runs both big and small jobs in the superrepo and the subcomponents. Nested submodules = Game over. Don't ever do it. This is the worst of both worlds. Common components included twice or more need to be aligned to exact same revision, without help of semantic versioning. Components far down the dependency tree need to be uplifted recursively through a chain of submodules, resulting in 10 commits with no common CI for one tiny change.
wocram 1y ago
Just today I saw someone try to set up a new project and stumble over setting it up as it's own repository, instead of just having a central place to add it. There's also a vicious feedback loop of separating projects across repos creating a huge impediment for touching anything outside of the project you're directly working in.

cmrdporcupine 1y ago

The push to fragmentation and atomism is so strong with this generation of devs. The obsession with microservices, dozens of small repositories, splitting everything up from fear of "monoliths." What they're doing is creating a mass of complexity that is turning org-chart problems into future technical ones and at the same time not recognizing the intrinsic internal dependencies of the software systems they're building. Luckily my current job is not like this, but the last one was, and I couldn't believe the wasted hours spent doing things as simple as updating the fields in a protobuf schema file.

bluGill 1y ago
That push to fragmentation is in large part because of hard lessons learned from the problems of a monolith. The answer is IMO somewhere in between. Microservices can get too tiny and thus the system becomes impossible to understand. However a monolith is impossible to understand as well. The real problem is you need good upfront architecture to figure out how the whole system fits together. However that is really hard to get right (and Agile discourages it - which is right for small projects where those architects add complex things to mitigate problems you will never have)
- jyounker 1y ago
  Monolith != Monorepo. They're independent concepts.
  - MiscCompFacts 1y ago
    I work at a company that has several micro services and a backend all in one monorepo. For some anecdata.
  - datadrivenangel 1y ago
    The tooling defaults around Github encourage having one thing per repo.
    - layer8 1y ago
      A monolith can (and often does) consist of separately versioned libraries.
- cmrdporcupine 1y ago
  https://askastaffengineer.com/ https://askastaffengineer.com/
mherkender 1y ago
Large software projects cycle back and forth between fragmentation and defragmentation. There is no right answer, only what's right for each project at the time. Relevant xkcd: https://xkcd.com/2044/ https://xkcd.com/2044/
ecoffey 1y ago
In my experience microservices are easier to manage and understand when organized in a monorepo.
- layer8 1y ago
  That indicates a strong coupling between those microservices.
  - ecoffey 1y ago
    Even loose coupling is still coupling. For the things that have to be coupled having the code organized in the same place, being able to easily read the source for “the other side”, make a change and verify that dependees test still pass, etc is immensely powerful.
  - folkrav 1y ago
    I like using the term “distributed monolith” for those systems with very tightly coupled microservices.

gengstrand 1y ago

It is true that there are significant benefits to monorepo but it comes at a cost. Managing a monorepo is more expensive than polyrepo. For the details behind that claim, check out https://www.exploravention.com/blogs/monorepo/ https://www.exploravention.com/blogs/monorepo/ The question is this. Do the costs of monorepo justify the benefits for your situation? The answer is not always yes.

codethief 1y ago
Note that the costs depend on the scale and size of the monorepo, and a polyrepo is not without costs either (which people often like to forget or ignore because they are less visible). I spent some years at small to mid-sized companies (~30-100 devs) that would have profited from a monorepo. However, people were in the habit of splitting repositories every other month. Sometimes a single team would own a given repository, but more often than not several teams would contribute to each repository. I have serious PTSD from that time. Every single pipeline in every repo worked differently, tools were different, setup, scripts and commands were different. In some repositories you could trust the CI pipeline, in others you absolutely couldn't. CI performance gains in one repo wouldn't translate to another. And of course you often still had some tech debt lying around from when repositories got split and people forgot to clean up thoroughly. Uggh. Now, to be fair, people did put quite a bit of effort into their setups and pipelines and all that, and it wasn't that the outcome of that effort in and by itself was totally bad. Not at all. But everyone did things differently. And often teams just re-did the work other people had already done & solved – there was a ton of overhead. Finally, the worst part were inter-repository dependencies. People argued coupling would be loose and all that, so we could easily split repos, but in reality there were so many implicit dependencies because ultimately all repositories made up one big application. Merge requests had to be coordinated. One repo imported files from another. Later a CI pipeline in one repo triggered a pipeline in another repo… This brings me to another problem with polyrepos: They harm discoverability and cause certain dependencies not to be encoded in code. In a monorepo, in contrast, when people wonder where they can find X, or whether anyone uses or depends on Y, the answer is usually only a `ripgrep` or `git grep` away. In a polyrepo, however, they need to know where to look first. Born from all these experiences, my mantra has been: Split repos only if you really have to and have a very good reason! (Entirely different applications or ecosystems; different parts of the code owned by different & independent teams; …)

yencabulator 1y ago

> To me the whole point of a monorepo is atomic commits for the whole org. The belief that a monorepo makes a change somehow more atomic is one of the traps. From the article: > The greatest power and biggest lie of the monorepo is that it is possible to make atomic commits across your entire codebase. [...] > Your monorepo now contains many different deployable artifacts that deploy at different times. It is also technically possible to make, for example, a breaking change to a service’s interface, a service’s implementation, and the service’s clients all in one PR. However, this PR will break when you deploy it because you do not deploy your service and all of its clients atomically. While this is also possible in a world with many repositories, the requirement to do this change in multiple pull requests is often enough to remind engineers that breaking changes to a service contract are not safe to make. > Your users must understand that your deployment system operates asynchronously with respect to what happens in the monorepo. Its primary interaction with the monorepo is to go and pick up the “latest” build artifacts for a particular service; everything else happens on timetables that are potentially not under your control and can happen arbitrarily far in the future. > A common CI job in a monorepo is to validate service contracts and make sure that they are not broken unless the author deliberately intended to do so, and they are required to provide a justification as to why such a change is OK.

crazygringo 1y ago
That's confusing two different things, though. A monorepo does make changes atomic in the code. There's no trap there. You're talking about deployment, and yes when deployment is staggered, then obviously all atomic changes need to be backward-compatible, or else be very carefully orchestrated. But that doesn't have anything to do with monorepo vs polyrepo. That's just staggered deployment. You have to deal with backwards compatibility in both cases. But at least with the monorepo you can see and track and merge all the changes related to a feature in one place, and be able to roll them back in one place.
- yencabulator 1y ago
  There's no such thing as when deployment is staggered. It's a distributed problem, so by definition it is not synchronous. (Or you turn services off for the duration of the deploy. Most companies do not want that these days.) Also, you're missing this part of the article: > While this is also possible in a world with many repositories, the requirement to do this change in multiple pull requests is often enough to remind engineers that breaking changes to a service contract are not safe to make.

marcosdumay 1y ago

> The power of this is really hard to overstate A really useful heuristics when you are designing programming environments is: the more power you give to a team of developers, the more problems you will have. Technically, atomic commits are not more power, they are less. But it does empower the team to work with bad interfaces. And that's a power that creates problems.

jayd16 1y ago

Here's the issue... Being able to track commits across multiple projects is just a nice to have. Does it even significantly increase your ability to track dependencies or trigger down stream tests? You can track that stuff with multi-repo automations. Maybe it helps a bit but it's not free and its not complete. Your deploys don't become atomic. Your builds don't become atomic. At best you get to tell yourself you can version a bit more loosely. Mono-repos themselves do not scale easily. Inherently its a harder technical problem to solve. You need to toss git and find something better, which is not easy. Its work. Its so much work that it is incredibly clear that you've never experienced it yourself.

gugagore 1y ago
Do you have go-to multi repo automations and other tools for operating across multiple repos?

nobodywillobsrv 1y ago

While things like git submodules have problems, are they not sort of the best of both worlds camp? I never quite got the arguments/grumblings about this stuff but perhaps I do not understand what friction people are hitting up agains. It feels to me the problems of multiple repositories is the same problem of having to manage versions of dependencies like in pip or something that you do not even own. Perhaps people are using tools that mean they are not able to search/click through their multi repo setup even though the code connects?

Too 1y ago
Yes and No. One top level "superrepo" is the best of both worlds. You don't break the limits of git, you can include third party repos, you can seal off access to some secret repos. All while still giving benefit of everything checked out in one place. Tooling for flattening the git log exists and is relatively cheap to build yourself, It's easy to design CI that runs both big and small jobs in the superrepo and the subcomponents. Nested submodules = Game over. Don't ever do it. This is the worst of both worlds. Common components included twice or more need to be aligned to exact same revision, without help of semantic versioning. Components far down the dependency tree need to be uplifted recursively through a chain of submodules, resulting in 10 commits with no common CI for one tiny change.

wocram 1y ago

Just today I saw someone try to set up a new project and stumble over setting it up as it's own repository, instead of just having a central place to add it. There's also a vicious feedback loop of separating projects across repos creating a huge impediment for touching anything outside of the project you're directly working in.