6 ms·

Monorepos: Please Don't (2019)

20 points by jrpelkonen 2y ago

gladwindos 2y ago
I’ve heard about how huge companies like google are using a giant monorepo, and I honestly don’t understand how that could work. But I guess it does…
- j45 2y ago
  It's sometimes important to remind ourselves that just because we can't see understanding in something doesn't mean there isn't understanding in it, and it might take effort on our part. It's not a negative thing as much as an opportunity to grow if we ask ourselves to learn and see it as an opportunity.
- inhumantsar 2y ago
  with lots of controls and custom tooling
  - verdverm 2y ago
    Facebook open sourced theirs https://github.com/facebook/sapling https://github.com/facebook/sapling
    - gladwindos 2y ago
      This looks interesting! I'll check it out and maybe it will help me wrap my head around how they handle the complexity.
      - verdverm 2y ago
        If you aren't worried about partial checkouts & permissions The biggest issue is deciding what to build across developers and their branches & products. You don't want to build the world on every commit, but you do want to ensure that you are building what the code would look like post-merge before the merge.
- tail_exchange 2y ago
  The company I work at is fairly large, with thousands of engineers, and we use monorepos. It works extremely well. What about them makes you think it can't work?
  - gladwindos 2y ago
    We use monorepos in my company too, but we use one for each product, and it works really well for that. But for the whole company to be in a single monorepo just seems like it would require a lot of effort to maintain.
    - tail_exchange 2y ago
      We have several projects per monorepo, and they are massive, currently being merged into a single one. Anything specific about maintainability that scares you?
  - CRConrad 2y ago
    > ...we use monorepos. MonorepoS, in the plural?
    - tail_exchange 2y ago
      Yes. We used to have dozens of repos, but started to merge them slowly. Nowadays we have 3 large ones, but the goal is 1.
      - CRConrad 2y ago
        So you have Trirepo, not monorepo.
        tail_exchange 2y ago
        Looking forward to the birepo.
shawabawa3 2y ago
Having seen a large org attempt a poly -> mono repo conversion project (which took almost 2 years and failed and was in the process of being rolled back when I left) I agree with everything in this article Once an org becomes large enough you need lots of custom tooling to make working across a huge codebase smoother, and that tooling is similar for mono and poly repos, but you need an additional metric ton of tooling to make monorepos work And please pray you didn't decide to go with bazel for builds My least favourite monorepo experience: I want to update my dependency for my tiny service for a security patch -> Oh, 400 other services depend on it and dozens of tests break when I try to update it -> this isn't worth my time I'll do something else instead We were promised a world where every dependency was kept up to date by necessity but ended up in a world where all dependencies atrophied due to the increased difficulty to update them
- jpc0 2y ago
  > I want to update my dependency for my tiny service for a security patch If the API changed it's not a security patch... Why would a change that only fixes a security bug cause tests to fail? Sounds like this is more a matter of needing to cherry-pick temporarily and then actually pushing for the codebase to update, probably by the security team. Sometimes you need to work with other people, that might necessitate doing "ugly" things to get the job done.
  - thornygreb 2y ago
    Yes, but very often you are trucking along just fine with some version of a dependency and then all of a sudden it gets a CVE and the fix has only been applied to the next major version and not backported because the version you are on is no longer supported. And now you are in dependency update cascading hell.
- tdb7893 2y ago
  Having worked at a place with a monorepo it was one of the things I actually really liked about working there. It took a lot more tooling but I found it to be much better than a multi-repo would have been, even with a similar amount of tooling. I didn't have to be checking different projects out all the time and changes across projects weren't too bad (large scales ones still took a lot of managing but not as much as changes across a multi-repo). I was always hoping that one of the big cloud providers would offer a monorepo and invest a lot in making tooling for it to actually be usable.
Hackbraten 2y ago
TFA assumes several things which may or may not hold for a particular company or team. For example, the Linux kernel works with its tens of millions of lines just fine, even though ownership is heavily distributed among many subsystem maintainers. Checking out a branch or pulling in changes rarely takes more than a few seconds. On the other hand, some organizations abuse their VCS, using it as a repository for large immutable files such as build artifacts, software packages or similar non-source archives. Orgs who let this happen are of course going to have a hard time. But that's hardly the monorepo's fault.
veidr 2y ago
This article sucks. Does your company have tens of thousands of employees? If so, hopefully you have some engineers working on your tooling. If not, you don't have the problems described here, and you will also happily get all the "theoretical benefits" that this guy apparently didn't experience. Moving to monorepo is often a win; also the name sucks, we have a a few "monorepos" at work, and I think it's the sweet spot. Rust and C firmware doesn't need to live with TypeScript frontend apps, but it's pretty wasteful to have apps of the same ecosystem unable to share any and all dependencies and utility code trivially. (Where "trivially" means literally one line of code import { bleargh } from 'hoge'; and not one single step more.)
- worksonmine 2y ago
  You can npm install repositories, no need to merge frontend and backend just to share code. Break out what can be shared into utilities and you can share it in completely unrelated projects as well. I get monorepos for SSR and the likes but why build and deploy an entire container to push a line of CSS?
  - verdverm 2y ago
    Polyrepo ends up with a mess of version management and flood of PRs to roll out minor changes Imagine adding a field to a table that must be reflected in the front end and some other micro service consumers. Coordination can be painful. Each system has its pros and cons, you're going to need tooling either way. I've moved to mono or mega repo as the default and break out smaller pieces into dedicated repo as needed
    - gorjusborg 2y ago
      I get the feeling what you call a monorepo is microscopic compared to what the article is discussing. Your monorepo sometimes approach is the polyrepo approach.
    - worksonmine 2y ago
      You're going to have to edit the code in 2 places either way. With a sane deployment strategy the backend would have the extra field, then you can deploy the frontend to render said change. Or sometimes the table is dynamic and doesn't even need changes on the frontend. To me the ability to push changes independent of other pieces far outweighs the extra git commit and push. I've never had the need for any other tooling than a container to run in, git to manage code, and CI/CD for deploys. If there's a need for special tooling to manage the projects there's to much complexity for my taste and I remove the complexity instead.
liampulles 2y ago
I think of org-wide monorepos the same way I think of zealous microservice use, or microfrontends, or platform engineering, etc. It is a "shiny thing" that FAANGs use, which get a disproportionate amount of tech evangelist coverage, and then transition into becoming the perceived "best-practice" by bored devs (myself included) to then create some tech-debt work. The reality is that these are somewhat niche usecases for very high scale orgs and/or very high-scale developer workforces which can afford to reinvent the wheel. The way the vast majority of companies should go about developing their systems is not new at all, its just to pragmatically apply tried and tested technologies, and transition to new shiny things where there is a demonstrable need. Worth saying that I'm definitely a fan of team-scoped monorepos though. Being able to automate building and testing across all my apps, update dependencies across all apps, and deploy with a single merge/pull request is great.
quectophoton 2y ago
I get what the article is saying. But if `ls -d ../*/.git/` prints the names of all the company's repos, and the same happens in a few coworker's computers, I start to question things.
erik_seaberg 2y ago
Keeping prod stable is more important than alpha testing every team's latest commit, but monorepos tend not to accommodate staying on known-good versions of your dependencies and opting into updates when ready.
whateveracct 2y ago
Monorepos are most painful when all the components are heavily coupled. Maybe a bunch of unrelated stuff in a single compilation unit (e.g. package) results in a lot of waste. And then people just end up not having clear interfaces and boundaries and ownership. But I guess that's not a result of a monorepo inherently. It's just the result of poor engineering leadership.
roughly 2y ago
At a previous gig, they went down the monorepo rabbit hole. For all the reasons the author notes, in not terribly long, we were hearing about the monobuild project, and then the monodeploy project, which was strange, because I think the same people behind the monorepo project had been the ones who advocated for breaking up what had been a monolith service. Plus ça change, I suppose.
mediumsmart 2y ago
Medium: Please Don’t (2024)
Eridrus 2y ago
I worked at 4 other companies before Google, and Google's source code tooling was the best I'd ever seen and definitely something the rest of the world can work towards. In particular, it fostered a culture where accepting code changes from outside teams was normal (though obviously not always uncontentious), and prototyping those changes was pretty easy. I've been using pantsbuild.org since I left and I think it hits a good sweet spot between blaze, which has a lot of overhead, and the rest of the world.
shaftway 2y ago
I'm in the opposite situation, poly-repo hell. Our team produces 4 libraries. The code for them along with test apps is spread across 7 repos. So when I want to make a core change (like updating the version of Gradle we use) it results in 7 different PRs that have to go in at the same time, and that everyone has to fetch in sync. It's pretty rare that a change only requires a single PR because of this. Releases are a mess. To do a release we tag the latest commit in each of the repos with the same release number and use Jenkins to pull that tag from each repo and put it all together. But that's just like a suggestion, and it's not uncommon for a release to be based off of master. Good luck doing the forensics to figure out the state of everything for a release. The best part is that each repo needs to be in a specific folder on disk because some of them look for the others via hard-coded paths. There is one repo that needs to be separate, because it's public. But I've been pushing hard to get most of the repos pushed together into a single one just to add some sanity to my PRs.

20 points by jrpelkonen 2y ago

I’ve heard about how huge companies like google are using a giant monorepo, and I honestly don’t understand how that could work. But I guess it does…

j45 2y ago
It's sometimes important to remind ourselves that just because we can't see understanding in something doesn't mean there isn't understanding in it, and it might take effort on our part. It's not a negative thing as much as an opportunity to grow if we ask ourselves to learn and see it as an opportunity.
inhumantsar 2y ago
with lots of controls and custom tooling
- verdverm 2y ago
  Facebook open sourced theirs https://github.com/facebook/sapling https://github.com/facebook/sapling
  - gladwindos 2y ago
    This looks interesting! I'll check it out and maybe it will help me wrap my head around how they handle the complexity.
    - verdverm 2y ago
      If you aren't worried about partial checkouts & permissions The biggest issue is deciding what to build across developers and their branches & products. You don't want to build the world on every commit, but you do want to ensure that you are building what the code would look like post-merge before the merge.
tail_exchange 2y ago
The company I work at is fairly large, with thousands of engineers, and we use monorepos. It works extremely well. What about them makes you think it can't work?
- gladwindos 2y ago
  We use monorepos in my company too, but we use one for each product, and it works really well for that. But for the whole company to be in a single monorepo just seems like it would require a lot of effort to maintain.
  - tail_exchange 2y ago
    We have several projects per monorepo, and they are massive, currently being merged into a single one. Anything specific about maintainability that scares you?
- CRConrad 2y ago
  > ...we use monorepos. MonorepoS, in the plural?
  - tail_exchange 2y ago
    Yes. We used to have dozens of repos, but started to merge them slowly. Nowadays we have 3 large ones, but the goal is 1.
    - CRConrad 2y ago
      So you have Trirepo, not monorepo.
      - tail_exchange 2y ago
        Looking forward to the birepo.

shawabawa3 2y ago

Having seen a large org attempt a poly -> mono repo conversion project (which took almost 2 years and failed and was in the process of being rolled back when I left) I agree with everything in this article Once an org becomes large enough you need lots of custom tooling to make working across a huge codebase smoother, and that tooling is similar for mono and poly repos, but you need an additional metric ton of tooling to make monorepos work And please pray you didn't decide to go with bazel for builds My least favourite monorepo experience: I want to update my dependency for my tiny service for a security patch -> Oh, 400 other services depend on it and dozens of tests break when I try to update it -> this isn't worth my time I'll do something else instead We were promised a world where every dependency was kept up to date by necessity but ended up in a world where all dependencies atrophied due to the increased difficulty to update them

jpc0 2y ago
> I want to update my dependency for my tiny service for a security patch If the API changed it's not a security patch... Why would a change that only fixes a security bug cause tests to fail? Sounds like this is more a matter of needing to cherry-pick temporarily and then actually pushing for the codebase to update, probably by the security team. Sometimes you need to work with other people, that might necessitate doing "ugly" things to get the job done.
- thornygreb 2y ago
  Yes, but very often you are trucking along just fine with some version of a dependency and then all of a sudden it gets a CVE and the fix has only been applied to the next major version and not backported because the version you are on is no longer supported. And now you are in dependency update cascading hell.
tdb7893 2y ago
Having worked at a place with a monorepo it was one of the things I actually really liked about working there. It took a lot more tooling but I found it to be much better than a multi-repo would have been, even with a similar amount of tooling. I didn't have to be checking different projects out all the time and changes across projects weren't too bad (large scales ones still took a lot of managing but not as much as changes across a multi-repo). I was always hoping that one of the big cloud providers would offer a monorepo and invest a lot in making tooling for it to actually be usable.

Hackbraten 2y ago

TFA assumes several things which may or may not hold for a particular company or team. For example, the Linux kernel works with its tens of millions of lines just fine, even though ownership is heavily distributed among many subsystem maintainers. Checking out a branch or pulling in changes rarely takes more than a few seconds. On the other hand, some organizations abuse their VCS, using it as a repository for large immutable files such as build artifacts, software packages or similar non-source archives. Orgs who let this happen are of course going to have a hard time. But that's hardly the monorepo's fault.

veidr 2y ago

This article sucks. Does your company have tens of thousands of employees? If so, hopefully you have some engineers working on your tooling. If not, you don't have the problems described here, and you will also happily get all the "theoretical benefits" that this guy apparently didn't experience. Moving to monorepo is often a win; also the name sucks, we have a a few "monorepos" at work, and I think it's the sweet spot. Rust and C firmware doesn't need to live with TypeScript frontend apps, but it's pretty wasteful to have apps of the same ecosystem unable to share any and all dependencies and utility code trivially. (Where "trivially" means literally one line of code import { bleargh } from 'hoge'; and not one single step more.)

worksonmine 2y ago
You can npm install repositories, no need to merge frontend and backend just to share code. Break out what can be shared into utilities and you can share it in completely unrelated projects as well. I get monorepos for SSR and the likes but why build and deploy an entire container to push a line of CSS?
- verdverm 2y ago
  Polyrepo ends up with a mess of version management and flood of PRs to roll out minor changes Imagine adding a field to a table that must be reflected in the front end and some other micro service consumers. Coordination can be painful. Each system has its pros and cons, you're going to need tooling either way. I've moved to mono or mega repo as the default and break out smaller pieces into dedicated repo as needed
  - gorjusborg 2y ago
    I get the feeling what you call a monorepo is microscopic compared to what the article is discussing. Your monorepo sometimes approach is the polyrepo approach.
  - worksonmine 2y ago
    You're going to have to edit the code in 2 places either way. With a sane deployment strategy the backend would have the extra field, then you can deploy the frontend to render said change. Or sometimes the table is dynamic and doesn't even need changes on the frontend. To me the ability to push changes independent of other pieces far outweighs the extra git commit and push. I've never had the need for any other tooling than a container to run in, git to manage code, and CI/CD for deploys. If there's a need for special tooling to manage the projects there's to much complexity for my taste and I remove the complexity instead.

liampulles 2y ago

I think of org-wide monorepos the same way I think of zealous microservice use, or microfrontends, or platform engineering, etc. It is a "shiny thing" that FAANGs use, which get a disproportionate amount of tech evangelist coverage, and then transition into becoming the perceived "best-practice" by bored devs (myself included) to then create some tech-debt work. The reality is that these are somewhat niche usecases for very high scale orgs and/or very high-scale developer workforces which can afford to reinvent the wheel. The way the vast majority of companies should go about developing their systems is not new at all, its just to pragmatically apply tried and tested technologies, and transition to new shiny things where there is a demonstrable need. Worth saying that I'm definitely a fan of team-scoped monorepos though. Being able to automate building and testing across all my apps, update dependencies across all apps, and deploy with a single merge/pull request is great.

quectophoton 2y ago

I get what the article is saying. But if `ls -d ../*/.git/` prints the names of all the company's repos, and the same happens in a few coworker's computers, I start to question things.

erik_seaberg 2y ago

Keeping prod stable is more important than alpha testing every team's latest commit, but monorepos tend not to accommodate staying on known-good versions of your dependencies and opting into updates when ready.

whateveracct 2y ago

Monorepos are most painful when all the components are heavily coupled. Maybe a bunch of unrelated stuff in a single compilation unit (e.g. package) results in a lot of waste. And then people just end up not having clear interfaces and boundaries and ownership. But I guess that's not a result of a monorepo inherently. It's just the result of poor engineering leadership.

roughly 2y ago

At a previous gig, they went down the monorepo rabbit hole. For all the reasons the author notes, in not terribly long, we were hearing about the monobuild project, and then the monodeploy project, which was strange, because I think the same people behind the monorepo project had been the ones who advocated for breaking up what had been a monolith service. Plus ça change, I suppose.

mediumsmart 2y ago

Medium: Please Don’t (2024)

Eridrus 2y ago

I worked at 4 other companies before Google, and Google's source code tooling was the best I'd ever seen and definitely something the rest of the world can work towards. In particular, it fostered a culture where accepting code changes from outside teams was normal (though obviously not always uncontentious), and prototyping those changes was pretty easy. I've been using pantsbuild.org since I left and I think it hits a good sweet spot between blaze, which has a lot of overhead, and the rest of the world.

shaftway 2y ago

I'm in the opposite situation, poly-repo hell. Our team produces 4 libraries. The code for them along with test apps is spread across 7 repos. So when I want to make a core change (like updating the version of Gradle we use) it results in 7 different PRs that have to go in at the same time, and that everyone has to fetch in sync. It's pretty rare that a change only requires a single PR because of this. Releases are a mess. To do a release we tag the latest commit in each of the repos with the same release number and use Jenkins to pull that tag from each repo and put it all together. But that's just like a suggestion, and it's not uncommon for a release to be based off of master. Good luck doing the forensics to figure out the state of everything for a release. The best part is that each repo needs to be in a specific folder on disk because some of them look for the others via hard-coded paths. There is one repo that needs to be separate, because it's public. But I've been pushing hard to get most of the repos pushed together into a single one just to add some sanity to my PRs.