3 ms·

> Only if you need to update all N other projects every time, which is unusual. This isn't true in my experience at large companies. For this to be true, you w

> Only if you need to update all N other projects every time, which is unusual. This isn't true in my experience at large companies. For this to be true, you would either need to have absolutely no shared core libraries, or core libraries that are so stable and unchanging that they never receive updates. Updates to those core libraries usually result in a cascade of changes across a ton of repos, tons of tests that need to be rerun, and a slew of now incompatible versions. All of these problems go away with a monorepo. Monorepos are absolutely the right solution for 98% of companies out there because they will never reach the engineering scale where monorepos start to struggle. For the 2% of companies that do reach that engineering scale, these kinds of blog posts get written. The mistake is reading blog posts like this and thinking is necessary or even relevant to any more than 2% of the companies out there. The vast majority of shops will never need anything like this.

morelisp 4y ago
You seem to have ruled out the obvious choice of "downstream projects just don't update." We update the projects we need the new library behavior in. There's no reason to touch the other 20 until there's some new behavior they do need. I don't really see this as a mono vs. multi repo issue either. In a mono repo you can still choose which projects update and the more you choose the bigger the diff and more involved the review process. In a multi repo case you can build analogous tooling to generate downstream MRs automatically. If you need to touch every downstream project regularly just to keep things functional, you don't have a mono-repo, you have a monolith. (Which can also be fine, really! But it's not the same thing at all.) (Yes, I know with 25 projects we're nowhere near Uber scale. But based on my experience at larger companies, this seems to scale more or less linearly - in terms of downstream project count, library count, and developer time available to dedicate to such things. If anything, developer time available is what seems to scale super-linearly.)
- GeneralMayhem 4y ago
  The "downstream projects just don't update" option is one I'd consider completely unacceptable. What if the update is a security patch? What happens when they inevitably do want a new feature, and have to fast-forward through months or years of interface updates all at once? What happens when a developer wants to work on a feature that cuts across systems, and now has to re-learn the "old" way of doing things?
  - morelisp 4y ago
    > What if the update is a security patch? Even a security fix is unlikely to affect every downstream user of a library except in egregious cases. But, if so, yes you update them all. (I'd say this is much less frequent than, I don't know, OpenSSL or log4j having a bug that makes us do this. The specific concern of an internal library having such a broad vulnerability is negligible in influencing our CI design.) > What happens when they inevitably do want a new feature, and have to fast-forward through months or years of interface updates all at once? You do it. > What happens when a developer wants to work on a feature that cuts across systems, and now has to re-learn the "old" way of doing things? You do it. I didn't say "it has no associated costs." But you need to weigh those costs against the other costs of a monorepo, and other costs in your CI/CD design generally. "Take longer to update a really old project once a year" is a much lower cost for us than "have a dedicated CI team to wrangle the tooling we need for automatic downstream pushes / a monorepo."
viraptor 4y ago
Did you miss the "every time" qualifier? Everyone will have some core libraries, but most work will be very limited in scope. Outside of large companies you can see this effect in nixpkgs for example. See how few entries have a rebuild count larger than 10 https://github.com/NixOS/nixpkgs/pulls?q=is%3Apr+is%3Aclosed https://github.com/NixOS/nixpkgs/pulls?q=is%3Apr+is%3Aclosed

morelisp 4y ago

You seem to have ruled out the obvious choice of "downstream projects just don't update." We update the projects we need the new library behavior in. There's no reason to touch the other 20 until there's some new behavior they do need. I don't really see this as a mono vs. multi repo issue either. In a mono repo you can still choose which projects update and the more you choose the bigger the diff and more involved the review process. In a multi repo case you can build analogous tooling to generate downstream MRs automatically. If you need to touch every downstream project regularly just to keep things functional, you don't have a mono-repo, you have a monolith. (Which can also be fine, really! But it's not the same thing at all.) (Yes, I know with 25 projects we're nowhere near Uber scale. But based on my experience at larger companies, this seems to scale more or less linearly - in terms of downstream project count, library count, and developer time available to dedicate to such things. If anything, developer time available is what seems to scale super-linearly.)

GeneralMayhem 4y ago
The "downstream projects just don't update" option is one I'd consider completely unacceptable. What if the update is a security patch? What happens when they inevitably do want a new feature, and have to fast-forward through months or years of interface updates all at once? What happens when a developer wants to work on a feature that cuts across systems, and now has to re-learn the "old" way of doing things?
- morelisp 4y ago
  > What if the update is a security patch? Even a security fix is unlikely to affect every downstream user of a library except in egregious cases. But, if so, yes you update them all. (I'd say this is much less frequent than, I don't know, OpenSSL or log4j having a bug that makes us do this. The specific concern of an internal library having such a broad vulnerability is negligible in influencing our CI design.) > What happens when they inevitably do want a new feature, and have to fast-forward through months or years of interface updates all at once? You do it. > What happens when a developer wants to work on a feature that cuts across systems, and now has to re-learn the "old" way of doing things? You do it. I didn't say "it has no associated costs." But you need to weigh those costs against the other costs of a monorepo, and other costs in your CI/CD design generally. "Take longer to update a really old project once a year" is a much lower cost for us than "have a dedicated CI team to wrangle the tooling we need for automatic downstream pushes / a monorepo."

viraptor 4y ago

Did you miss the "every time" qualifier? Everyone will have some core libraries, but most work will be very limited in scope. Outside of large companies you can see this effect in nixpkgs for example. See how few entries have a rebuild count larger than 10 https://github.com/NixOS/nixpkgs/pulls?q=is%3Apr+is%3Aclosed https://github.com/NixOS/nixpkgs/pulls?q=is%3Apr+is%3Aclosed