5 ms·

While certainly interesting, the article leaves out the juicy bits monorepos and their workflows offer: - library developers can easily see the impact their ch

by jupp0r 8y ago

While certainly interesting, the article leaves out the juicy bits monorepos and their workflows offer: - library developers can easily see the impact their changes will have on their consumers - library changes that introduce regressions in their consumers can be caught pre-merge given good test coverage - dependency version updates between packages cause less mayhem because they are performed atomically and only merged when green At the same time, many drawbacks are also left out: - the incentive to have long living branches for stability reasons can negate most benefits mentioned above - build times for compiled languages can become problematic even for moderately sized organizations (I’m looking at you, C++) - in my experience, you pretty much need a dedicated dev team working on workflow tooling because ready solutions are fragmented and hard to integrate (code review, merge bots, CI/CD, ...)

adrianN 8y ago
The impact of changes across an ecosystem can also be checked in other ways. See for example how Rust checks the whole cargo ecosystem for problems caused by compiler changes.
- klodolph 8y ago
  Sure… monorepos provide solutions to certain problems, but aren’t the only way to solve those problems.
mlthoughts2018 8y ago
You list three items here: “- library developers can easily see the impact their changes will have on their consumers - library changes that introduce regressions in their consumers can be caught pre-merge given good test coverage - dependency version updates between packages cause less mayhem because they are performed atomically and only merged when green” But I think these are actually signs of big failure modes of monorepos, and each one has analog for solving it with polyrepos and/or versioned artifacts that’s actually much safer and more practical. Firstly, wishing to see how library changes will cause issues in downstream consumers is often a sign of very deep problems, because the library designer should be free to make changes or enhancements as they deem necessary to solve the problems they need to solve, which may require validating new changes don’t introduce unexpected problems for consumers, but, critically, also may not involve that, and in fact the usual case should be that it doesn’t involve that. Instead, it’s up to consumers to consume versioned artifacts of the library (more on this in a moment), so that consumers are totally in control of “opting in” to new changes. If the consumer doesn’t want to opt in, they should not be forced to (e.g. in monorepos where a successful merge of library X is a silent, de facto upgrade for all consumers required to consume library X from the same commit, etc.) Instead, the consumer should be the one creating an experimental branch with the intended version upgrade, and running it through the consumer’s validation tests to see if opting to upgrade the version will work. If it fails, they can open a bug report, and the library maintainer will decide if that regression is now necessary because of a constraint in the new version’s design, or if it can or should be patched. To be clear, this all should be happening inside a monorepo or outside of one, truly doesn’t matter. The point is to get rid of the horrible practice of forcing de facto software upgrades on one section of the code merely via a successful merge in a another section. That idea is terrible in concept, a very misguided thing that shouldn’t be desired. Instead, when you merge library changes in one part of the monorepo, CI or other tooling should produce a fixed artifact from it, like a Python wheel, jar file, Docker container, shared object binaries, whatever, and automatically upload to an artifact store like artifactory, with versioned identifiers. The code in some other section of the monorepo then can just happily keep plugging along, not blindsided by new changes, hit by errors uncaught in testing, etc., and the maintainer of that code can plan for their version upgrade according to their own timeline and testing. In particular, this is what actually reduces “mayhem” as you put it in dependency upgrades, because the standard of doing it implicitly when there is a green merge is just a form of putting your head in the sand and acting like the library author can trust green merge status as a reliable indicator that downstream consumers can auto-accept new changes, when really only those downstream consumer team devs actually know if that’s true or desirable.
- aidenn0 8y ago
  > Firstly, wishing to see how library changes will cause issues in downstream consumers is often a sign of very deep problems, because the library designer should be free to make changes or enhancements as they deem necessary to solve the problems they need to solve, which may require validating new changes don’t introduce unexpected problems for consumers, but, critically, also may not involve that, and in fact the usual case should be that it doesn’t involve that. ... > In particular, this is what actually reduces “mayhem” as you put it in dependency upgrades, because the standard of doing it implicitly when there is a green merge is just a form of putting your head in the sand and acting like the library author can trust green merge status as a reliable indicator that downstream consumers can auto-accept new changes, when really only those downstream consumer team devs actually know if that’s true or desirable. The only reason for a library's existence is to help solve problems for consumers. I would say that if the usual case does not involve worrying about issues for consumers then whether or not you choose polyrepo or monorepo is the least of your issues.
  - mlthoughts2018 8y ago
    > “The only reason for a library's existence is to help solve problems for consumers.“ I agree and this is the whole line of thinking behind my comment. In order to help consumers with what they need to solve, I might be required to break backward compatibility, delete old code to reduce complexity, refactor/redesign in ways that require consumers to change usage patterns, etc. If I cannot do these necessary things because every change I make is beholden to every regression test for every consumer for every legacy behavior, then it means I am failing to help my consumers, and the library is failing to give them what’s needed. Instead, if they can isolate their dependence on the library away from possibly breaking changes needed to reach new solutions in the newest version, then the library is still healthy and capable of solving consumer requirements.
    - Shish2k 8y ago
      > I might be required to break backward compatibility > ... > I cannot do these necessary things because every change I make is beholden to every regression test for every consumer Given that backward-incompatible changes are going to be a fact of life, then at some point, somebody will have to deal with them. We can either write one commit which deals with the library and the consumers at the same time; or we can modify the library in isolation and 3 months later the consumer tries to upgrade and is surprised to find out that the API has changed and the author of that change got hit by a bus. It's the same amount of code either way, but I find small incremental atomic changes are easier to manage than large batches of non-atomic changes.
rightbyte 8y ago
"- library developers can easily see the impact their changes will have on their consumers" So, to commit breaking changes you need to file 30 Jira tickets first and wait for a synced commit with 30 branches? Or how do you do it? Do monorepo shops work without releases and stable versions? If dependent projects (in use) pull your head and there is no synced development I assume you will end up with a complete legacy mess to work around other projects unit tests to get anything done.
- Shish2k 8y ago
  If I update a library function to require a new parameter, then in the same commit, I will also change every application which uses that library to supply the new parameter. The promise of the monorepo is "you can pick any random commit from any point in history, and all of those applications and libraries will be compatible with each other". For major changes that are too big to migrate every user in a single commit - create a new API, do development there, migrate users from the old API to the new API, delete the old API. (I wonder if you're coming from a world where library developers and application developers don't have permission to touch each other's code? At FB at least, the library developer would be allowed to create a pull request modifying the application code - though they may need a stamp of approval from the application owner before that pull request is merged into master)
  - adrianN 8y ago
    How does that work in code bases where you're not authorized to change everything?
    - speedkills 8y ago
      Painfully. File a ticket with another team and pray they don't put it off until next quarter because your teams priorities don't match up with theirs.

by jupp0r 8y ago

adrianN 8y ago

The impact of changes across an ecosystem can also be checked in other ways. See for example how Rust checks the whole cargo ecosystem for problems caused by compiler changes.

klodolph 8y ago
Sure… monorepos provide solutions to certain problems, but aren’t the only way to solve those problems.

mlthoughts2018 8y ago

You list three items here: “- library developers can easily see the impact their changes will have on their consumers - library changes that introduce regressions in their consumers can be caught pre-merge given good test coverage - dependency version updates between packages cause less mayhem because they are performed atomically and only merged when green” But I think these are actually signs of big failure modes of monorepos, and each one has analog for solving it with polyrepos and/or versioned artifacts that’s actually much safer and more practical. Firstly, wishing to see how library changes will cause issues in downstream consumers is often a sign of very deep problems, because the library designer should be free to make changes or enhancements as they deem necessary to solve the problems they need to solve, which may require validating new changes don’t introduce unexpected problems for consumers, but, critically, also may not involve that, and in fact the usual case should be that it doesn’t involve that. Instead, it’s up to consumers to consume versioned artifacts of the library (more on this in a moment), so that consumers are totally in control of “opting in” to new changes. If the consumer doesn’t want to opt in, they should not be forced to (e.g. in monorepos where a successful merge of library X is a silent, de facto upgrade for all consumers required to consume library X from the same commit, etc.) Instead, the consumer should be the one creating an experimental branch with the intended version upgrade, and running it through the consumer’s validation tests to see if opting to upgrade the version will work. If it fails, they can open a bug report, and the library maintainer will decide if that regression is now necessary because of a constraint in the new version’s design, or if it can or should be patched. To be clear, this all should be happening inside a monorepo or outside of one, truly doesn’t matter. The point is to get rid of the horrible practice of forcing de facto software upgrades on one section of the code merely via a successful merge in a another section. That idea is terrible in concept, a very misguided thing that shouldn’t be desired. Instead, when you merge library changes in one part of the monorepo, CI or other tooling should produce a fixed artifact from it, like a Python wheel, jar file, Docker container, shared object binaries, whatever, and automatically upload to an artifact store like artifactory, with versioned identifiers. The code in some other section of the monorepo then can just happily keep plugging along, not blindsided by new changes, hit by errors uncaught in testing, etc., and the maintainer of that code can plan for their version upgrade according to their own timeline and testing. In particular, this is what actually reduces “mayhem” as you put it in dependency upgrades, because the standard of doing it implicitly when there is a green merge is just a form of putting your head in the sand and acting like the library author can trust green merge status as a reliable indicator that downstream consumers can auto-accept new changes, when really only those downstream consumer team devs actually know if that’s true or desirable.

aidenn0 8y ago
> Firstly, wishing to see how library changes will cause issues in downstream consumers is often a sign of very deep problems, because the library designer should be free to make changes or enhancements as they deem necessary to solve the problems they need to solve, which may require validating new changes don’t introduce unexpected problems for consumers, but, critically, also may not involve that, and in fact the usual case should be that it doesn’t involve that. ... > In particular, this is what actually reduces “mayhem” as you put it in dependency upgrades, because the standard of doing it implicitly when there is a green merge is just a form of putting your head in the sand and acting like the library author can trust green merge status as a reliable indicator that downstream consumers can auto-accept new changes, when really only those downstream consumer team devs actually know if that’s true or desirable. The only reason for a library's existence is to help solve problems for consumers. I would say that if the usual case does not involve worrying about issues for consumers then whether or not you choose polyrepo or monorepo is the least of your issues.
- mlthoughts2018 8y ago
  > “The only reason for a library's existence is to help solve problems for consumers.“ I agree and this is the whole line of thinking behind my comment. In order to help consumers with what they need to solve, I might be required to break backward compatibility, delete old code to reduce complexity, refactor/redesign in ways that require consumers to change usage patterns, etc. If I cannot do these necessary things because every change I make is beholden to every regression test for every consumer for every legacy behavior, then it means I am failing to help my consumers, and the library is failing to give them what’s needed. Instead, if they can isolate their dependence on the library away from possibly breaking changes needed to reach new solutions in the newest version, then the library is still healthy and capable of solving consumer requirements.
  - Shish2k 8y ago
    > I might be required to break backward compatibility > ... > I cannot do these necessary things because every change I make is beholden to every regression test for every consumer Given that backward-incompatible changes are going to be a fact of life, then at some point, somebody will have to deal with them. We can either write one commit which deals with the library and the consumers at the same time; or we can modify the library in isolation and 3 months later the consumer tries to upgrade and is surprised to find out that the API has changed and the author of that change got hit by a bus. It's the same amount of code either way, but I find small incremental atomic changes are easier to manage than large batches of non-atomic changes.

rightbyte 8y ago

"- library developers can easily see the impact their changes will have on their consumers" So, to commit breaking changes you need to file 30 Jira tickets first and wait for a synced commit with 30 branches? Or how do you do it? Do monorepo shops work without releases and stable versions? If dependent projects (in use) pull your head and there is no synced development I assume you will end up with a complete legacy mess to work around other projects unit tests to get anything done.

Shish2k 8y ago
If I update a library function to require a new parameter, then in the same commit, I will also change every application which uses that library to supply the new parameter. The promise of the monorepo is "you can pick any random commit from any point in history, and all of those applications and libraries will be compatible with each other". For major changes that are too big to migrate every user in a single commit - create a new API, do development there, migrate users from the old API to the new API, delete the old API. (I wonder if you're coming from a world where library developers and application developers don't have permission to touch each other's code? At FB at least, the library developer would be allowed to create a pull request modifying the application code - though they may need a stamp of approval from the application owner before that pull request is merged into master)
- adrianN 8y ago
  How does that work in code bases where you're not authorized to change everything?
  - speedkills 8y ago
    Painfully. File a ticket with another team and pray they don't put it off until next quarter because your teams priorities don't match up with theirs.