4 ms·

A well factored system should not have cross-project changes. Individual projects should be able to upgrade dependencies separately.

by spion 4y ago

idontknowifican 4y ago
can you provide more info here? i’m finding it hard to think of a system at a large company that would have no internal libraries used by multiple projects, that wouldn’t require ever making a cross project change.
- spion 4y ago
  Changes will be handled just like external library updates.
  - idontknowifican 4y ago
    i suspect your internal model of a monorepo is fairly disjoint with an actual implementation. you’re arguing for non-monorepo based version concepts, without giving space to why having everyone locked to “latest” has large cost savings long term for maintenance and security. without those constraints, a monorepo is a bad choice.
    - spion 4y ago
      I've worked with both models, and the monorepo model has been far more demanding in terms of time and resources to maintain as you push past the limits of the available tools one by one. Standard build tools go out of the window almost immediately (lots of work to wrap everything with Bazel), then standard collaboration tools (GH) and so on. A dedicated tooling team becomes a must. What helps maintain polyrepos sane 1. A healthy methodology for dependency management Evolve APIs with deprecations. Decide on a healthy amount of time for a deprecations to live. Set up alerts when deprecations reach certain amount of time. Set up dashboards (e.g. Grafana) to track dependencies. Help other teams update, and think about how to make updates less painful. 2. Define good boundaries and API contracts to adhere to. The most important thing for a good API contract is stability. 3. Don't prematurely split into microservices. Better ideas for stable API contracts emerge the longer you can wait.
      - physicsguy 4y ago
        The issue comes when business requirements don’t allow you to arbitrarily decide that your API should be deprecated. If it’s depended on by 10 teams who are doing higher impact customer facing work than you are, at some point you’re likely to hit a “no” when it comes to disrupting them.
        spion 4y ago
        How is that different in a monorepo? Would you be tasked in implementing the changes for those teams yourself? If so, why not just do that as PRs in their repos?
chrisseaton 4y ago
You can use this argument against anything you don't like. 'A well-designed system should not have X therefore you don't need Y.' When 'A well-designed system should not have X' is just a matter of opinion, or doesn't acknowledge trade-offs that might make X the better option than Z, then this isn't a useful argument against Y.
sokoloff 4y ago
One of the examples given was a go compiler upgrade for all of their go code. It seems reasonable that some companies would want to have a single team investigate and perform this upgrade all at once. Sure, each project team could upgrade independently, but the company that’s chosen a mono-repo for their go code is likely to desire to have a single team tackle this upgrade.
jlouis 4y ago
In theory. In practice, you often end up with larger changes which are mostly janitorial. Bumping a version of a log library across the infrastructure, or a version of a serialization system. This simplifies dependency convergence and a best handled as if they are cross-project changes. The other problem is that over time, good factoring tend to deteriorate. A large software project will invariably have people with different brains working on it, and they'll have different needs for the factoring. So the project naturally pushes itself toward situations where cross-project changes become a necessary thing.
- spion 4y ago
  We already have tools that handle this for us for OSS libraries, such as dependabot etc. Good factoring can probably be quantified too: dependency chains should be shallow and graph connectivity should be low. The more dependants a module has, the more stable its API contract should be.
likortera 4y ago
Well, almost everyone talking here about nonorepos why they are actually building are distributed monoliths. That's why they need changes cross all clients on the same PR.
shepwalker 4y ago
yeaaah, I used to hold this view and drove a team - hard - to essentially pull apart a monorepo into different components with clear contract boundaries. It was by far one of my greatest errors in professional judgement. at the risk of coming up with a contrived example: let's say you own a service that need to deserialize a datetime in a request in a format you don't currently support. assuming you own the stack, you need to a) update your date library b) update your webserver stack c) possibly update an intermediate webserver stack that includes primitives like logging, telemetry, tracing, auth, service discovery and d) your actual service. If a->d are all independent, separate components, you have to orchestrate those changes through 4 separate repositories. And god forbid something you did at the lowest point in the stack is completely unworkable higher up. There's all sorts of rocket science you could do to orchestrate these changes, but it ends up being contrived and edgecasey. Most of the pain from monorepos can also be addressed with a dash of rocketscience (see:bazel), but the end model tends to have a) have an easier mental model for the user b) allow for consolidation of infrastructure work. Ie, your build/ci/language tooling teams can focus their efforts on one place c) can coordinate changes across the entire stack within one field of view d) can coral some of the worst, disparate instincts of a growing engineering org (ie, tons of teams optimizing for local maximas without internalizing knock-on effects). e) fewer weird edgecases.
- spion 4y ago
  Its important to avoid deep dependency chains when doing this, yes. I usually recommend a "framework" multi-package repo and then multiple "application" / "service" repos. I like having examples, even contrived ones, but I'm not sure I understood this one. Can you elaborate on what you mean? Is it about adding support for a new serialization format for dates in requests to a service? Why would this affect the webserver stack and logging/telemetry/tracing/auth primitives? I find that a lot of organizations have really strange thoughts on how to factor things into separate microservices and libraries. Usually I approach by asking the following question: if this was an open source-library or service (e.g. like elasticsearch), would you use it? If not, then its probably not a great candidate for a separate thing - lets try and come up with something else. One way to handle CI/CD is using standardized pipelines e.g. you tag your repo with a tag `app:node` or `lib:js` and the github org pipeline scanner will find it and assign the standard `app:node` or `lib:js` pipeline to it. A way that I like better but most tools unfortunatley don't support it yet is for the infra teams to publish libraries that are essentially functions taking some parameters and generating (standard) pipelines/configuration. Those can then be tracked together the same as other dependencies.
okino 4y ago
Good articulation of why things don’t quite play out that way within a company here https://rushjs.io/pages/intro/why_mono/ https://rushjs.io/pages/intro/why_mono/
- spion 4y ago
  For that particular article, I would respond with the following: 1. I don't see how this is a problem specific to polyrepos. Have an "open PRs" link in the onboarding handbook that gives you a view of pull requests from all repos in the organization. GitHub automatically shows you notifications from all repos. If engineers still chose to focus on one or two repos after that, I'm not sure why. - Have a (Grafana) dashboard where you can see the latest / newest stuff. Use standard GH tools you use for OSS, such as follows etc to keep up. 2. Don't prematurely split into multiple repos. "No monorepo" doesn't mean not having poly-package repos. It means thinking what the sensible (library or service) API boundary is - treating your projects as you would treat library / service development. In this case a separate repo with lib3, lib2 and lib1 sounds like a good way to go - at most one repo per orthogonal internal framework (e.g. core-react-components). Repo dependency chains should be as shallow as possible, and differenting between public and internal packages is important. 3. Help other teams upgrade. If you are responsible for repo A, once you publish a new version tagged appropriately with semver, use the dashboard to look at your dependants and work with them (or rather, for them) to upgrade. Think of your dependants as internal customers, and make sure you add enough value for them to justify the upgrade effort. Cultivate a culture that values updates. 4. There are other alternatives to `npm link` e.g. see `yalc` https://github.com/wclr/yalc https://github.com/wclr/yalc Another pet peeve of mine is that the real issues get lost when you try to generalize. The article attempts to do this but that makes it hard to evaluate its claims. The best way to evaluate (alternative) solutions is to take a more concrete example repo. For scalability of this model I'll just point to the OSS community; individual maintainers often several dozen active repositories, but also they have an API contract worthy of a documentation website, versioning scheme and planned deprecation, and they typically avoid cross-project dependencies
tveita 4y ago
Splitting your code over many repositories massively increases the cost of refactoring, making it that less likely that your system will be "well factored". Besides the "N pull requests" problem you now lose history whenever you move a file across repo boundaries, and you'll eventually have straggler projects staying on old versions for years - so switching to a new way of doing something essentially means supporting both versions forever. Code for common needs gets duplicated, or worse, split out into yet another repository and /then/ duplicated, because no one wants to check if any of 100 repositories rely on the buggy behaviour they want to fix. I find this to be this a level headed explanation of the advantages of monorepos: https://danluu.com/monorepo/ https://danluu.com/monorepo/. I'm surprised to see so many comments summarily dismissing them, as if sanely managing thousands of smaller interdependent repositories doesn't require at least as much investment in custom tooling.
- spion 4y ago
  > Splitting your code over many repositories massively increases the cost of refactoring, making it that less likely that your system will be "well factored" The project should stay a full monolith until the factoring is more clear. > I find this to be this a level headed explanation of the advantages of monorepos: https://danluu.com/monorepo/ https://danluu.com/monorepo/. There are two issues I have with this article. One of them is when it describes drawbacks of multiple repositories, its not specific enough e.g. > That sounds like it ought to be straightforward, but in practice, most solutions are cumbersome and involve a lot of overhead. The other is that it assumes you have Google scale of resources to throw at the problem. If thats the case you can make anything work / monorepos or polyrepos. The issue is that small-to-medium sized organizations will not be prepared to invest the amount of resources needed to keep a monorepo working well, as the org will largly need to rely on existing available (OSS) tools which often have poor monorepo support. (Bazelifying everything has a significant cost, bazel rules are often not generic enough to work with e.g. the variety of JS ecosystem tools)
- preseinger 4y ago
  Only if you break your code up on boundaries that don't reflect reality. If one business change requires code changes to ~every module in your source tree, then of course microservices, and therefore separate repos per service or whatever, make no sense at all.
jogjayr 4y ago
> Individual projects should be able to upgrade dependencies separately. No, that way lies sorrow and despair. There should only be one version of any dependency being used in your company. Any deviations should require like VP-level approvals or something.

A well factored system should not have cross-project changes. Individual projects should be able to upgrade dependencies separately.

by spion 4y ago

A well factored system should not have cross-project changes. Individual projects should be able to upgrade dependencies separately.

idontknowifican 4y ago

can you provide more info here? i’m finding it hard to think of a system at a large company that would have no internal libraries used by multiple projects, that wouldn’t require ever making a cross project change.

spion 4y ago
Changes will be handled just like external library updates.
- idontknowifican 4y ago
  i suspect your internal model of a monorepo is fairly disjoint with an actual implementation. you’re arguing for non-monorepo based version concepts, without giving space to why having everyone locked to “latest” has large cost savings long term for maintenance and security. without those constraints, a monorepo is a bad choice.
  - spion 4y ago
    I've worked with both models, and the monorepo model has been far more demanding in terms of time and resources to maintain as you push past the limits of the available tools one by one. Standard build tools go out of the window almost immediately (lots of work to wrap everything with Bazel), then standard collaboration tools (GH) and so on. A dedicated tooling team becomes a must. What helps maintain polyrepos sane 1. A healthy methodology for dependency management Evolve APIs with deprecations. Decide on a healthy amount of time for a deprecations to live. Set up alerts when deprecations reach certain amount of time. Set up dashboards (e.g. Grafana) to track dependencies. Help other teams update, and think about how to make updates less painful. 2. Define good boundaries and API contracts to adhere to. The most important thing for a good API contract is stability. 3. Don't prematurely split into microservices. Better ideas for stable API contracts emerge the longer you can wait.
    - physicsguy 4y ago
      The issue comes when business requirements don’t allow you to arbitrarily decide that your API should be deprecated. If it’s depended on by 10 teams who are doing higher impact customer facing work than you are, at some point you’re likely to hit a “no” when it comes to disrupting them.
      - spion 4y ago
        How is that different in a monorepo? Would you be tasked in implementing the changes for those teams yourself? If so, why not just do that as PRs in their repos?

chrisseaton 4y ago

You can use this argument against anything you don't like. 'A well-designed system should not have X therefore you don't need Y.' When 'A well-designed system should not have X' is just a matter of opinion, or doesn't acknowledge trade-offs that might make X the better option than Z, then this isn't a useful argument against Y.

sokoloff 4y ago

One of the examples given was a go compiler upgrade for all of their go code. It seems reasonable that some companies would want to have a single team investigate and perform this upgrade all at once. Sure, each project team could upgrade independently, but the company that’s chosen a mono-repo for their go code is likely to desire to have a single team tackle this upgrade.

jlouis 4y ago

In theory. In practice, you often end up with larger changes which are mostly janitorial. Bumping a version of a log library across the infrastructure, or a version of a serialization system. This simplifies dependency convergence and a best handled as if they are cross-project changes. The other problem is that over time, good factoring tend to deteriorate. A large software project will invariably have people with different brains working on it, and they'll have different needs for the factoring. So the project naturally pushes itself toward situations where cross-project changes become a necessary thing.

spion 4y ago
We already have tools that handle this for us for OSS libraries, such as dependabot etc. Good factoring can probably be quantified too: dependency chains should be shallow and graph connectivity should be low. The more dependants a module has, the more stable its API contract should be.

likortera 4y ago

Well, almost everyone talking here about nonorepos why they are actually building are distributed monoliths. That's why they need changes cross all clients on the same PR.

shepwalker 4y ago

yeaaah, I used to hold this view and drove a team - hard - to essentially pull apart a monorepo into different components with clear contract boundaries. It was by far one of my greatest errors in professional judgement. at the risk of coming up with a contrived example: let's say you own a service that need to deserialize a datetime in a request in a format you don't currently support. assuming you own the stack, you need to a) update your date library b) update your webserver stack c) possibly update an intermediate webserver stack that includes primitives like logging, telemetry, tracing, auth, service discovery and d) your actual service. If a->d are all independent, separate components, you have to orchestrate those changes through 4 separate repositories. And god forbid something you did at the lowest point in the stack is completely unworkable higher up. There's all sorts of rocket science you could do to orchestrate these changes, but it ends up being contrived and edgecasey. Most of the pain from monorepos can also be addressed with a dash of rocketscience (see:bazel), but the end model tends to have a) have an easier mental model for the user b) allow for consolidation of infrastructure work. Ie, your build/ci/language tooling teams can focus their efforts on one place c) can coordinate changes across the entire stack within one field of view d) can coral some of the worst, disparate instincts of a growing engineering org (ie, tons of teams optimizing for local maximas without internalizing knock-on effects). e) fewer weird edgecases.

spion 4y ago
Its important to avoid deep dependency chains when doing this, yes. I usually recommend a "framework" multi-package repo and then multiple "application" / "service" repos. I like having examples, even contrived ones, but I'm not sure I understood this one. Can you elaborate on what you mean? Is it about adding support for a new serialization format for dates in requests to a service? Why would this affect the webserver stack and logging/telemetry/tracing/auth primitives? I find that a lot of organizations have really strange thoughts on how to factor things into separate microservices and libraries. Usually I approach by asking the following question: if this was an open source-library or service (e.g. like elasticsearch), would you use it? If not, then its probably not a great candidate for a separate thing - lets try and come up with something else. One way to handle CI/CD is using standardized pipelines e.g. you tag your repo with a tag `app:node` or `lib:js` and the github org pipeline scanner will find it and assign the standard `app:node` or `lib:js` pipeline to it. A way that I like better but most tools unfortunatley don't support it yet is for the infra teams to publish libraries that are essentially functions taking some parameters and generating (standard) pipelines/configuration. Those can then be tracked together the same as other dependencies.

okino 4y ago

Good articulation of why things don’t quite play out that way within a company here https://rushjs.io/pages/intro/why_mono/ https://rushjs.io/pages/intro/why_mono/

spion 4y ago
For that particular article, I would respond with the following: 1. I don't see how this is a problem specific to polyrepos. Have an "open PRs" link in the onboarding handbook that gives you a view of pull requests from all repos in the organization. GitHub automatically shows you notifications from all repos. If engineers still chose to focus on one or two repos after that, I'm not sure why. - Have a (Grafana) dashboard where you can see the latest / newest stuff. Use standard GH tools you use for OSS, such as follows etc to keep up. 2. Don't prematurely split into multiple repos. "No monorepo" doesn't mean not having poly-package repos. It means thinking what the sensible (library or service) API boundary is - treating your projects as you would treat library / service development. In this case a separate repo with lib3, lib2 and lib1 sounds like a good way to go - at most one repo per orthogonal internal framework (e.g. core-react-components). Repo dependency chains should be as shallow as possible, and differenting between public and internal packages is important. 3. Help other teams upgrade. If you are responsible for repo A, once you publish a new version tagged appropriately with semver, use the dashboard to look at your dependants and work with them (or rather, for them) to upgrade. Think of your dependants as internal customers, and make sure you add enough value for them to justify the upgrade effort. Cultivate a culture that values updates. 4. There are other alternatives to `npm link` e.g. see `yalc` https://github.com/wclr/yalc https://github.com/wclr/yalc Another pet peeve of mine is that the real issues get lost when you try to generalize. The article attempts to do this but that makes it hard to evaluate its claims. The best way to evaluate (alternative) solutions is to take a more concrete example repo. For scalability of this model I'll just point to the OSS community; individual maintainers often several dozen active repositories, but also they have an API contract worthy of a documentation website, versioning scheme and planned deprecation, and they typically avoid cross-project dependencies

tveita 4y ago

Splitting your code over many repositories massively increases the cost of refactoring, making it that less likely that your system will be "well factored". Besides the "N pull requests" problem you now lose history whenever you move a file across repo boundaries, and you'll eventually have straggler projects staying on old versions for years - so switching to a new way of doing something essentially means supporting both versions forever. Code for common needs gets duplicated, or worse, split out into yet another repository and /then/ duplicated, because no one wants to check if any of 100 repositories rely on the buggy behaviour they want to fix. I find this to be this a level headed explanation of the advantages of monorepos: https://danluu.com/monorepo/ https://danluu.com/monorepo/. I'm surprised to see so many comments summarily dismissing them, as if sanely managing thousands of smaller interdependent repositories doesn't require at least as much investment in custom tooling.

spion 4y ago
> Splitting your code over many repositories massively increases the cost of refactoring, making it that less likely that your system will be "well factored" The project should stay a full monolith until the factoring is more clear. > I find this to be this a level headed explanation of the advantages of monorepos: https://danluu.com/monorepo/ https://danluu.com/monorepo/. There are two issues I have with this article. One of them is when it describes drawbacks of multiple repositories, its not specific enough e.g. > That sounds like it ought to be straightforward, but in practice, most solutions are cumbersome and involve a lot of overhead. The other is that it assumes you have Google scale of resources to throw at the problem. If thats the case you can make anything work / monorepos or polyrepos. The issue is that small-to-medium sized organizations will not be prepared to invest the amount of resources needed to keep a monorepo working well, as the org will largly need to rely on existing available (OSS) tools which often have poor monorepo support. (Bazelifying everything has a significant cost, bazel rules are often not generic enough to work with e.g. the variety of JS ecosystem tools)
preseinger 4y ago
Only if you break your code up on boundaries that don't reflect reality. If one business change requires code changes to ~every module in your source tree, then of course microservices, and therefore separate repos per service or whatever, make no sense at all.

jogjayr 4y ago

> Individual projects should be able to upgrade dependencies separately. No, that way lies sorrow and despair. There should only be one version of any dependency being used in your company. Any deviations should require like VP-level approvals or something.