8 ms·

The advantage of a monorepo in this particular case is that it makes easy things easy: if you want to remove a parameter of a function in some library and that

by bla3 5y ago

The advantage of a monorepo in this particular case is that it makes easy things easy: if you want to remove a parameter of a function in some library and that function has just a few callers in dependent executables, you can just do that in a single commit. Without a monorepo, you have to do the full-blown iterative rollout described in the OP even for small changes, if they cross VCS boundaries.

draw_down 5y ago
Exactly, I don’t understand what is so difficult to get here.
lkschubert8 5y ago
Wouldn't you still have to do the iterative thing to avoid temporary outages during deployments? Or do you try to synchronize all deployments?
- sparsely 5y ago
  I don't think synchronized deployments is really possible - you'd have to either still do the iterative thing, or possibly have some versioning system in place
  - bluGill 5y ago
    It is possible for trivial cases. What I do in my basement for example - though even there I have come to prefer keeping things intentionally unsynchronized: it ensures that after updates I have some system that still works.
- bpicolo 5y ago
  It takes the guesswork out of library migrations. API migrations still need forwards/backwards compat hygiene, unless you blue/green your entire infrastructure to compatible versions, which is possible but not necessarily practical
- rohitpaulk 5y ago
  If the change is in a shared library (and not a service shared over the network), it's fine to change all usages at once. Deploying a service wouldn't affect the others. If the change affects the public interface of a service, then there's no option but to make your changes backward-compatible.
  - tantalor 5y ago
    > no option but to... Not necessarily; you can accept downtime/breakage instead. That is always an option!
    - bluGill 5y ago
      You must not work on safety critical code where downtime/breakage mean people die.
      - ralmeida 5y ago
        Most people don't, to be fair. The case where zero-downtime strategies are adopted for (at least) a questionable ROI is far more common.
      - closeparen 5y ago
        I hope you're not doing anything "safety critical" with general purpose computers, let alone microservices.
        bluGill 5y ago
        Well the arm CPUs we use are in general purpose computers as well. Though you are correct, we don't follow the same practices as general purpose computers.
      - marcosdumay 5y ago
        I hope nobody's life depends on the uptime of a web-based distributed system. But, well, I also expect nobody's life to depend on it. There would be a short window between people getting into that situation and they not have any life to depend on anything.
- goodpoint 5y ago
  Most likely bla3 is referring to statically linked applications. This is the big downside of monorepos: they strongly encourage tight coupling and poor modularity.
  - tantalor 5y ago
    > they strongly encourage tight coupling and poor modularity No, that's not true. Why would you say that?
    - bluGill 5y ago
      Because they do nothing to make it hard to add a coupling or break modularity. You should of course use good discipline to ensure that doesn't happen. Compared to mutli-repo it is a lot easier to violate coupling and modularity and not be detected. Anyone who is using a monorepo needs to be aware of this downside and deal with it. There are other downsides of multi-repo, and those dealing with them need to be aware of those and mitigate them. There is no perfect answer, just compromises.
    - majormajor 5y ago
      They make it easy, and then human nature and dev laziness does the rest. If you can reach across the repo and import any random piece of code, you end up with devs doing just that. It's a huge huge pain to try to untangle later. That's why tools like Bazel are strict about visibility and put more friction and explicitness on those sorts of things. But this tends to not be the first thing at the top of people's minds when starting a new project... so in the monorepos I've worked on, it's never been noticed until it's too late to easily fix.
- sam0x17 5y ago
  If you design with a "no deprecations" mentality and deploy backend before frontend, in _most cases_ this isn't an issue -- the frontend code that needs the new table or column or endpoint that doesn't exist yet won't run until those things are deployed, and the new backend endpoints will be fully backwards compatible with the old frontend, so no issues. You don't even need to be that dogmatic to make this work either -- simply stipulating backwards compatibility between the two previous deploys should be sufficient. The better version of this is simply versioning your backend and frontend but I've never been that fancy.
- fwip 5y ago
  Depends on how you deploy and your scaling strategy. e.g: If your smallest deployable unit is a Kubernetes pod, and all your affected applications live in that pod, you can treat it as a private change.
echelon 5y ago
This is the reason for monorepos. It's not about migrating APIs or coordinating deployments. That's an impossible problem to solve with your repo. It's to update libraries and shared code uniformly and patch dependencies (eg. for vulns) all in one go. Imagine updating Guava 1.0 -> 2.0. Either you require each team to do this independently over the course of several months with no coordination, or in a monorepo, one person can update every single project and service with relative ease. Let's say there's an npm vuln in leftpad 5.0. You can update everything to leftpad 5.0.1 at once and know that everything has been updated. Then you just tell teams to deploy. (Caveat: this doesn't really work as cleanly for a dynamically typed language like javascript, but it's a world wonder in a language like Java.) I can't fathom how hard it would be to coordinate all of these changes with polyrepos. You'd have to burden every team with a required change and force them to accommodate. Someone not familiar with the problem has to take time out of their day or week to learn the new context. Then search and apply changes. And there's no auditability or guarantee everyone did it. Some isolated or unknown repos somewhere won't ever see the upgrade. But in a monorepo, you're done in a day. Now, here's a key win: you're really at an advantage when updating "big things". Like getting all apps on gRPC. Or changing the metrics system wholesale. These would be year long projects in a polyrepo world. With monorepos, they're almost an afterthought. Monorepos are magical at scale. Until you experience one, it's really hard to see how easy this makes life for big scale problems.
- thelopa 5y ago
  Do you really feel that confident working in another team’s code base? I work in a multi-repo company, and almost every time I’ve gotten a patch from outside my team it’s been wrong in some way. Why would I want to make it easier for people who don’t understand (and aren’t interested in understanding) my project to land code in it?
  - z0r 5y ago
    Because any company reaching monorepo scale will have integration tests that cut across the boundaries of your projects. It's possible for an outside contribution to break your corner of the repo, but the flip side is that you will know much more quickly if your own changes break another part of the repo.
pierrebai 5y ago
In multi-repo, on all projects I've worked on, there is no such problem. First, inter-repo dependencies are managed by pulling specific a commit or tag. So changing a library has zero effect on the program depending on that library. Second, if you are introducing breaking change and you know not all client will want the change, you can have multiple branches. No, you do not want this as the default choice and not on the long term, but for the short-term transition, that is possible. From that point on, clients of the library can upgrade to new versions with the changes on their own schedule. The client is never forced to upgrade until there a feature is absolutely needs. The library is not forced to support two versions of code. That last point is not trivial. If every braking change need to go through this dual-support period in the same single code base it can become a support and testing nightmare. You need to duplicate tests and as the number of such dual-version of API increase, the compatibility matrix grows exponentially. This is entirely avoided in the multi-repos scenario.
- thayne 5y ago
  The flip side of this is that if you need a change to go out everywhere, say because it fixes a security vulnerability, or fixes a bug that affects all users, or you want to remove the old behavior from a service in a reasonable amount of time, then you have to update the commit/version/whatever in every dependent project as well, recursively. And I've seen the pain of doing that cause bad incentives, like not splitting up projects that really need to be, or delaying critical fixes because you don't want to update depencies twice.
- deschutes 5y ago
  If it is possible to have multiple revisions of libraries you will have multiple revisions of libraries. I've seen this play out "at scale" and what happens in practice is software rots until some kind of crisis or feature demands an upgrade. At which point somebody creates another set of branches to solve precisely their problem and nothing else. Maybe it's just where I've worked, but atomic commit carries with it some strong cultural norms that make it really tractable to have one version of everything. If code compiles and automated tests pass, the change is safe and may be committed unilaterally. Inevitably things go wrong, but the post mortems for incidents don't lead back to the lack of permission from the affected projects. I wouldn't maintain software that can kill people in this way, but for everything else it strikes a nice balance.

by bla3 5y ago

draw_down 5y ago

Exactly, I don’t understand what is so difficult to get here.

lkschubert8 5y ago

Wouldn't you still have to do the iterative thing to avoid temporary outages during deployments? Or do you try to synchronize all deployments?

sparsely 5y ago
I don't think synchronized deployments is really possible - you'd have to either still do the iterative thing, or possibly have some versioning system in place
- bluGill 5y ago
  It is possible for trivial cases. What I do in my basement for example - though even there I have come to prefer keeping things intentionally unsynchronized: it ensures that after updates I have some system that still works.
bpicolo 5y ago
It takes the guesswork out of library migrations. API migrations still need forwards/backwards compat hygiene, unless you blue/green your entire infrastructure to compatible versions, which is possible but not necessarily practical
rohitpaulk 5y ago
If the change is in a shared library (and not a service shared over the network), it's fine to change all usages at once. Deploying a service wouldn't affect the others. If the change affects the public interface of a service, then there's no option but to make your changes backward-compatible.
- tantalor 5y ago
  > no option but to... Not necessarily; you can accept downtime/breakage instead. That is always an option!
  - bluGill 5y ago
    You must not work on safety critical code where downtime/breakage mean people die.
    - ralmeida 5y ago
      Most people don't, to be fair. The case where zero-downtime strategies are adopted for (at least) a questionable ROI is far more common.
    - closeparen 5y ago
      I hope you're not doing anything "safety critical" with general purpose computers, let alone microservices.
      - bluGill 5y ago
        Well the arm CPUs we use are in general purpose computers as well. Though you are correct, we don't follow the same practices as general purpose computers.
    - marcosdumay 5y ago
      I hope nobody's life depends on the uptime of a web-based distributed system. But, well, I also expect nobody's life to depend on it. There would be a short window between people getting into that situation and they not have any life to depend on anything.
goodpoint 5y ago
Most likely bla3 is referring to statically linked applications. This is the big downside of monorepos: they strongly encourage tight coupling and poor modularity.
- tantalor 5y ago
  > they strongly encourage tight coupling and poor modularity No, that's not true. Why would you say that?
  - bluGill 5y ago
    Because they do nothing to make it hard to add a coupling or break modularity. You should of course use good discipline to ensure that doesn't happen. Compared to mutli-repo it is a lot easier to violate coupling and modularity and not be detected. Anyone who is using a monorepo needs to be aware of this downside and deal with it. There are other downsides of multi-repo, and those dealing with them need to be aware of those and mitigate them. There is no perfect answer, just compromises.
  - majormajor 5y ago
    They make it easy, and then human nature and dev laziness does the rest. If you can reach across the repo and import any random piece of code, you end up with devs doing just that. It's a huge huge pain to try to untangle later. That's why tools like Bazel are strict about visibility and put more friction and explicitness on those sorts of things. But this tends to not be the first thing at the top of people's minds when starting a new project... so in the monorepos I've worked on, it's never been noticed until it's too late to easily fix.
sam0x17 5y ago
If you design with a "no deprecations" mentality and deploy backend before frontend, in _most cases_ this isn't an issue -- the frontend code that needs the new table or column or endpoint that doesn't exist yet won't run until those things are deployed, and the new backend endpoints will be fully backwards compatible with the old frontend, so no issues. You don't even need to be that dogmatic to make this work either -- simply stipulating backwards compatibility between the two previous deploys should be sufficient. The better version of this is simply versioning your backend and frontend but I've never been that fancy.
fwip 5y ago
Depends on how you deploy and your scaling strategy. e.g: If your smallest deployable unit is a Kubernetes pod, and all your affected applications live in that pod, you can treat it as a private change.

echelon 5y ago

This is the reason for monorepos. It's not about migrating APIs or coordinating deployments. That's an impossible problem to solve with your repo. It's to update libraries and shared code uniformly and patch dependencies (eg. for vulns) all in one go. Imagine updating Guava 1.0 -> 2.0. Either you require each team to do this independently over the course of several months with no coordination, or in a monorepo, one person can update every single project and service with relative ease. Let's say there's an npm vuln in leftpad 5.0. You can update everything to leftpad 5.0.1 at once and know that everything has been updated. Then you just tell teams to deploy. (Caveat: this doesn't really work as cleanly for a dynamically typed language like javascript, but it's a world wonder in a language like Java.) I can't fathom how hard it would be to coordinate all of these changes with polyrepos. You'd have to burden every team with a required change and force them to accommodate. Someone not familiar with the problem has to take time out of their day or week to learn the new context. Then search and apply changes. And there's no auditability or guarantee everyone did it. Some isolated or unknown repos somewhere won't ever see the upgrade. But in a monorepo, you're done in a day. Now, here's a key win: you're really at an advantage when updating "big things". Like getting all apps on gRPC. Or changing the metrics system wholesale. These would be year long projects in a polyrepo world. With monorepos, they're almost an afterthought. Monorepos are magical at scale. Until you experience one, it's really hard to see how easy this makes life for big scale problems.

thelopa 5y ago
Do you really feel that confident working in another team’s code base? I work in a multi-repo company, and almost every time I’ve gotten a patch from outside my team it’s been wrong in some way. Why would I want to make it easier for people who don’t understand (and aren’t interested in understanding) my project to land code in it?
- z0r 5y ago
  Because any company reaching monorepo scale will have integration tests that cut across the boundaries of your projects. It's possible for an outside contribution to break your corner of the repo, but the flip side is that you will know much more quickly if your own changes break another part of the repo.

pierrebai 5y ago

In multi-repo, on all projects I've worked on, there is no such problem. First, inter-repo dependencies are managed by pulling specific a commit or tag. So changing a library has zero effect on the program depending on that library. Second, if you are introducing breaking change and you know not all client will want the change, you can have multiple branches. No, you do not want this as the default choice and not on the long term, but for the short-term transition, that is possible. From that point on, clients of the library can upgrade to new versions with the changes on their own schedule. The client is never forced to upgrade until there a feature is absolutely needs. The library is not forced to support two versions of code. That last point is not trivial. If every braking change need to go through this dual-support period in the same single code base it can become a support and testing nightmare. You need to duplicate tests and as the number of such dual-version of API increase, the compatibility matrix grows exponentially. This is entirely avoided in the multi-repos scenario.

thayne 5y ago
The flip side of this is that if you need a change to go out everywhere, say because it fixes a security vulnerability, or fixes a bug that affects all users, or you want to remove the old behavior from a service in a reasonable amount of time, then you have to update the commit/version/whatever in every dependent project as well, recursively. And I've seen the pain of doing that cause bad incentives, like not splitting up projects that really need to be, or delaying critical fixes because you don't want to update depencies twice.
deschutes 5y ago
If it is possible to have multiple revisions of libraries you will have multiple revisions of libraries. I've seen this play out "at scale" and what happens in practice is software rots until some kind of crisis or feature demands an upgrade. At which point somebody creates another set of branches to solve precisely their problem and nothing else. Maybe it's just where I've worked, but atomic commit carries with it some strong cultural norms that make it really tractable to have one version of everything. If code compiles and automated tests pass, the change is safe and may be committed unilaterally. Inevitably things go wrong, but the post mortems for incidents don't lead back to the lack of permission from the affected projects. I wouldn't maintain software that can kill people in this way, but for everything else it strikes a nice balance.