5 ms·

Eventual consistency. The great thing about multi repo is the ease of decoupling the pieces so they can evolves separately (you can do that in monorepos too, bu

by shados 9y ago

Eventual consistency. The great thing about multi repo is the ease of decoupling the pieces so they can evolves separately (you can do that in monorepos too, but it's not quite as natural). You're free to PR changes gradually, making sure things work a couple of repos at a time, until you eventually get everything. If you can tolerate temporary inconsistencies, it allows you to scale to infinity, essentially for free.

joshuamorton 9y ago
>Eventual consistency. The great thing about multi repo is the ease of decoupling the pieces so they can evolves separately (you can do that in monorepos too, but it's not quite as natural). I don't see how you can do this any better in a multi-repo than a monorepo though, unless you mean to the extent of simultaneously having multiple versions of the same library in your transitive deps (and thus kind of kludgily sidestepping the diamond dependency issues). Would you mind elaborating? >You're free to PR changes gradually This is possible in a monorepo too, by much the same means, I'd expect: you define an adaptor that you slowly migrate everyone onto, deprecate the old thing, and then optionally remove the adaptor and deprecate it too. Am I missing something?
- monksy 9y ago
  The diamond problem is more due to the lack of tooling to alert you when that's going on. (Tooling on the building side and the CI/CD/Jenkins side) [Jenkins should be able to kick off other builds that depend on whats being built.
- shados 9y ago
  > I don't see how you can do this any better in a multi-repo than a monorepo though, unless you mean to the extent of simultaneously having multiple versions of the same library in your transitive deps (and thus kind of kludgily sidestepping the diamond dependency issues). Would you mind elaborating? Leaf repos (projects nothing depends on, like apps) can do literally whatever they want at any time without affecting anyone else. Right there is a big win. Dependencies can have multiple versions and the leaf can depend on whichever version they want at any given time. You can do the same thing in a monorepo if everything is in independant folders, but that's just the worse of both worlds. > you define an adaptor that you slowly migrate everyone onto No no. The way we do it is: make breaking change, people upgrade to it whenever (with gentle pushes so that we're eventually all on the same version sooner than later). No adapter, no transient state within a service. Just upgrade repos one by one until you got them all, no magic involved. This assumes that your repos represent loosely coupled components (micro services or micro apps).
  - joshuamorton 9y ago
    These two things you state only work if you have no shared deps. If I depend on A and B, and B also depends on A, I can't use whatever version of A I want, it has to be compatible with B. This means that I'm forced to delay my upgrade of A until B has done so. There longer the chain of deps is the worse this is. Even if everyone takes just couple days to upgrade, which ime is generous, your leaves end up being forced to wait weeks to upgrade in the worst case.
    - shados 9y ago
      Depends: if B declares that it is compatible with both versions of A (because it uses methods that did not break), then you sure can upgrade whenever you want (and potentially start using new methods that contained breaking change). If the change is breaking for B, then yeah, you have to wait until B upgrades (or you can upgrade it yourself!). During that time, other projects that don't depend on B can go on using the new A. The alternative is "stop the press, everyone is upgrading to the latest A NOOOOOOOOOW", which if the change is not automatable, might either be non-realistic, be pretty large in scope, or require you to never make drastic changes in A (which is tricky if A is a 3rd party you don't control). You can also just have these long-running transient state where somehow A is always compatible with everything no matter via compatibility wrappers. It's tradeoffs. I like our world where we don't have to migrate everything at the same time all the time, even with drastic changes. Works quite well for us, with thousands of repos and 10s of millions of lines of code. We enjoy the flexibility. It makes certain cross-project efforts harder. That's the tradeoff.
imiric 9y ago
This is great in theory, but it requires _a lot_ of discipline and responsibility around dependency management from the developers. What ends up happening in practice is that all these seemingly independent components have very strict—and sometimes even unspecified—dependencies between each other. People create "base" packages that all other packages have a strict or loose dependency on, so when that package changes, it's a guessing game if it introduced breaking changes downstream. With a monorepo, these dependencies are tracked and always visible, and integration testing between all dependent components becomes much easier. It's very unlikely that you'll find truly decoupled code bases within an organization. It goes against the point of grouping people to work on a common goal to begin with.
- shados 9y ago
  > This is great in theory, but it requires _a lot_ of discipline and responsibility around dependency management from the developers. Why? Again, the point is eventual consistency. If I make a breaking change, apps/services that are ready to migrate do so. The ones that aren't keep using the old version. Eventually, everyone's on the same page. The whole point is that it requires a lot less discipline.
Too 9y ago
Developers are way to lazy for this to work in practice. Imagine every second commit you make has to be backwards compatible, that for sure is a great way to effectively stop any refactoring of your code base, unless all your interfaces are already perfect and very fixed and you find this a feature. Or people will not care about partial cross repo dependencies resulting in randomly broken builds for others during a short time. I don't want to pull down a broken build just because some other team at that moment were only half way into pushing a feature. What should I do against that? Retry pull all repos and rebuild again? I think the problem is that people misuse their version control as a package manager. If you want that type of behaviour, with semver and all to manage compatibility, just use a package manager to manage your dependencies, not a multirepo git contraption. You can still store your packages in one repo each if you like but at least you now get some control over interface compatibility which is a requirenent when you have multiple repos.
- shados 9y ago
  > Imagine every second commit you make has to be backwards compatible No. The whole point is that they don't have to think about it. Aside for micro services used machine to machine with breaking changes that cannot exist in parallel, the point is that they don't have to worry about this. Do whatever you want, other projects do whatever the hell they want, and we just slowly go toward consistency as it becomes convenient until everyone's the same. Repeat. I can upgrade a dependency with a breaking change to my repo. The other team isn't ready yet so they keep using the old version (which works) until they are.

by shados 9y ago

joshuamorton 9y ago

>Eventual consistency. The great thing about multi repo is the ease of decoupling the pieces so they can evolves separately (you can do that in monorepos too, but it's not quite as natural). I don't see how you can do this any better in a multi-repo than a monorepo though, unless you mean to the extent of simultaneously having multiple versions of the same library in your transitive deps (and thus kind of kludgily sidestepping the diamond dependency issues). Would you mind elaborating? >You're free to PR changes gradually This is possible in a monorepo too, by much the same means, I'd expect: you define an adaptor that you slowly migrate everyone onto, deprecate the old thing, and then optionally remove the adaptor and deprecate it too. Am I missing something?

monksy 9y ago
The diamond problem is more due to the lack of tooling to alert you when that's going on. (Tooling on the building side and the CI/CD/Jenkins side) [Jenkins should be able to kick off other builds that depend on whats being built.
shados 9y ago
> I don't see how you can do this any better in a multi-repo than a monorepo though, unless you mean to the extent of simultaneously having multiple versions of the same library in your transitive deps (and thus kind of kludgily sidestepping the diamond dependency issues). Would you mind elaborating? Leaf repos (projects nothing depends on, like apps) can do literally whatever they want at any time without affecting anyone else. Right there is a big win. Dependencies can have multiple versions and the leaf can depend on whichever version they want at any given time. You can do the same thing in a monorepo if everything is in independant folders, but that's just the worse of both worlds. > you define an adaptor that you slowly migrate everyone onto No no. The way we do it is: make breaking change, people upgrade to it whenever (with gentle pushes so that we're eventually all on the same version sooner than later). No adapter, no transient state within a service. Just upgrade repos one by one until you got them all, no magic involved. This assumes that your repos represent loosely coupled components (micro services or micro apps).
- joshuamorton 9y ago
  These two things you state only work if you have no shared deps. If I depend on A and B, and B also depends on A, I can't use whatever version of A I want, it has to be compatible with B. This means that I'm forced to delay my upgrade of A until B has done so. There longer the chain of deps is the worse this is. Even if everyone takes just couple days to upgrade, which ime is generous, your leaves end up being forced to wait weeks to upgrade in the worst case.
  - shados 9y ago
    Depends: if B declares that it is compatible with both versions of A (because it uses methods that did not break), then you sure can upgrade whenever you want (and potentially start using new methods that contained breaking change). If the change is breaking for B, then yeah, you have to wait until B upgrades (or you can upgrade it yourself!). During that time, other projects that don't depend on B can go on using the new A. The alternative is "stop the press, everyone is upgrading to the latest A NOOOOOOOOOW", which if the change is not automatable, might either be non-realistic, be pretty large in scope, or require you to never make drastic changes in A (which is tricky if A is a 3rd party you don't control). You can also just have these long-running transient state where somehow A is always compatible with everything no matter via compatibility wrappers. It's tradeoffs. I like our world where we don't have to migrate everything at the same time all the time, even with drastic changes. Works quite well for us, with thousands of repos and 10s of millions of lines of code. We enjoy the flexibility. It makes certain cross-project efforts harder. That's the tradeoff.

imiric 9y ago

This is great in theory, but it requires _a lot_ of discipline and responsibility around dependency management from the developers. What ends up happening in practice is that all these seemingly independent components have very strict—and sometimes even unspecified—dependencies between each other. People create "base" packages that all other packages have a strict or loose dependency on, so when that package changes, it's a guessing game if it introduced breaking changes downstream. With a monorepo, these dependencies are tracked and always visible, and integration testing between all dependent components becomes much easier. It's very unlikely that you'll find truly decoupled code bases within an organization. It goes against the point of grouping people to work on a common goal to begin with.

shados 9y ago
> This is great in theory, but it requires _a lot_ of discipline and responsibility around dependency management from the developers. Why? Again, the point is eventual consistency. If I make a breaking change, apps/services that are ready to migrate do so. The ones that aren't keep using the old version. Eventually, everyone's on the same page. The whole point is that it requires a lot less discipline.

Too 9y ago

Developers are way to lazy for this to work in practice. Imagine every second commit you make has to be backwards compatible, that for sure is a great way to effectively stop any refactoring of your code base, unless all your interfaces are already perfect and very fixed and you find this a feature. Or people will not care about partial cross repo dependencies resulting in randomly broken builds for others during a short time. I don't want to pull down a broken build just because some other team at that moment were only half way into pushing a feature. What should I do against that? Retry pull all repos and rebuild again? I think the problem is that people misuse their version control as a package manager. If you want that type of behaviour, with semver and all to manage compatibility, just use a package manager to manage your dependencies, not a multirepo git contraption. You can still store your packages in one repo each if you like but at least you now get some control over interface compatibility which is a requirenent when you have multiple repos.

shados 9y ago
> Imagine every second commit you make has to be backwards compatible No. The whole point is that they don't have to think about it. Aside for micro services used machine to machine with breaking changes that cannot exist in parallel, the point is that they don't have to worry about this. Do whatever you want, other projects do whatever the hell they want, and we just slowly go toward consistency as it becomes convenient until everyone's the same. Repeat. I can upgrade a dependency with a breaking change to my repo. The other team isn't ready yet so they keep using the old version (which works) until they are.