7 ms·

My only counter argument here, is when those 4 things deploy independently. Sometimes, people will get tricked into thinking a code change is atomic because it

by taeric 2y ago

My only counter argument here, is when those 4 things deploy independently. Sometimes, people will get tricked into thinking a code change is atomic because it is in one commit, when it will lead to a mixed fleet because of deployment realities. In that world, having them separate is easier to work with, as you may have to revert one of the deployments separately from the others.

derefr 2y ago
That's just an argument for not doing "implicit GitOps", treating the tip of your monorepo's main branch as the source-of-truth on the correct deployment state of your entire system. ("Implicit GitOps" sorta-kinda works when you have a 1:1 correspondence between repos and deployable components — though not always! — but it isn't tenable for a monorepo.) What instead, then? Explicit GitOps. Explicit, reified release specifications (think k8s resource manifests, or Erlang .relup files), one per separately-deploy-cadenced component. If you have a monorepo, then these live also as a dir in the monorepo. CD happens only when these files change. With this approach, a single PR can atomically merge code and update one or more release specifications (triggering CD for those components), if and when that is a sensible thing to do. But there can also be separate PRs for updating the code vs. "integrating and deploying changes" to a component, if-and-when that is sensible.
- scubbo 2y ago
  ...I can't believe I'd never thought about the fact that a "Deployment Repo" can, in fact, just be a directory within the Code Repo. Interesting thought - thanks!
- taeric 2y ago
  I mean... sure? Yes, if you add extra structure on top of your code that is there to model the deployments, then you get a bit closer to modeling your deployments. Isn't that the exact argument for why you might want multiple repositories, as well?
- ramchip 2y ago
  > With this approach, a single PR can atomically merge code and update one or more release specifications (triggering CD for those components), if and when that is a sensible thing to do. How do you avoid the chicken-and-egg problem? Like if the k8s manifest contains a container tag, and the tag is created by CI when the PR is merged to main, it would seem you can’t add code and deploy that code in the same PR.
  - derefr 2y ago
    I haven't seen this approach in action personally, but I assume you'd set it up similarly to how Homebrew builds + pins bottles for formulae: - PR creation (or any update to the PR branch by a human) would trigger a CI workflow to build+push the container; - if this succeeds, the workflow would push a commit to the PR feature branch that pins the container ref; - and the base branch would have a branch protection rule that makes the success of this CI workflow a prerequisite for PR mergeability. In the explicit-GitOps-in-monorepo case, you'd probably want to do this by having the PR author modify the release spec file of each component they want to build a release for, replacing the old static ref with a temporary symbolic ref meaning "hey CI system, calculate this." Then the CI's added commit would rewrite those temporary symbolic refs into new static ones. --- Although, that's all assuming that you even want to do CI builds in the first place. If you're developing containerized server software — rather than an operating system, a web browser, etc — then it shouldn't matter where you're building, and there isn't much impetus for deterministic, auditable builds, either. So why bottleneck builds by shoving them all onto some wimpy CI system—when every engineer already has a powerful workstation that can do builds often 4x faster, sitting there idle? Here's what I call "local-first explicit GitOps": 1. you give your engineers the ability to push refs to the container registry (but you never use symbolic refs, always static manifest SHAs, so a malicious engineer can't do anything dangerous with this. 2. you add a script to your monorepo, that an engineer can run against their feature branch, that'll notice their current branch's local symbolic refs, and build+push containers locally in order to rewrite those into static refs. 3. On success, the script can use `gh` (or equivalent) to trigger PR creation. Now every PR will come into existence already pinning valid new build artifacts by their static refs!
  - someone654 2y ago
lmz 2y ago
Isn't a mixed fleet always the case once you have more than one server and do rolling updates?
- taeric 2y ago
  Yes. And if you structure your code to explicitly do this, it is a lot easier to reason about.
- xyzzy123 2y ago
  Sort of; at medium scale you can blue/green your whole system out of the monorepo (even if its say 20 services) in k8s and flip the ingresses to cut over during release. Of course k8s not required, you can do it in straight IaC etc (i.e deploy a whole parallel system and switch). It's still "mixed fleet" in terms of any shared external resources (queues, db state, etc) but you can change service interfaces etc with impunity and not worry about compatibility / versioning between services. Throwing temporary compute at the problem can save a lot of busywork and/or thinking about integration problems. This stops being practical if you get _very_ big but at that point you presumably have more money and engineers to throw at the problem.
  - chipdart 2y ago
    > Sort of; at medium scale you can blue/green your whole system out of the monorepo (even if its say 20 services) in k8s and flip the ingresses to cut over during release. That's overall a bad idea, and negates the whole point of blue-green deployments. This is particularly bad if you have in place any form of distributed transaction. There are very good reasons why deployment strategies such as rolling deployments and one-box deployments were developed. You need to be able to gradually roll out a change to prevent problems from escalating and causing downtime. If you use all that infrastructure to then flip global switches then you're building up all this infrastructure to the negate any of that with practices that invariably cause problems. And this is being done just because someone thinks it's a good idea to keep all code in a single repo?
    - xyzzy123 2y ago
hinkley 2y ago
If the version numbers of all services built from the PR are identical, you at least have a pretty clear trail to figuring out WTF happened. Even with a few services, we saw some pretty crunchy issues with people not understanding that service A had version 1.3.1234 of a module and Service B had version 1.3.1245 and that was enough skew to cause problems. Distinct repos tend to have distinct builds, and sooner or later one of the ten you're building will glitch out and have to be run twice, or the trigger will fail and it won't build at all until the subsequent merge, and having numbers that are close results in a false sense of confidence.
oftenwrong 2y ago
This is generally seen as a feature. Most developers won't need to confront the messy reality of the deployed "world". They can work within the relatively pure world of the code history. Of course, some will have to do the work of actually producing releases, performing deployments, and operating the systems.
chipdart 2y ago
> My only counter argument here, is when those 4 things deploy independently. This brings us to the elephant in the room: monorepo strategies are just naive and futile attempts at fixing the problems created by an absolute lack of support for versioning. If you support versioning, you don't care where and how you store your code. You just push the changes you need to push, and make sure you bump versions to reflect the nature of your changes. Consumers are left with the responsibility of picking which to consume, and when to consume it.
- dieortin 2y ago
  Versioning introduces a big maintenance cost, both on the side of the developer of a service and on the consumer side. On a monorepo you have a single version, and can be sure that everything works together on it. You are forced to address the costs of changing APIs continuously, but all components are always up to date. With multiple repos, versioning hell is real.
  - chipdart 2y ago
    > Versioning introduces a big maintenance cost, both on the side of the developer of a service and on the consumer side. Not really. From the producer side it only requires that a) you do not break your contract, b) when you need to break a contract, you bump your version number, c) when no one uses older versions, remove the code. From the consumer, it's even easier: you just need to point to the version you're consuming. What exactly do you think is the problem?
    - dezgeg 2y ago
      Bugs can and will happen, eg. when A depends on B and some developer accidentally makes a change to B that will pass B's tests but only break in A's use case. In monorepo case such bug can be noticed before the broken code hits master. In versioned case it might take long time (until B releases, and A takes the release into use). Knowing when no one uses older versions is also hard (especially when you can no longer grep for them, unlike a monorepo) so some people get into defensive mode and never delete or fix any interfaces because "someone might still be using it".

by taeric 2y ago

derefr 2y ago

That's just an argument for not doing "implicit GitOps", treating the tip of your monorepo's main branch as the source-of-truth on the correct deployment state of your entire system. ("Implicit GitOps" sorta-kinda works when you have a 1:1 correspondence between repos and deployable components — though not always! — but it isn't tenable for a monorepo.) What instead, then? Explicit GitOps. Explicit, reified release specifications (think k8s resource manifests, or Erlang .relup files), one per separately-deploy-cadenced component. If you have a monorepo, then these live also as a dir in the monorepo. CD happens only when these files change. With this approach, a single PR can atomically merge code and update one or more release specifications (triggering CD for those components), if and when that is a sensible thing to do. But there can also be separate PRs for updating the code vs. "integrating and deploying changes" to a component, if-and-when that is sensible.

scubbo 2y ago
...I can't believe I'd never thought about the fact that a "Deployment Repo" can, in fact, just be a directory within the Code Repo. Interesting thought - thanks!
taeric 2y ago
I mean... sure? Yes, if you add extra structure on top of your code that is there to model the deployments, then you get a bit closer to modeling your deployments. Isn't that the exact argument for why you might want multiple repositories, as well?
ramchip 2y ago
> With this approach, a single PR can atomically merge code and update one or more release specifications (triggering CD for those components), if and when that is a sensible thing to do. How do you avoid the chicken-and-egg problem? Like if the k8s manifest contains a container tag, and the tag is created by CI when the PR is merged to main, it would seem you can’t add code and deploy that code in the same PR.
- derefr 2y ago
  I haven't seen this approach in action personally, but I assume you'd set it up similarly to how Homebrew builds + pins bottles for formulae: - PR creation (or any update to the PR branch by a human) would trigger a CI workflow to build+push the container; - if this succeeds, the workflow would push a commit to the PR feature branch that pins the container ref; - and the base branch would have a branch protection rule that makes the success of this CI workflow a prerequisite for PR mergeability. In the explicit-GitOps-in-monorepo case, you'd probably want to do this by having the PR author modify the release spec file of each component they want to build a release for, replacing the old static ref with a temporary symbolic ref meaning "hey CI system, calculate this." Then the CI's added commit would rewrite those temporary symbolic refs into new static ones. --- Although, that's all assuming that you even want to do CI builds in the first place. If you're developing containerized server software — rather than an operating system, a web browser, etc — then it shouldn't matter where you're building, and there isn't much impetus for deterministic, auditable builds, either. So why bottleneck builds by shoving them all onto some wimpy CI system—when every engineer already has a powerful workstation that can do builds often 4x faster, sitting there idle? Here's what I call "local-first explicit GitOps": 1. you give your engineers the ability to push refs to the container registry (but you never use symbolic refs, always static manifest SHAs, so a malicious engineer can't do anything dangerous with this. 2. you add a script to your monorepo, that an engineer can run against their feature branch, that'll notice their current branch's local symbolic refs, and build+push containers locally in order to rewrite those into static refs. 3. On success, the script can use `gh` (or equivalent) to trigger PR creation. Now every PR will come into existence already pinning valid new build artifacts by their static refs!
- someone654 2y ago

lmz 2y ago

Isn't a mixed fleet always the case once you have more than one server and do rolling updates?

taeric 2y ago
Yes. And if you structure your code to explicitly do this, it is a lot easier to reason about.
xyzzy123 2y ago
Sort of; at medium scale you can blue/green your whole system out of the monorepo (even if its say 20 services) in k8s and flip the ingresses to cut over during release. Of course k8s not required, you can do it in straight IaC etc (i.e deploy a whole parallel system and switch). It's still "mixed fleet" in terms of any shared external resources (queues, db state, etc) but you can change service interfaces etc with impunity and not worry about compatibility / versioning between services. Throwing temporary compute at the problem can save a lot of busywork and/or thinking about integration problems. This stops being practical if you get _very_ big but at that point you presumably have more money and engineers to throw at the problem.
- chipdart 2y ago
  > Sort of; at medium scale you can blue/green your whole system out of the monorepo (even if its say 20 services) in k8s and flip the ingresses to cut over during release. That's overall a bad idea, and negates the whole point of blue-green deployments. This is particularly bad if you have in place any form of distributed transaction. There are very good reasons why deployment strategies such as rolling deployments and one-box deployments were developed. You need to be able to gradually roll out a change to prevent problems from escalating and causing downtime. If you use all that infrastructure to then flip global switches then you're building up all this infrastructure to the negate any of that with practices that invariably cause problems. And this is being done just because someone thinks it's a good idea to keep all code in a single repo?
  - xyzzy123 2y ago

hinkley 2y ago

If the version numbers of all services built from the PR are identical, you at least have a pretty clear trail to figuring out WTF happened. Even with a few services, we saw some pretty crunchy issues with people not understanding that service A had version 1.3.1234 of a module and Service B had version 1.3.1245 and that was enough skew to cause problems. Distinct repos tend to have distinct builds, and sooner or later one of the ten you're building will glitch out and have to be run twice, or the trigger will fail and it won't build at all until the subsequent merge, and having numbers that are close results in a false sense of confidence.

oftenwrong 2y ago

This is generally seen as a feature. Most developers won't need to confront the messy reality of the deployed "world". They can work within the relatively pure world of the code history. Of course, some will have to do the work of actually producing releases, performing deployments, and operating the systems.

chipdart 2y ago

> My only counter argument here, is when those 4 things deploy independently. This brings us to the elephant in the room: monorepo strategies are just naive and futile attempts at fixing the problems created by an absolute lack of support for versioning. If you support versioning, you don't care where and how you store your code. You just push the changes you need to push, and make sure you bump versions to reflect the nature of your changes. Consumers are left with the responsibility of picking which to consume, and when to consume it.

dieortin 2y ago
Versioning introduces a big maintenance cost, both on the side of the developer of a service and on the consumer side. On a monorepo you have a single version, and can be sure that everything works together on it. You are forced to address the costs of changing APIs continuously, but all components are always up to date. With multiple repos, versioning hell is real.
- chipdart 2y ago
  > Versioning introduces a big maintenance cost, both on the side of the developer of a service and on the consumer side. Not really. From the producer side it only requires that a) you do not break your contract, b) when you need to break a contract, you bump your version number, c) when no one uses older versions, remove the code. From the consumer, it's even easier: you just need to point to the version you're consuming. What exactly do you think is the problem?
  - dezgeg 2y ago
    Bugs can and will happen, eg. when A depends on B and some developer accidentally makes a change to B that will pass B's tests but only break in A's use case. In monorepo case such bug can be noticed before the broken code hits master. In versioned case it might take long time (until B releases, and A takes the release into use). Knowing when no one uses older versions is also hard (especially when you can no longer grep for them, unlike a monorepo) so some people get into defensive mode and never delete or fix any interfaces because "someone might still be using it".