7 ms·

The more I've come to rely on techniques like canary testing and opt-in version upgrades the more firmly I believe one of the main motivations for monorepos is

The more I've come to rely on techniques like canary testing and opt-in version upgrades the more firmly I believe one of the main motivations for monorepos is flawed: at any given time there may not be a fact of the matter as to which single version of an app or service is running in an environment. At places I’ve worked when we thought about canary testing we ignored the fact that there were multiple versions of the software running in parallel, we classified it as part of an orchestration process and not a reality about the code or env, but we really did have multiple versions of a service running at once, sometimes for days. Similarly if you’ve got a setup where you can upgrade users/regions/etc piecemeal (opt-in or by other selection criteria) I don’t know how you reflect this in a monorepo. I’m curious how Google actually does this as I recall they have offered user opt-in upgrades for Gmail. My suspicion is this gets solved with something like directories ./v2/ and ./v3/ — but that’s far from ideal.

ASinclair 5y ago
> I’m curious how Google actually does this Branches are cut for releases. Binaries are versioned.
coryrc 5y ago
Only binaries are released. Binaries are timestamped and linked to a changelist. The "opt-in upgrades" are all live code. I know more than a few "foo" "foo2" directories, but I wouldn't want an actively-delivered, long-running service to be living in a feature branch so I would still expect anyone to be using a similar naming scheme.
deleted 5y ago
[deleted]
eximius 5y ago
that doesnt seem like a problem with monorepos (or otherwise). youd just need to tag your releases, right?
- aranchelk 5y ago
  I don't think it's a problem, rather I'm challenging a touted benefit. In large monorepos the supposition is you've got a class of compatible apps and services bundled together. Version dependencies are somewhat implicit: the correct version for each project to interoperate is whatever was checked-in together in your commit. I don't know how it works in practice at different orgs, but there's certainly an idea I've heard repeated that you can essentially test, build, and deploy your monorepo atomically, but the reality in my experience is you can't escape the need to think about compatibility across multiple versions of services once you use techniques like canary testing or targeted upgrades.
  - staticassertion 5y ago
    You still have to think about compatibility across versions - that does not go away in a monorepo, and you should use protocols that enforce compatible changes. The monorepo just tells you that all tests pass across your entire codebase given a change you made to some other part.
    - eximius 5y ago
      thats fair. you require reasonable deployment intervals and may need to wait to merge based on deployment. Workflow actions that can check whether a commit is deployed in a given environment are invaluable
      - jeffbee 5y ago
        > may need to wait to merge based on deployment Again, this fundamentally misunderstands the purpose of the source code repo and how it relates to the build artifacts deployed in production. If you are waiting for something to happen in production before landing some change, that tells me right there you have committed some kind of serious error.
        joshuamorton 5y ago
jeffbee 5y ago
> the main motivations for monorepos is flawed: at any given time there may not be a fact of the matter as to which single version of an app or service is running in an environment. Your understanding of the motivations for monorepo is flawed. I've never heard anyone even advocate for this as a reason for monorepos. For some actual reasons people use monorepos, see https://danluu.com/monorepo/ https://danluu.com/monorepo/ Regarding your question, which I re-emphasize has got nothing to do with the arrangement of the source code, the solution is to simply treat your protocol as API and follow these rules: 1) Follow "Postel's Law", accepting to the best of your abilities unknown messages and fields; 2) never change the meaning of anything in an existing protocol. Do not change between incompatible types, or change an optional item to required; 3) Never re-use a retired aspect of the protocol with a different meaning; 4) Generally do not make any incompatible change to an existing protocol. If you must change the semantics, then you are making a new protocol, not changing the old one, > I don’t know how you reflect this in a monorepo You don't. Why would the deployed version of some application be coded in your repo? It's simply a fact on the ground and there's no reason for that to appear in source control.
- aranchelk 5y ago
  We may just be talking past each other, but in the link you provided, sections "Simplified dependencies" and (to a lesser extent) "Cross-project changes" are pretty much exactly what I'm talking about.
  - howinteresting 5y ago
    In general, it is a good practice to try and maximize compile-time resolution of dependencies and minimize network resolution of them. Services are great when the working set doesn't fit in RAM or the different parts have different hardware needs, but trying to make every little thing its own service is foolish. Doing so makes this a less pertinent problem.
  - joshuamorton 5y ago
    They aren't, because those discussions are all related to link-time stuff (if I update foo.h and bar.c that depends on foo.h, I can do so atomically, because those are built into the same artifact). As soon as you discuss network traffic (or really anything that crosses an RPC boundary), things get more complicated, but none of that has anything to do with a monorepo, and monorepos still sometimes simplify things. So there's a few tools that are common: feature flags, 3 stage-rollouts, and probably more that are relevant, but let's dive into those first two. Feature "flags" are often dynamically scoped and runtime-modifiable. You can change a feature flag via an RPC, without restarting the binary running. This is done by having something along the lines of if (condition_that_enables_feature()) { do_feature_thing() } else { do_old_thing() } A/B testing tools like optimizely and co provide this, and there are generic frameworks too. `condition_that_enables_feature()`, here is a dynamic function that may change value based on the time of day, the user, etc. (think something like `hash(user.username).startswith(b'00') and user.locale == 'EN'`). The tools allow you to modify these conditions and push and change the conditions all without restarts. That's how you get per-user opt-in to certain behaviors. Fundamentally, you might have an app that is capable of serving two completely different UIs for the same user journey. Then you have "3-phase" updates. In this process, you have a client and server. You want to update them to use "v2" of some api, that's totally incompatible with v1. You start by updating the server to accept requests in either v1 or v2 format. That's stage one. Then you update the clients to sent requests in v2 format. That's stage two. Then you remove all support for v1. That's stage three. When you canary a new version of a binary, you'll have the old version that only supports v1, and the canary version that supports v1 and v2. If it's the server, none of the clients use v2 yet, so this is fine. If it's the client, you've already updated the server to support v2, so it works fine. Note again that all of this happens whether or not you use a monorepo.
deleted 5y ago
[deleted]

ASinclair 5y ago

> I’m curious how Google actually does this Branches are cut for releases. Binaries are versioned.

coryrc 5y ago

Only binaries are released. Binaries are timestamped and linked to a changelist. The "opt-in upgrades" are all live code. I know more than a few "foo" "foo2" directories, but I wouldn't want an actively-delivered, long-running service to be living in a feature branch so I would still expect anyone to be using a similar naming scheme.

deleted 5y ago

[deleted]

eximius 5y ago

that doesnt seem like a problem with monorepos (or otherwise). youd just need to tag your releases, right?

aranchelk 5y ago
I don't think it's a problem, rather I'm challenging a touted benefit. In large monorepos the supposition is you've got a class of compatible apps and services bundled together. Version dependencies are somewhat implicit: the correct version for each project to interoperate is whatever was checked-in together in your commit. I don't know how it works in practice at different orgs, but there's certainly an idea I've heard repeated that you can essentially test, build, and deploy your monorepo atomically, but the reality in my experience is you can't escape the need to think about compatibility across multiple versions of services once you use techniques like canary testing or targeted upgrades.
- staticassertion 5y ago
  You still have to think about compatibility across versions - that does not go away in a monorepo, and you should use protocols that enforce compatible changes. The monorepo just tells you that all tests pass across your entire codebase given a change you made to some other part.
  - eximius 5y ago
    thats fair. you require reasonable deployment intervals and may need to wait to merge based on deployment. Workflow actions that can check whether a commit is deployed in a given environment are invaluable
    - jeffbee 5y ago
      > may need to wait to merge based on deployment Again, this fundamentally misunderstands the purpose of the source code repo and how it relates to the build artifacts deployed in production. If you are waiting for something to happen in production before landing some change, that tells me right there you have committed some kind of serious error.
      - joshuamorton 5y ago

jeffbee 5y ago

> the main motivations for monorepos is flawed: at any given time there may not be a fact of the matter as to which single version of an app or service is running in an environment. Your understanding of the motivations for monorepo is flawed. I've never heard anyone even advocate for this as a reason for monorepos. For some actual reasons people use monorepos, see https://danluu.com/monorepo/ https://danluu.com/monorepo/ Regarding your question, which I re-emphasize has got nothing to do with the arrangement of the source code, the solution is to simply treat your protocol as API and follow these rules: 1) Follow "Postel's Law", accepting to the best of your abilities unknown messages and fields; 2) never change the meaning of anything in an existing protocol. Do not change between incompatible types, or change an optional item to required; 3) Never re-use a retired aspect of the protocol with a different meaning; 4) Generally do not make any incompatible change to an existing protocol. If you must change the semantics, then you are making a new protocol, not changing the old one, > I don’t know how you reflect this in a monorepo You don't. Why would the deployed version of some application be coded in your repo? It's simply a fact on the ground and there's no reason for that to appear in source control.

aranchelk 5y ago
We may just be talking past each other, but in the link you provided, sections "Simplified dependencies" and (to a lesser extent) "Cross-project changes" are pretty much exactly what I'm talking about.
- howinteresting 5y ago
  In general, it is a good practice to try and maximize compile-time resolution of dependencies and minimize network resolution of them. Services are great when the working set doesn't fit in RAM or the different parts have different hardware needs, but trying to make every little thing its own service is foolish. Doing so makes this a less pertinent problem.
- joshuamorton 5y ago
  They aren't, because those discussions are all related to link-time stuff (if I update foo.h and bar.c that depends on foo.h, I can do so atomically, because those are built into the same artifact). As soon as you discuss network traffic (or really anything that crosses an RPC boundary), things get more complicated, but none of that has anything to do with a monorepo, and monorepos still sometimes simplify things. So there's a few tools that are common: feature flags, 3 stage-rollouts, and probably more that are relevant, but let's dive into those first two. Feature "flags" are often dynamically scoped and runtime-modifiable. You can change a feature flag via an RPC, without restarting the binary running. This is done by having something along the lines of if (condition_that_enables_feature()) { do_feature_thing() } else { do_old_thing() } A/B testing tools like optimizely and co provide this, and there are generic frameworks too. `condition_that_enables_feature()`, here is a dynamic function that may change value based on the time of day, the user, etc. (think something like `hash(user.username).startswith(b'00') and user.locale == 'EN'`). The tools allow you to modify these conditions and push and change the conditions all without restarts. That's how you get per-user opt-in to certain behaviors. Fundamentally, you might have an app that is capable of serving two completely different UIs for the same user journey. Then you have "3-phase" updates. In this process, you have a client and server. You want to update them to use "v2" of some api, that's totally incompatible with v1. You start by updating the server to accept requests in either v1 or v2 format. That's stage one. Then you update the clients to sent requests in v2 format. That's stage two. Then you remove all support for v1. That's stage three. When you canary a new version of a binary, you'll have the old version that only supports v1, and the canary version that supports v1 and v2. If it's the server, none of the clients use v2 yet, so this is fine. If it's the client, you've already updated the server to support v2, so it works fine. Note again that all of this happens whether or not you use a monorepo.

deleted 5y ago

[deleted]