3 ms·

So, the thing that is still buzzing in my head now and not mentioned in the article (maybe I didn't read carefully enough), what actually gets released into pro

So, the thing that is still buzzing in my head now and not mentioned in the article (maybe I didn't read carefully enough), what actually gets released into prod after a change is reviewed and merged. If the monorep contains let's say five different products and in a day only one of them gets a merge, then Blaze still builds all five and all five are released (based on successful integration testing)? OR only releases the changed product (and any other ones which depended on the changed one) EDIT: Also, the "canary" server is still for testing ? There may exist practically a set of canaries running very different versions ? Is there any correlation or any version "roll-up" constraints between various canaries ?

kyrra 8y ago
Releases are controlled by each team. For example, my team has multiple binaries that run in production. Some of them are stable and don't receive any active code changes but are pushed every time we do a release. But for those that don't get direct code changes, they will still pickup any changes to shared libraries. Canaries are live/production traffic only. When a release is deployed, it goes to the canary instances of a job first, and it will take a small subset of traffic. This allows the job owners to see if the new binary has any adverse effects before rolling out more widely. More details about canaries can be found here[0]. Once important thing to think about with Google's source control is that it is closer to SVN as far as versioning goes. There are no (for the most part) feature branches or anything like that. Everyone is always working on HEAD. when you do a release, you will start near HEAD when cutting the release. [0] https://landing.google.com/sre/sre-book/chapters/testing-reliability/ https://landing.google.com/sre/sre-book/chapters/testing-rel...
ASinclair 8y ago
Like kyrra said. Releases are handled by independently by each team. Teams are in various states of maturity in relation to their release practices. Some are fully automated. Some involve manual QA. I've supported teams across that continuum as a SETI at Google. I'm not as well-versed in canarying though I've set it up for a team or two. I've only ever seen a single canary version for any particular binary. Canarying is done. I haven't seen canaries running multiple versions of the same binary. Though teams will often guard new behavior behind experiment flags.
- stillworks 8y ago
  Thanks for all replys above (Kyrra, Joshuamorton and ASinclair), these have been very helpful.
joshuamorton 8y ago
>If the monorep contains let's say five different products and in a day only one of them gets a merge Releases are done by each team. Look into blaze/bazel. It allows me to say "I depend on these sources only". So a day with only one change, you might only build and release the changed artifact (in practice this never happens). Canarying: More mature teams do a lot of complex stuff. Large teams may have multiple stages of canary, multiple canaries, feature experiments, etc.
deleted 8y ago
[deleted]

kyrra 8y ago

Releases are controlled by each team. For example, my team has multiple binaries that run in production. Some of them are stable and don't receive any active code changes but are pushed every time we do a release. But for those that don't get direct code changes, they will still pickup any changes to shared libraries. Canaries are live/production traffic only. When a release is deployed, it goes to the canary instances of a job first, and it will take a small subset of traffic. This allows the job owners to see if the new binary has any adverse effects before rolling out more widely. More details about canaries can be found here[0]. Once important thing to think about with Google's source control is that it is closer to SVN as far as versioning goes. There are no (for the most part) feature branches or anything like that. Everyone is always working on HEAD. when you do a release, you will start near HEAD when cutting the release. [0] https://landing.google.com/sre/sre-book/chapters/testing-reliability/ https://landing.google.com/sre/sre-book/chapters/testing-rel...

ASinclair 8y ago

Like kyrra said. Releases are handled by independently by each team. Teams are in various states of maturity in relation to their release practices. Some are fully automated. Some involve manual QA. I've supported teams across that continuum as a SETI at Google. I'm not as well-versed in canarying though I've set it up for a team or two. I've only ever seen a single canary version for any particular binary. Canarying is done. I haven't seen canaries running multiple versions of the same binary. Though teams will often guard new behavior behind experiment flags.

stillworks 8y ago
Thanks for all replys above (Kyrra, Joshuamorton and ASinclair), these have been very helpful.

joshuamorton 8y ago

>If the monorep contains let's say five different products and in a day only one of them gets a merge Releases are done by each team. Look into blaze/bazel. It allows me to say "I depend on these sources only". So a day with only one change, you might only build and release the changed artifact (in practice this never happens). Canarying: More mature teams do a lot of complex stuff. Large teams may have multiple stages of canary, multiple canaries, feature experiments, etc.

deleted 8y ago

[deleted]