10 ms·

Canary analysis: Lessons learned and best practices from Google and Waze

113 points by Daviey 8y ago

btmiller 8y ago
> Spinnaker is an open-source, continuous delivery system built by Netflix and Google Politely correct me if I'm wrong...isn't Spinnaker originally a Netflix system, having nothing to do with Google? Unless perhaps the author is alluding to open source contributions by Google after the tool went open source?
- svachalek 8y ago
  It is originally a Netflix creation but Google has been a contributor since it went open source. In particular, the canary analysis features were co-developed with Google. (I was a contributor on the Netflix side.)
  - techcofounder 8y ago
    That is correct. Google played a big role in open-sourcing Spinnaker alongside Netflix back in Nov 2015.
- jedberg 8y ago
  It's just a question of where you draw the line. Google made a lot of contributions after it went open source, particularly with the canary analysis. But I mean, I evaluated the initial design docs, so did I build it too? :)
  - ajordens 8y ago
    Not that it really matters but Google had been an active participant in the project for at least a year prior to the open sourcing in 2015. Meh.
- mrtrustor 8y ago
  Author here. As mentioned in the other comments, Google is a very active contributor to Spinnaker. Today, Netflix and Google are the 2 companies contributing the most to the project, hence this sentence.
not_kurt_godel 8y ago
There are 2 gaps here, at least as they relate to my understanding of what "canaries" are (based on experience): 1. Requiring manual validation of canary results is antithetical to CI/CD principles. If you can't trust your canaries enough to automatically promote or block based on results, doing manual validation is just a band-aid that will have to be ripped off painfully eventually. 2. Canaries should run continuously against all deployment stages, not just as a one-time approval process in your pipeline. Good canaries give visibility into baseline metrics at all times, not just when validating a new application version. Overall I would say this guide aligns more with what I would term as a non-CI/CD load-test approval workflow rather than a "canary".
- mrtrustor 8y ago
  Author here. 1. Yes. This is what the article explains in the "Monitor your new pipeline before you trust it" section. This is more of a "how to get started", and when you just created your canary config, you probably don't want to trust it just yet to push to production. 2. I'm not sure I understand here. What version of your application is your canary running if you're not using it to validate a new version? The same as the baseline? But then, what are you using it for?
  - donavanm 8y ago
    For #2 canaries are commonly run against the stable/“production” deployment on some sort of periodic basis. This is used to approximate a customer experience and detect faults in underlying components, intermediate infrastructure, or changes outside of “the software” like configuration data. Its an adjunct or backstop to metric based anomoly detection. From what Ive seen. Edit: as an aside theres a very interesting area of discusssion around the spectrum of integration tests, canaries, & user experience monitoring. If you change the periods and the sources they seem to blend in to the same outcome. Ie write your integ tests to cover the ux. Run them continuosly. Associate results with underlying inputs. Suddenly theyre very much the same thing.
    - not_kurt_godel 8y ago
joatmon-snoo 8y ago
For clarification, "canary" in this article refers to rolling out a new release to some small subset of production traffic, a la canary release (eg https://martinfowler.com/bliki/CanaryRelease.html https://martinfowler.com/bliki/CanaryRelease.html). At least a few other people in the comments are saying that in their experience, "canaries" are black-box monitoring programs that simulate critical user journeys. This is not what this article is discussing.

113 points by Daviey 8y ago

> Spinnaker is an open-source, continuous delivery system built by Netflix and Google Politely correct me if I'm wrong...isn't Spinnaker originally a Netflix system, having nothing to do with Google? Unless perhaps the author is alluding to open source contributions by Google after the tool went open source?

svachalek 8y ago
It is originally a Netflix creation but Google has been a contributor since it went open source. In particular, the canary analysis features were co-developed with Google. (I was a contributor on the Netflix side.)
- techcofounder 8y ago
  That is correct. Google played a big role in open-sourcing Spinnaker alongside Netflix back in Nov 2015.
jedberg 8y ago
It's just a question of where you draw the line. Google made a lot of contributions after it went open source, particularly with the canary analysis. But I mean, I evaluated the initial design docs, so did I build it too? :)
- ajordens 8y ago
  Not that it really matters but Google had been an active participant in the project for at least a year prior to the open sourcing in 2015. Meh.
mrtrustor 8y ago
Author here. As mentioned in the other comments, Google is a very active contributor to Spinnaker. Today, Netflix and Google are the 2 companies contributing the most to the project, hence this sentence.

not_kurt_godel 8y ago

There are 2 gaps here, at least as they relate to my understanding of what "canaries" are (based on experience): 1. Requiring manual validation of canary results is antithetical to CI/CD principles. If you can't trust your canaries enough to automatically promote or block based on results, doing manual validation is just a band-aid that will have to be ripped off painfully eventually. 2. Canaries should run continuously against all deployment stages, not just as a one-time approval process in your pipeline. Good canaries give visibility into baseline metrics at all times, not just when validating a new application version. Overall I would say this guide aligns more with what I would term as a non-CI/CD load-test approval workflow rather than a "canary".

mrtrustor 8y ago
Author here. 1. Yes. This is what the article explains in the "Monitor your new pipeline before you trust it" section. This is more of a "how to get started", and when you just created your canary config, you probably don't want to trust it just yet to push to production. 2. I'm not sure I understand here. What version of your application is your canary running if you're not using it to validate a new version? The same as the baseline? But then, what are you using it for?
- donavanm 8y ago
  For #2 canaries are commonly run against the stable/“production” deployment on some sort of periodic basis. This is used to approximate a customer experience and detect faults in underlying components, intermediate infrastructure, or changes outside of “the software” like configuration data. Its an adjunct or backstop to metric based anomoly detection. From what Ive seen. Edit: as an aside theres a very interesting area of discusssion around the spectrum of integration tests, canaries, & user experience monitoring. If you change the periods and the sources they seem to blend in to the same outcome. Ie write your integ tests to cover the ux. Run them continuosly. Associate results with underlying inputs. Suddenly theyre very much the same thing.
  - not_kurt_godel 8y ago

joatmon-snoo 8y ago

For clarification, "canary" in this article refers to rolling out a new release to some small subset of production traffic, a la canary release (eg https://martinfowler.com/bliki/CanaryRelease.html https://martinfowler.com/bliki/CanaryRelease.html). At least a few other people in the comments are saying that in their experience, "canaries" are black-box monitoring programs that simulate critical user journeys. This is not what this article is discussing.