6 ms·

Author here. 1. Yes. This is what the article explains in the "Monitor your new pipeline before you trust it" section. This is more of a "how to get started",

Author here. 1. Yes. This is what the article explains in the "Monitor your new pipeline before you trust it" section. This is more of a "how to get started", and when you just created your canary config, you probably don't want to trust it just yet to push to production. 2. I'm not sure I understand here. What version of your application is your canary running if you're not using it to validate a new version? The same as the baseline? But then, what are you using it for?

donavanm 8y ago
For #2 canaries are commonly run against the stable/“production” deployment on some sort of periodic basis. This is used to approximate a customer experience and detect faults in underlying components, intermediate infrastructure, or changes outside of “the software” like configuration data. Its an adjunct or backstop to metric based anomoly detection. From what Ive seen. Edit: as an aside theres a very interesting area of discusssion around the spectrum of integration tests, canaries, & user experience monitoring. If you change the periods and the sources they seem to blend in to the same outcome. Ie write your integ tests to cover the ux. Run them continuosly. Associate results with underlying inputs. Suddenly theyre very much the same thing.
- not_kurt_godel 8y ago
  Yup, the key point is they let you treat complex systems as a blackbox that either does or does not successfully support the end-user functionality/experience. Ideally the components of the blackbox are sufficiently self-regulating that the end-to-end functionality is never interrupted, but complex systems by their nature exhibit emergent behavior that may not register as anomalous at a per-component level. Totally agree the nuances regarding levels of coverage between various end-to-end testing mechanisms is an interesting subject. Generally speaking I think it's preferable for canaries to essentially be continuously-run integ tests, but this is not universally applicable. Deep integ tests may be too resource-intensive to run continuously. Integ tests also aren't always ideal for monitoring behavior of long-lived resources as they typically start from a blank slate, create some new resources, test their functionality, and then clean them up. Canaries may be geared towards validating state that persists indefinitely by design.
- cavisne 8y ago
  I suspect you are confusing the terms. What Google/Netflix call a "Canary" other companies call "deploying a single host/percentage of production traffic with a new version". When other companies talk about canaries they mean regular tests against production to detect issues.
  - donavanm 8y ago
    Youre correct. I didnt catch the distinction when I first skimmed the article and parent comment. Part of it is that my active “canary” tests themselves emit relevant TSD indicative of system performance. The general concept outlined Id lump in to “approval workflows.” The gradual, intentional, deployment of mixed versions to the same workload Id call something like A/B or red/blue version deployments.
    - joshuamorton 8y ago
      At least from what I've seen red/green (I've heard blue/green) or A/B deployments represent a different thing. A blue/green deployment says you have 2 environments, each able to handle all of your traffic. So you have 2x the servers you need running, and move traffic between environments to upgrade. Its double buffering, but with binary versions. The (traffic) canarying process that Google and Netflix use, and that is described in this article is distinct from that, since you don't need a significant amount of overhead.
      - donavanm 8y ago
        Huh. Nomenclature. FWIW Ive also never heard of A/B being limited to binary or requiring full N sets of resources. I've only seen it as small subsets of traffic that is ramped up to some confidence interval. Similarly two concurrent variants is the simplest and minimal value. But Ive also seen literally thousands of concurrent variants with enough workload & consumers. Agree on overhead, as it's essentially a version management + stable routing problem you dont/shouldnt increase resource requirements.
not_kurt_godel 8y ago
> 2. I'm not sure I understand here. What version of your application is your canary running if you're not using it to validate a new version? The same as the baseline? But then, what are you using it for? We run canaries continuously against all stages of our pipelines, including prod. Once a change is deployed to a stage (initially triggered by commit to git repo), there is a time window where canary failures will trigger approval workflow failure and rollback (note that we additionally have separate integration tests that also run as part of this approval workflow). Canary failures at other times trigger steady state alarms. Thus, our canaries serve both to validate new versions and continuously monitor the existing versions.
- joshuamorton 8y ago
  > Canary failures at other times trigger steady state alarms. Thus, our canaries serve both to validate new versions and continuously monitor the existing versions. This is bad. Canaries are not alerts. Alerting should be separate from canary. If you're getting steady state alerts from your canaries, you're steady-state alerting is bad. A system that isn't undergoing change should be able to entirely disable its canary, and you should still remain confident that any changes in traffic or whatnot should be handled by other alerting tools. If not, then your canaries aren't correctly balanced, and your getting canary failures due to traffic imbalances or something that shouldn't be affecting a good canary. In other words, if you're canary is alerting in steady state, your experimental setup is invalid and you don't have a good test/control pair. If you did, the only steady state alerts you'd get would be noise. You can't have the same experimental setup control for production differences to isolate changes due to binary version bumps, while also detecting changes in production traffic independent of binary version bumps. At least one of those is broken.
  - not_kurt_godel 8y ago
    > This is bad. I would agree that it is not appropriate for all applications/systems, and in fact I have pushed against initiatives to run canaries where they're not necessary. > A system that isn't undergoing change This is an assumption which may not always be valid for sufficiently complex systems/services. Systems may be distributed amongst multiple independent microservices that are versioned and deployed independently, with the interdependencies between microservices being very difficult to capture and validate at a per-microservice level. For example, imagine microservice A receives events from microservice B which it then transforms and passes to microservice C. A deployment is made to service A which intentionally ignores/filters certain events from B, but contains a bug which causes A to filter additional events that it should not. C's low traffic alarms begin firing due to a drop in received events but from A's perspective, all is well (assuming the bug is overlooked both by its unit tests and integration tests). Likewise, B sees no issues since it is still able to successfully hand off events to A as normal. Thus, C experiences a steady state failure without having undergone any changes itself. Without an end-to-end canary, the system is stuck in a broken state until the oncall is able to figure out that the alerts generated by C have been triggered by a change in A (requiring them to subsequently manually roll A back). Alternatively, with an end-to-end canary, A's deployment is automatically rolled back and the oncall is alerted to a pipeline blockage which can be investigated while the system as a whole continues to function properly as normal. Of course, it's certainly a possibility that such a scenario is indicative of poor architecture decisions, whether due to initial oversight or organic growth in complexity over time, but the reality is that such systems commonly exist and that it's very difficult to completely avoid such undesirable emergent behavior on sufficiently large timescales. Even if your service is 'perfectly' well-architected on its own, it may have external dependencies out of its control and SLAs that require reporting when overall functionality is impacted by issues with those dependencies. A steady state failure may not be directly actionable but still require reporting to customers regardless of the underlying cause, and canaries are an excellent way to isolate and quantify such impacts independently of variable customer usage. As you state, ideally your experimental setup would completely isolate variables to the extent that alerting can be completely self-contained/decoupled without external input & measurement, and one should strive to meet this goal regardless of whether canaries are implemented or not. However, at the end of the day, your system needs to be working end-to-end at all times and canaries can be a powerful and elegant mechanism to validate this without creating a gordian knot of granular entwined inter-microservice operational dependencies. Continuous canaries should not be your first line of defense against bad deployments, but they can serve well as complementary guardrails in many circumstances.

donavanm 8y ago

For #2 canaries are commonly run against the stable/“production” deployment on some sort of periodic basis. This is used to approximate a customer experience and detect faults in underlying components, intermediate infrastructure, or changes outside of “the software” like configuration data. Its an adjunct or backstop to metric based anomoly detection. From what Ive seen. Edit: as an aside theres a very interesting area of discusssion around the spectrum of integration tests, canaries, & user experience monitoring. If you change the periods and the sources they seem to blend in to the same outcome. Ie write your integ tests to cover the ux. Run them continuosly. Associate results with underlying inputs. Suddenly theyre very much the same thing.

not_kurt_godel 8y ago
Yup, the key point is they let you treat complex systems as a blackbox that either does or does not successfully support the end-user functionality/experience. Ideally the components of the blackbox are sufficiently self-regulating that the end-to-end functionality is never interrupted, but complex systems by their nature exhibit emergent behavior that may not register as anomalous at a per-component level. Totally agree the nuances regarding levels of coverage between various end-to-end testing mechanisms is an interesting subject. Generally speaking I think it's preferable for canaries to essentially be continuously-run integ tests, but this is not universally applicable. Deep integ tests may be too resource-intensive to run continuously. Integ tests also aren't always ideal for monitoring behavior of long-lived resources as they typically start from a blank slate, create some new resources, test their functionality, and then clean them up. Canaries may be geared towards validating state that persists indefinitely by design.
cavisne 8y ago
I suspect you are confusing the terms. What Google/Netflix call a "Canary" other companies call "deploying a single host/percentage of production traffic with a new version". When other companies talk about canaries they mean regular tests against production to detect issues.
- donavanm 8y ago
  Youre correct. I didnt catch the distinction when I first skimmed the article and parent comment. Part of it is that my active “canary” tests themselves emit relevant TSD indicative of system performance. The general concept outlined Id lump in to “approval workflows.” The gradual, intentional, deployment of mixed versions to the same workload Id call something like A/B or red/blue version deployments.
  - joshuamorton 8y ago
    At least from what I've seen red/green (I've heard blue/green) or A/B deployments represent a different thing. A blue/green deployment says you have 2 environments, each able to handle all of your traffic. So you have 2x the servers you need running, and move traffic between environments to upgrade. Its double buffering, but with binary versions. The (traffic) canarying process that Google and Netflix use, and that is described in this article is distinct from that, since you don't need a significant amount of overhead.
    - donavanm 8y ago
      Huh. Nomenclature. FWIW Ive also never heard of A/B being limited to binary or requiring full N sets of resources. I've only seen it as small subsets of traffic that is ramped up to some confidence interval. Similarly two concurrent variants is the simplest and minimal value. But Ive also seen literally thousands of concurrent variants with enough workload & consumers. Agree on overhead, as it's essentially a version management + stable routing problem you dont/shouldnt increase resource requirements.

not_kurt_godel 8y ago

> 2. I'm not sure I understand here. What version of your application is your canary running if you're not using it to validate a new version? The same as the baseline? But then, what are you using it for? We run canaries continuously against all stages of our pipelines, including prod. Once a change is deployed to a stage (initially triggered by commit to git repo), there is a time window where canary failures will trigger approval workflow failure and rollback (note that we additionally have separate integration tests that also run as part of this approval workflow). Canary failures at other times trigger steady state alarms. Thus, our canaries serve both to validate new versions and continuously monitor the existing versions.

joshuamorton 8y ago
> Canary failures at other times trigger steady state alarms. Thus, our canaries serve both to validate new versions and continuously monitor the existing versions. This is bad. Canaries are not alerts. Alerting should be separate from canary. If you're getting steady state alerts from your canaries, you're steady-state alerting is bad. A system that isn't undergoing change should be able to entirely disable its canary, and you should still remain confident that any changes in traffic or whatnot should be handled by other alerting tools. If not, then your canaries aren't correctly balanced, and your getting canary failures due to traffic imbalances or something that shouldn't be affecting a good canary. In other words, if you're canary is alerting in steady state, your experimental setup is invalid and you don't have a good test/control pair. If you did, the only steady state alerts you'd get would be noise. You can't have the same experimental setup control for production differences to isolate changes due to binary version bumps, while also detecting changes in production traffic independent of binary version bumps. At least one of those is broken.
- not_kurt_godel 8y ago
  > This is bad. I would agree that it is not appropriate for all applications/systems, and in fact I have pushed against initiatives to run canaries where they're not necessary. > A system that isn't undergoing change This is an assumption which may not always be valid for sufficiently complex systems/services. Systems may be distributed amongst multiple independent microservices that are versioned and deployed independently, with the interdependencies between microservices being very difficult to capture and validate at a per-microservice level. For example, imagine microservice A receives events from microservice B which it then transforms and passes to microservice C. A deployment is made to service A which intentionally ignores/filters certain events from B, but contains a bug which causes A to filter additional events that it should not. C's low traffic alarms begin firing due to a drop in received events but from A's perspective, all is well (assuming the bug is overlooked both by its unit tests and integration tests). Likewise, B sees no issues since it is still able to successfully hand off events to A as normal. Thus, C experiences a steady state failure without having undergone any changes itself. Without an end-to-end canary, the system is stuck in a broken state until the oncall is able to figure out that the alerts generated by C have been triggered by a change in A (requiring them to subsequently manually roll A back). Alternatively, with an end-to-end canary, A's deployment is automatically rolled back and the oncall is alerted to a pipeline blockage which can be investigated while the system as a whole continues to function properly as normal. Of course, it's certainly a possibility that such a scenario is indicative of poor architecture decisions, whether due to initial oversight or organic growth in complexity over time, but the reality is that such systems commonly exist and that it's very difficult to completely avoid such undesirable emergent behavior on sufficiently large timescales. Even if your service is 'perfectly' well-architected on its own, it may have external dependencies out of its control and SLAs that require reporting when overall functionality is impacted by issues with those dependencies. A steady state failure may not be directly actionable but still require reporting to customers regardless of the underlying cause, and canaries are an excellent way to isolate and quantify such impacts independently of variable customer usage. As you state, ideally your experimental setup would completely isolate variables to the extent that alerting can be completely self-contained/decoupled without external input & measurement, and one should strive to meet this goal regardless of whether canaries are implemented or not. However, at the end of the day, your system needs to be working end-to-end at all times and canaries can be a powerful and elegant mechanism to validate this without creating a gordian knot of granular entwined inter-microservice operational dependencies. Continuous canaries should not be your first line of defense against bad deployments, but they can serve well as complementary guardrails in many circumstances.