4 ms·

The article talks about a bunch of features around canary routing that Kubernetes does not have. All of these features are misfeatures. They are ingrained becau

The article talks about a bunch of features around canary routing that Kubernetes does not have. All of these features are misfeatures. They are ingrained because it used to be easiest to deploy that way - you deployed to your Australian data center and so all your Australian users got the new version, and that was a good enough sample. Kubernetes makes deploying a canary easy, so deploy a canary everywhere. Likewise, it complains that "10% canary is hard" because you need to scale the deployments in sync. This is a problem with your monitoring layer - Your alerts and triggers and thresholds should all be based on proportion of traffic, not the other way around. Your graphs should scale to the number of canary instances. Every canary metric should either be an absolute, or a proportion of traffic. In short: Your canary should look as close to normal traffic as possible. There should be no appreciable differences between the traffic going to your canaries, and the traffic going to the rest of your production deployment. Your canary is a production deployment and any deviance needs to either be intentional or is a sign of defect.

shereadsthenews 7y ago
In practice it is easier to have same-sized experiment and control deployments than it is to normalize all the stats on a per-instance or per-request basis. I get more signal out of having identical populations than I have ever been able to achieve trying to normalize for known differences.
hinkley 7y ago
Some data collection tools allow you to add metadata to the metrics, so for aggregate alerts you can use all of the data and for things like canaries you filter based on the metadata. If the canary is generating too much traffic to a service we need to know about that, and that will be a proportional alert. But at the same time we also need to know that the overall request rate to all servers combined has dropped precipitously and may mean we are having gross routing issues. In the rare event where someone is rolling out a huge improvement, like better cache infrastructure, you disable the alert, and possibly offer a new one.

shereadsthenews 7y ago

In practice it is easier to have same-sized experiment and control deployments than it is to normalize all the stats on a per-instance or per-request basis. I get more signal out of having identical populations than I have ever been able to achieve trying to normalize for known differences.

hinkley 7y ago

Some data collection tools allow you to add metadata to the metrics, so for aggregate alerts you can use all of the data and for things like canaries you filter based on the metadata. If the canary is generating too much traffic to a service we need to know about that, and that will be a proportional alert. But at the same time we also need to know that the overall request rate to all servers combined has dropped precipitously and may mean we are having gross routing issues. In the rare event where someone is rolling out a huge improvement, like better cache infrastructure, you disable the alert, and possibly offer a new one.