4 ms·

As a high traffic customer of Google, I've been this person far too many times. [...] if it breaks, real users get affected, so canarying should be the firs

by mrj 10y ago

As a high traffic customer of Google, I've been this person far too many times. [...] if it breaks, real users get affected, so canarying should be the first step in your deployment process, as opposed to the last step in testing. It's a fine pattern and all, but not an excuse to throw stuff at prod and see what happens.

draw_down 10y ago
What's a high traffic customer of Google?
- nbclark 10y ago
  I assume they mean they visit the site frequently
- jwatte 10y ago
  Presumably someone who sells or buys lots of ads?
- mrj 10y ago
  Host a high-traffic site on Google's infrastructure. Since you can see the version number of the platform, it's obvious when they're rolling out changes. This has cause many partial outages until the change was (I assume) automatically rolled back. It's a little hard to take this advice from Google after being the victim of so many bad rollouts. Because we use a lot of services, we are far more likely to have problems. We seem to always be the canary. That's not fun.
  - jpatokal 10y ago
    But without canaries, you would have complete outages instead of partial ones.
    - mrj 10y ago
      Sure, but I'm pointing out that this strategy relies on real customers encountering an error. I caution people to not forget that is a failure for us trying to ensure reliable websites.
  - londons_explore 10y ago
    Not mentioned in this blog post is rollout related outages. It is common for a system to work fine before and after a rollout, but during a rollout clients experience errors. One might imagine downloading a big file for example, which takes an hour. If you are downloading that from http-server-v1, which is being upgraded to http-server-v2, there will be a small grace period for clients of v1 to complete their operations. That grace period in many datacenters is often 30 seconds. That means if your operation is long running, you would see a failure. The error code is usually HTTP 503, for which your client logic should retry the request with an exponential backoff. If your client doesn't retry/resume the request, now you see the service as down, when in fact the error you are seeing is by design. It will happen for every release, but also when servers come and go for maintanance, or for a bunch of other reasons. Good libraries will handle retries for you, but some don't properly, and thats a bug.
- rrdharan 10y ago
  [disclaimer: I work at Google] https://cloud.google.com/customers/ https://cloud.google.com/customers/ Spotify, SnapChat, eBay are the examples I usually give when asked that question. I'm not sure what numbers I am allowed to provide for any of them, but there's some public information available that gives you a sense of the scale involved: https://labs.spotify.com/2016/03/03/spotifys-event-delivery-the-road-to-the-cloud-part-ii/ https://labs.spotify.com/2016/03/03/spotifys-event-delivery-... https://www.itnews.com.au/news/do-not-fear-the-cloud-ebay-migrates-113bn-marketplace-454066 https://www.itnews.com.au/news/do-not-fear-the-cloud-ebay-mi...
teraflop 10y ago
I think the author of that sentence agrees with you. It's saying you should think of a canary as something to do only after testing is complete, not as a means of testing.
bognition 10y ago
>but not an excuse to throw stuff at prod and see what happens The problem with production traffic is you will never be able to simulate it perfectly. At the end of the day you are required to flip a switch and test things out in the wild.
cromwellian 10y ago
We don't just throw stuff at prod. Google has a culture of testing and code review, so code review, unit testing and integration tests are the first line of defense. Then we do QA testing, where a lot of problems not caught by tests are found, which then inform us where tests may be lacking. We also roll out to internal dogfood first, given Google's number of employees this is kind of like a pre-canary. The point of the canary is it gives you one last real world test where you can limit the damage if anything goes wrong. Without it, you'll have to assume your tests are perfect, and I doubt that's ever achieved in practice in a real environment, except maybe in avionics and space control systems.

by mrj 10y ago

draw_down 10y ago

What's a high traffic customer of Google?

nbclark 10y ago
I assume they mean they visit the site frequently
jwatte 10y ago
Presumably someone who sells or buys lots of ads?
mrj 10y ago
Host a high-traffic site on Google's infrastructure. Since you can see the version number of the platform, it's obvious when they're rolling out changes. This has cause many partial outages until the change was (I assume) automatically rolled back. It's a little hard to take this advice from Google after being the victim of so many bad rollouts. Because we use a lot of services, we are far more likely to have problems. We seem to always be the canary. That's not fun.
- jpatokal 10y ago
  But without canaries, you would have complete outages instead of partial ones.
  - mrj 10y ago
    Sure, but I'm pointing out that this strategy relies on real customers encountering an error. I caution people to not forget that is a failure for us trying to ensure reliable websites.
- londons_explore 10y ago
  Not mentioned in this blog post is rollout related outages. It is common for a system to work fine before and after a rollout, but during a rollout clients experience errors. One might imagine downloading a big file for example, which takes an hour. If you are downloading that from http-server-v1, which is being upgraded to http-server-v2, there will be a small grace period for clients of v1 to complete their operations. That grace period in many datacenters is often 30 seconds. That means if your operation is long running, you would see a failure. The error code is usually HTTP 503, for which your client logic should retry the request with an exponential backoff. If your client doesn't retry/resume the request, now you see the service as down, when in fact the error you are seeing is by design. It will happen for every release, but also when servers come and go for maintanance, or for a bunch of other reasons. Good libraries will handle retries for you, but some don't properly, and thats a bug.
rrdharan 10y ago
[disclaimer: I work at Google] https://cloud.google.com/customers/ https://cloud.google.com/customers/ Spotify, SnapChat, eBay are the examples I usually give when asked that question. I'm not sure what numbers I am allowed to provide for any of them, but there's some public information available that gives you a sense of the scale involved: https://labs.spotify.com/2016/03/03/spotifys-event-delivery-the-road-to-the-cloud-part-ii/ https://labs.spotify.com/2016/03/03/spotifys-event-delivery-... https://www.itnews.com.au/news/do-not-fear-the-cloud-ebay-migrates-113bn-marketplace-454066 https://www.itnews.com.au/news/do-not-fear-the-cloud-ebay-mi...

teraflop 10y ago

I think the author of that sentence agrees with you. It's saying you should think of a canary as something to do only after testing is complete, not as a means of testing.

bognition 10y ago

>but not an excuse to throw stuff at prod and see what happens The problem with production traffic is you will never be able to simulate it perfectly. At the end of the day you are required to flip a switch and test things out in the wild.

cromwellian 10y ago

We don't just throw stuff at prod. Google has a culture of testing and code review, so code review, unit testing and integration tests are the first line of defense. Then we do QA testing, where a lot of problems not caught by tests are found, which then inform us where tests may be lacking. We also roll out to internal dogfood first, given Google's number of employees this is kind of like a pre-canary. The point of the canary is it gives you one last real world test where you can limit the damage if anything goes wrong. Without it, you'll have to assume your tests are perfect, and I doubt that's ever achieved in practice in a real environment, except maybe in avionics and space control systems.