3 ms·

You seem to have thought about it in depth and my comments maybe missing some insight A) Your argument is optimised for a certain type of failure which leaves

You seem to have thought about it in depth and my comments maybe missing some insight A) Your argument is optimised for a certain type of failure which leaves you open to the others ( issues that are detected after high ratio deployment ) B) You can monitor just the jobs impacted by the canary deployment C) the progressions you discuss seem strawman like. You can have a doubling progression starting at 1 or 5, a tripling progression D) the point about detecting at 7.5 and living at 10, what if it’s detectable at 20 and you’ve gone all out by then

joshuamorton 8y ago
A) like I say, in practice issues that are only detected after large deployment are small. If they weren't small, you'd have been able to detect them earlier. B) absolutely, but if a large subset of the jobs recently restarted you get noise and nonsense C) indeed you can do exponential continuous rollouts, but then you are getting pretty close to a 3 or 4 stage where each stage is a 5x increase in size or whatever.
dub 8y ago
Another thing to keep in mind is that you don't just want problems to be detectable: you want them to be diagnosable and fixable. If you detect a problem with FooBar frontend at 1pm, the first question anyone's going to ask is, "What changed in FooBar frontend or any of its dependencies at or shortly before 1pm?" If the answer is, "Bazz backend deployed at 12:50pm" then someone with no understanding of the innards of FooBar or Bazz can roll back Bazz and there's a pretty darn good chance they'll have fixed the problem quickly. This is a scalable approach to fixing things that works well in practice. If the answer is, "Nothing just changed. Every layer of the system has been gradually changing all day" then your oncall person will need an in-depth understanding of every service involved in the system as well as creativity in guessing where to start looking. It can work, but it's going to be slower and more difficult and fewer people will have the system-specific knowledge and debugging skills do it well and quickly.

joshuamorton 8y ago

A) like I say, in practice issues that are only detected after large deployment are small. If they weren't small, you'd have been able to detect them earlier. B) absolutely, but if a large subset of the jobs recently restarted you get noise and nonsense C) indeed you can do exponential continuous rollouts, but then you are getting pretty close to a 3 or 4 stage where each stage is a 5x increase in size or whatever.

dub 8y ago

Another thing to keep in mind is that you don't just want problems to be detectable: you want them to be diagnosable and fixable. If you detect a problem with FooBar frontend at 1pm, the first question anyone's going to ask is, "What changed in FooBar frontend or any of its dependencies at or shortly before 1pm?" If the answer is, "Bazz backend deployed at 12:50pm" then someone with no understanding of the innards of FooBar or Bazz can roll back Bazz and there's a pretty darn good chance they'll have fixed the problem quickly. This is a scalable approach to fixing things that works well in practice. If the answer is, "Nothing just changed. Every layer of the system has been gradually changing all day" then your oncall person will need an in-depth understanding of every service involved in the system as well as creativity in guessing where to start looking. It can work, but it's going to be slower and more difficult and fewer people will have the system-specific knowledge and debugging skills do it well and quickly.