> Yes, multiple times. It just didn't help with each and every outage, which is a different thing. So then maybe the idea that a prompt global rollout is a bad

> Yes, multiple times. It just didn't help with each and every outage, which is a different thing. So then maybe the idea that a prompt global rollout is a bad one even if you have good tests and coverage. > For the ones not preventable by CI (e.g. a disk was borked, power was down, but no integration / programming error etc a CI process could possibly spot) the question is moot. I'm here to tell you, as someone who has been at this for a long time and someone who is actually doing SRE at Google at the outer layers: No. It matters. We do canaries because they let you find problems no reasonable person cold catch with tests. Tests themselves are only cone component of a chain of reliability best practices, starting at coding practice and expressive type systems with static checks and moving outwards to reploying the code carefully and with intentionality. > Besides the concern about outages is totally unrelated to "speed of deployment". This is obviously false. Slower rollouts of new software and canaries are a proven way to reduce the impact of a bad push. Once your system gets modestly complicated, it can be very difficult to spot a bad push with naive unit testing which (let's be real here) often just mocks out dependencies anyways. > Increased speed of deployment helps testing and iteration (obviously faster deployment speeds also means faster deployment to testing and staging environments, not just to the final server). Right, but very fast deployment to client machines is not hard. The CD model is just inherently dangerous unless it has automated safeguards against the complexities of even a modestly large production app. Personally, if your staging environment is CD I think that's great (up until your org gets big enough that a broken build pisses off other devs). But to prod? I will resign before I let you do it. Everyone who does software at scale agrees it's not a good plan to do CD on prod. There are whole books about why you shouldn't do it.