Another thing to keep in mind is that you don't just want problems to be detectable: you want them to be diagnosable and fixable. If you detect a problem with

Another thing to keep in mind is that you don't just want problems to be detectable: you want them to be diagnosable and fixable. If you detect a problem with FooBar frontend at 1pm, the first question anyone's going to ask is, "What changed in FooBar frontend or any of its dependencies at or shortly before 1pm?" If the answer is, "Bazz backend deployed at 12:50pm" then someone with no understanding of the innards of FooBar or Bazz can roll back Bazz and there's a pretty darn good chance they'll have fixed the problem quickly. This is a scalable approach to fixing things that works well in practice. If the answer is, "Nothing just changed. Every layer of the system has been gradually changing all day" then your oncall person will need an in-depth understanding of every service involved in the system as well as creativity in guessing where to start looking. It can work, but it's going to be slower and more difficult and fewer people will have the system-specific knowledge and debugging skills do it well and quickly.