I don't think people notice how difficult rollback is. Some people think because their app is stateless, rollback is trivial. But virtually every single piece

I don't think people notice how difficult rollback is. Some people think because their app is stateless, rollback is trivial. But virtually every single piece of technology that has an indirect effect on your app does have state, and each one moves your app away from a known good state. Everything from firewall rules and network routes, to switch firmware, to DNS, to BIOS updates, to LDAP directory changes, compute and storage configuration changes, software updates, etc all affect whether you can roll back successfully. Do you know when your cloud provider updates its software, and how it will affect your app? If they change something and don't notify you, and it causes a problem, at what point will you notice when their latest changes were? Can they even roll back their change, and how long will that take? This was an example of one tier or service failing. But it could have easily been a firewall change, a DNS change, a time sync issue, etc. If you want to roll back to the last time your app worked, all of those need a rollback method, too.