If you don't depend on clusters being 100% available all the time and design your applications to handle cluster-wide outages (which you need to do at Google sc

If you don't depend on clusters being 100% available all the time and design your applications to handle cluster-wide outages (which you need to do at Google scale anyway), then simply doing progressive rollouts across clusters is good enough. If a cluster rollout fails and knocks some workloads offline, so be it. Just revert the rollout and figure out what went wrong. You can also throw in some smaller clusters that replicate the 'main' clusters' software stack and have some workloads whose downtime does not impact production jobs (not just dev workloads, but replicas of prod workloads!). These can be the earliest in your rollout pipeline and serve as an early stage canary.