4 ms·
Spit balling here... For progressive rollouts, what if config changes where pulled instead of pushed? Each system would be responsible for itself updating, ve
by specialist 11y ago
Spit balling here...
For progressive rollouts, what if config changes where pulled instead of pushed?
Each system would be responsible for itself updating, verifying (canary, smoketest, make sure other systems successfully updated, etc), bouncing, and then rolling back as needed.
- nostrademons 11y agoA bunch of that's in-place already, eg. all Google servers have health checks that run basic smoke tests on a configuration, and if a large number of replicas become unhealthy after a config change, the rollout process automatically aborts and rolls back to the last known good conversion. The problem here was that there was a bug in the health check that masked the problem by assigning the last-good configuration, and then there was a bug in that code that had saved "nothing" as the last-good configuration. So rather than failing and having the error caught at the top level, it failed and buggy failure-recovery code made the problem worse.