3 ms·
As others have mentioned: testing configuration changes is very hard. Why is it very hard? Because when you are the size of Google, there is no second version
by Eridrus 7y ago
As others have mentioned: testing configuration changes is very hard.
Why is it very hard? Because when you are the size of Google, there is no second version of prod to test things in, so the usual software engineering solution of trying the new thing in isolation and checking if it worked is unrealistic.
I think these constant failures from config changes should cause folks to re-evaluate how they do config changes though. If we can't just do a green/blue deploy of config changes like this, we probably need some other solution, whether it be the watchdog timers mentioned elsewhere in the thread, or some system that is able to show you the impact of a config change before it takes effect (probably more realistic for single services such as networking, rather than all config changes).