3 ms·
When I was at Google, my team and other SRE teams around us adopted a similar approach to sensitive operations - deploys and data migrations and the like. We'd
by GauntletWizard 8y ago
When I was at Google, my team and other SRE teams around us adopted a similar approach to sensitive operations - deploys and data migrations and the like. We'd have one of us on the keyboard operating, and another looking over our shoulder. The operator would type a command and they verbally confirm the action they were going to take. Their partner would look it over and give verbal acknowledgement. We certainly still had mistakes, but I found that environment very helpful - and it was an almost necessary part of zero-blame postmortems, because every action was not one person, but equally shared between two.
- walshemj 8y agoI used to use check lists for this sort of critical tasks obvisly much smaller systems back then a 16 machine cluster was a big deal