4 ms·
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of
by seanwilson 10y ago
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.
I find making errors on production when you think you're on staging are a big one for similar errors. One of the best things I ever did on one job was to change the deployment script so that when you deployed you would get a prompt saying "Are you sure you want to deploy to production? Type 'production' to confirm". This helped stop several "oh my god, no!" situations when you repeated previous commands without thinking. For cases where you need to use SSH as well (best avoided but not always practical), it helps to use different colours, login banners and prompts for the terminals.
- fixermark 10y ago"All teams have a test server. Some teams are fortunate enough to also have a separate production server."
- seanwilson 10y ago> "All teams have a test server. Some teams are fortunate enough to also have a separate production server." That's what I meant about the SSH comment. Not every team has the automation or infrastructure that allows you to avoid SSH.
- eric_h 10y agoWe have a deploy script that does exactly this, unfortunately we've all gotten our muscle memory so trained that most of us type the deploy command, press enter, type yes and press enter before we're ever even prompted. Fortunately in most cases a quick ctrl-c can prevent any actual damage.
- seanwilson 10y agoType 'yes'? Make it ask you to type out the deploy location. I think that helps a lot.
- eric_h 10y agoI think the only way to nuke the muscle memory from this equation would be to have it make one type a random dictionary word (or solve an arithmetic problem or something - that one might help prevent drunk deploys ;). If the prompt was type "production" to confirm, I'm sure I'd just as readily train myself to jump the gun on that one.