I'm drawing conclusions based on my time at AWS, but I believe this is due to the service discovery mechanism most of AWS uses. It's a gossip protocol, with a d

I'm drawing conclusions based on my time at AWS, but I believe this is due to the service discovery mechanism most of AWS uses. It's a gossip protocol, with a daemon running on each service host. There are very valid reasons you would be manipulating the set of hosts currently being gossiped- to remove a host for maintenance for example. In this use case I think they wanted to take out an entire service, so ensuring a majority is still alive isn't necessarily a solution. There are two distinct failures here, the tool being too liberal and then the failure mode not being well understood for this index subsystem. If I remember correctly, all of our processes were organized through a weekly change management process that went through reviews of the exact commands to be run. Being oncall was a bit more liberal, you would typically execute commands on production as needed based on your experience and with others over your shoulder if you had any doubt. Interacting with the gossip protocol was a pretty common thing to do when you were triaging issues. Unrelated, I was briefly a SME on the EBS billing system and probably interacted with the poor guy whom executed this command.