4 ms·

heard a few years back, someone accidentally deleted everything across all AWS accounts at our company; had to reach out to AWS and they helped recover everythi

• xtracto 5y ago
  A similar thing happened on a previous team: One of the engineer's was experimenting with Ansible, and apparently setup some command that Terminated all instances in a VPC... he ran it for production. Suddenly I start getting alerts of unavailable instances and whatnot. Fortunately, most of the important servers had "termination protection". Another time, one of the Engineers was doing some analysis in the MongoDB production Database, and found it easier to setup a tunnel to 27017 in his computer to connect to the DB directly. While multitasking, he decided to run the test suite of some changes he was doing to one of the programs.... but that test had as a first step to delete all tables from your local Mongo database. Next morning when I come back Ops people tell me that "the system is empty". My first thought was "What do they mean the system is empty? maybe one of the dashboards failed to update or something". My surprise when I login and see what really happened...oh boy. We spent like 5 or 6 hours getting everything back. Fortunately a combination of Backups and MongoDB replication logs allowed us to recover everything.