4 ms·
I was at a large company some time ago that had a script to delete any instances not tagged with a cost centre (from the testing account - they had planned to e
by danielheath 6y ago
I was at a large company some time ago that had a script to delete any instances not tagged with a cost centre (from the testing account - they had planned to eventually run it in production but that was still a ways off).
One day an AWS API had an outage causing it to return an empty array for the tags list; the script deleted over 500 in-use instances.
- rantwasp 6y agoclassic. in the professional leagues whenever you do something like this it's properly vetted and even with that you want to bake a velocity check inside of it (ie don't erase more that 5 instances at a time). I cannot believe if you're expecting to see 3 instances go away but the tool says it's gonna do 500 you're going to be: sure thing boss. go ahead.
- RandoHolmes 6y agoI built an automated VPS system that would do things like create new VPS's, install various pieces of software, and delete test VPS's, etc. I kept asking the sys admins to create a limited access account for my testing so that it flat couldn't delete existing customers VM's. I would walk into the office of the lead for that team once every few months and make the request again. Until 1 day a bug in the VPS automation accidentally deleted a customer VPS thinking it was a failed deploy. They finally got around to giving me a limited account for dev/testing work. It's scary how often this stuff falls through the cracks, even when employees KNOW it can happen.
- user5994461 6y agoTip: Set the termination protection flag on important EC2 instances to prevent accidental deletion.