5 ms·

That seems like a lot of resources to delete... It’s “possible” but I’m not sure they I find it plausible. Is there some magic terraform or cloud formation th

That seems like a lot of resources to delete... It’s “possible” but I’m not sure they I find it plausible. Is there some magic terraform or cloud formation that just destroys everything? And a Cisco engineer with devops like experience wouldn’t at least sniff around before running it? Doesn’t seem realistic. Cisco should wear this too though, this is shockingly negligent. The only reason I can think of suggests a lot more problems and likely noncompliance with regulations and standards I’m sure they claim to comply with.

danielheath 6y ago
I was at a large company some time ago that had a script to delete any instances not tagged with a cost centre (from the testing account - they had planned to eventually run it in production but that was still a ways off). One day an AWS API had an outage causing it to return an empty array for the tags list; the script deleted over 500 in-use instances.
- rantwasp 6y ago
  classic. in the professional leagues whenever you do something like this it's properly vetted and even with that you want to bake a velocity check inside of it (ie don't erase more that 5 instances at a time). I cannot believe if you're expecting to see 3 instances go away but the tool says it's gonna do 500 you're going to be: sure thing boss. go ahead.
- RandoHolmes 6y ago
  I built an automated VPS system that would do things like create new VPS's, install various pieces of software, and delete test VPS's, etc. I kept asking the sys admins to create a limited access account for my testing so that it flat couldn't delete existing customers VM's. I would walk into the office of the lead for that team once every few months and make the request again. Until 1 day a bug in the VPS automation accidentally deleted a customer VPS thinking it was a failed deploy. They finally got around to giving me a limited account for dev/testing work. It's scary how often this stuff falls through the cracks, even when employees KNOW it can happen.
- user5994461 6y ago
  Tip: Set the termination protection flag on important EC2 instances to prevent accidental deletion.
rantwasp 6y ago
yes. there are tools out there that can wipe away everything: https://github.com/rebuy-de/aws-nuke https://github.com/rebuy-de/aws-nuke My money is that this guy was just negligent an did not remove his old credentials + used whatever thing he was using before without doublechecking. It's extremely plausible he had this setup for a test account and ran it with the wrong credentials. On Cisco side this bad all around. Revoke the credentials. Audit the credentials periodically. Don't allow direct production access even for engineers that work there unless it's a live production issue or a deployment that's going on (and even then you allow them access to the tooling that does the deployment not to run whatever they want with : permissions)
tgsovlerkhgsel 6y ago
He "deployed code" that caused the deletions. I can definitely come up with many scenarios where something automatically removes VMs. Edit: And https://news.ycombinator.com/item?id=24320495 https://news.ycombinator.com/item?id=24320495 provides a terraform command that is claimed to be potentially very destructive.
viraptor 6y ago
Sure. Essentially "for X in list-stacks; delete-stack $X". In this case it's only VMs (instances, I guess?) so it could be any cleanup script like "terminate not tagged", "terminate anything running over N days", or even just "delete-asg" which happened to have 456 instances in it.
goatinaboat 6y ago
Is there some magic terraform or cloud formation that just destroys everything? Terraform has a nasty habit of instead of changing what you have into what you want, destroying what you have and then building what you want from scratch. It needs to be used very, very carefully.
weego 6y ago
A reused cloudformation template accidentally run on the wrong account could do an awful lot if it's set to tear down existing infra first so the env is clean

danielheath 6y ago

I was at a large company some time ago that had a script to delete any instances not tagged with a cost centre (from the testing account - they had planned to eventually run it in production but that was still a ways off). One day an AWS API had an outage causing it to return an empty array for the tags list; the script deleted over 500 in-use instances.

rantwasp 6y ago
classic. in the professional leagues whenever you do something like this it's properly vetted and even with that you want to bake a velocity check inside of it (ie don't erase more that 5 instances at a time). I cannot believe if you're expecting to see 3 instances go away but the tool says it's gonna do 500 you're going to be: sure thing boss. go ahead.
RandoHolmes 6y ago
I built an automated VPS system that would do things like create new VPS's, install various pieces of software, and delete test VPS's, etc. I kept asking the sys admins to create a limited access account for my testing so that it flat couldn't delete existing customers VM's. I would walk into the office of the lead for that team once every few months and make the request again. Until 1 day a bug in the VPS automation accidentally deleted a customer VPS thinking it was a failed deploy. They finally got around to giving me a limited account for dev/testing work. It's scary how often this stuff falls through the cracks, even when employees KNOW it can happen.
user5994461 6y ago
Tip: Set the termination protection flag on important EC2 instances to prevent accidental deletion.

rantwasp 6y ago

yes. there are tools out there that can wipe away everything: https://github.com/rebuy-de/aws-nuke https://github.com/rebuy-de/aws-nuke My money is that this guy was just negligent an did not remove his old credentials + used whatever thing he was using before without doublechecking. It's extremely plausible he had this setup for a test account and ran it with the wrong credentials. On Cisco side this bad all around. Revoke the credentials. Audit the credentials periodically. Don't allow direct production access even for engineers that work there unless it's a live production issue or a deployment that's going on (and even then you allow them access to the tooling that does the deployment not to run whatever they want with : permissions)

tgsovlerkhgsel 6y ago

He "deployed code" that caused the deletions. I can definitely come up with many scenarios where something automatically removes VMs. Edit: And https://news.ycombinator.com/item?id=24320495 https://news.ycombinator.com/item?id=24320495 provides a terraform command that is claimed to be potentially very destructive.

viraptor 6y ago

Sure. Essentially "for X in list-stacks; delete-stack $X". In this case it's only VMs (instances, I guess?) so it could be any cleanup script like "terminate not tagged", "terminate anything running over N days", or even just "delete-asg" which happened to have 456 instances in it.

goatinaboat 6y ago

Is there some magic terraform or cloud formation that just destroys everything? Terraform has a nasty habit of instead of changing what you have into what you want, destroying what you have and then building what you want from scratch. It needs to be used very, very carefully.

weego 6y ago

A reused cloudformation template accidentally run on the wrong account could do an awful lot if it's set to tear down existing infra first so the env is clean