2 ms·
The last time there were API outages in AWS, our autoscaling logic could not determine the number of running instances, so it felt it had too few. It kept launc
by samstave 11y ago
The last time there were API outages in AWS, our autoscaling logic could not determine the number of running instances, so it felt it had too few. It kept launching instances, and due to the API outage we couldn't manually kill the instances either...
So we wound up with over 1,000 of these machines running which then due to our fan out of their DB they needed to load into memory from other machines, our whole environment crashed until we could kill off the erroneously launched instances.
This meant an effective full reboot of our entire platform...
It's was not a fun weekend.