3 ms·
If your load is still consistently high enough, it is possible to be caught in an ASG cycling situation with t2s that can affect your application. Ex: - Say yo
by sudosteph 9y ago
If your load is still consistently high enough, it is possible to be caught in an ASG cycling situation with t2s that can affect your application.
Ex:
- Say your ASG is set to 4 max instances, min 2
- You get a sustained increase in traffic and scale up to 4 instances
- Sustained 70% across 4 instances consumes more credits than are earned
- As instances run out of CPU credits, they'll be pegged at 100% and stop responding correctly to TCP health checks.
- Autoscaling will see that it's passed the threshold for too many failures and will stop routing traffic there, and terminate the instance and add another
- Other instances fail at about the same time for the same reason (though hopefully this is staggered) and also get terminated and a new instance is created.
- each time an instance gets pegged, the elb traffic gets routed to one of the remaining ones, causing the remaining instances to consume their remaining cpu credits faster
Now, it's unlikely that the timings would work out so that all 4 instances are concurrently unavailable, and launching new instances is relatively quick. However it's still possible that each time an instance runs out of credits that it could be serving a request and thus send an error to the user. It also would create a lot of monitoring noise.
The other thing I could see being problematic is that depending on your ASG termination policy, when you scale back in, it's not guaranteed to terminate the instance with the fewest remaining cpu credis. So you may scale down with traffic at closer to 40% cpu, and run out of credits anyhow because the instance with the most credits was terminated on scale-in.
Edit: And I should add that the scale-in scenario can be especially problematic if a cooldown period is configured (default is 5 min). So if it just scaled back in to 2 instances, and those two were close to max and get pegged, it will not scale back up for 5 mintues. It will still attempt to replace them for health check failures, but once you get to a scenario where you only have 2 up, your likelihood of a full application outage goes up quite a bit (even if it's only for 2-3 minutes).
So I would say that even if using an ASG, special knowledge is required for effective use of T2 instances.