3 ms·

Sorry, I already commented but did want to recognize an important point OP made. > The examples above are contrived but it does illustrate the danger of not pr

Sorry, I already commented but did want to recognize an important point OP made. > The examples above are contrived but it does illustrate the danger of not properly architecting your AWS system and monitoring it once it’s in place (especially if you’re using T2 instances). > I often see this issue with a website that receives increases in the baseline traffic over long periods of time with occasional spikes. If the instance type is not changed and sized appropriately, this eventually leads to all the accrued CPU credits being used paired with downtime and a significant amount of head scratching. This is so true. Proper architecture and planning up front will save all kinds of headaches. Anyone serious about using EC2 needs to understand and use autoscaling groups with ELBs to manage scaling. If production availability is important, and you don't want to be bugged to fix something at 3AM, ASGs are a nice way to mitigate for potential AZ issues (you can do that with an ASG with max size of 1 even!). The biggest problem I always saw was that people would not take the time to test what happens when an ASG instance is terminated. Also You can't be ssh'ing in to manually change things and expect those changes to stick around. Proper monitoring and alerts if of course the other side to this.

sudhirj 9y ago
If I remember correctly, just setting autoscaling based on CPU usages already works fine and does the right thing with T2 instances as well - if you don't have enough credits to drive your load your CPU is going to be stuck at 100%, irrespective of credit balance or instance size. So the common sense method of scaling up at 70% / 80% CPU capacity should already work fine for any instance, no special knowledge required.
- sudosteph 9y ago
  If your load is still consistently high enough, it is possible to be caught in an ASG cycling situation with t2s that can affect your application. Ex: - Say your ASG is set to 4 max instances, min 2 - You get a sustained increase in traffic and scale up to 4 instances - Sustained 70% across 4 instances consumes more credits than are earned - As instances run out of CPU credits, they'll be pegged at 100% and stop responding correctly to TCP health checks. - Autoscaling will see that it's passed the threshold for too many failures and will stop routing traffic there, and terminate the instance and add another - Other instances fail at about the same time for the same reason (though hopefully this is staggered) and also get terminated and a new instance is created. - each time an instance gets pegged, the elb traffic gets routed to one of the remaining ones, causing the remaining instances to consume their remaining cpu credits faster Now, it's unlikely that the timings would work out so that all 4 instances are concurrently unavailable, and launching new instances is relatively quick. However it's still possible that each time an instance runs out of credits that it could be serving a request and thus send an error to the user. It also would create a lot of monitoring noise. The other thing I could see being problematic is that depending on your ASG termination policy, when you scale back in, it's not guaranteed to terminate the instance with the fewest remaining cpu credis. So you may scale down with traffic at closer to 40% cpu, and run out of credits anyhow because the instance with the most credits was terminated on scale-in. Edit: And I should add that the scale-in scenario can be especially problematic if a cooldown period is configured (default is 5 min). So if it just scaled back in to 2 instances, and those two were close to max and get pegged, it will not scale back up for 5 mintues. It will still attempt to replace them for health check failures, but once you get to a scenario where you only have 2 up, your likelihood of a full application outage goes up quite a bit (even if it's only for 2-3 minutes). So I would say that even if using an ASG, special knowledge is required for effective use of T2 instances.

sudhirj 9y ago

If I remember correctly, just setting autoscaling based on CPU usages already works fine and does the right thing with T2 instances as well - if you don't have enough credits to drive your load your CPU is going to be stuck at 100%, irrespective of credit balance or instance size. So the common sense method of scaling up at 70% / 80% CPU capacity should already work fine for any instance, no special knowledge required.

sudosteph 9y ago
If your load is still consistently high enough, it is possible to be caught in an ASG cycling situation with t2s that can affect your application. Ex: - Say your ASG is set to 4 max instances, min 2 - You get a sustained increase in traffic and scale up to 4 instances - Sustained 70% across 4 instances consumes more credits than are earned - As instances run out of CPU credits, they'll be pegged at 100% and stop responding correctly to TCP health checks. - Autoscaling will see that it's passed the threshold for too many failures and will stop routing traffic there, and terminate the instance and add another - Other instances fail at about the same time for the same reason (though hopefully this is staggered) and also get terminated and a new instance is created. - each time an instance gets pegged, the elb traffic gets routed to one of the remaining ones, causing the remaining instances to consume their remaining cpu credits faster Now, it's unlikely that the timings would work out so that all 4 instances are concurrently unavailable, and launching new instances is relatively quick. However it's still possible that each time an instance runs out of credits that it could be serving a request and thus send an error to the user. It also would create a lot of monitoring noise. The other thing I could see being problematic is that depending on your ASG termination policy, when you scale back in, it's not guaranteed to terminate the instance with the fewest remaining cpu credis. So you may scale down with traffic at closer to 40% cpu, and run out of credits anyhow because the instance with the most credits was terminated on scale-in. Edit: And I should add that the scale-in scenario can be especially problematic if a cooldown period is configured (default is 5 min). So if it just scaled back in to 2 instances, and those two were close to max and get pegged, it will not scale back up for 5 mintues. It will still attempt to replace them for health check failures, but once you get to a scenario where you only have 2 up, your likelihood of a full application outage goes up quite a bit (even if it's only for 2-3 minutes). So I would say that even if using an ASG, special knowledge is required for effective use of T2 instances.