4 ms·

The way I've been happiest using EC2 Auto Scaling was to have a single cron-job continuously calculating how many instances I should be running, and it sets the

The way I've been happiest using EC2 Auto Scaling was to have a single cron-job continuously calculating how many instances I should be running, and it sets the desired capacity manually with the Auto Scaling API[1]. This may seem to defeat the purpose of Auto Scaling, but it's actually much more convenient than spinning up/down EC2 instances with the EC2 API. You get to precisely control how to scale, and won't be at the mercy of the Auto Scaling heuristics. [1] https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-manual-scaling.html https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-man...

d3sandoval 8y ago
We did the same thing! Glad to hear we weren't too crazy in doing so
teej 8y ago
I don’t think it’s crazy to think you are more capable of predicting the unique demand curves of your business better than a heuristic designed to be good enough for the median Amazon customer.
samstave 8y ago
So we had this cause a spectacular outage a few years ago. We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down. So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually launching instances. The increased scale of instances pummeled the control plane with thousands of instances all trying to come online and pull down their needed data to get operational -- which them killed our DBs, pipeline etc... We had to reboot our entire production environment at peak service time...
- robrtsql 8y ago
  I don't understand--how were you launching instances if the API(s) was/were down? Your system was unable to determine that there were instances running, but it was able to send RunInstances requests to EC2?
  - tudelo 8y ago
    Queue them up?
  - samstave 8y ago
    Correct. Couldn't query, but could initiate.
- aequitas 8y ago
  AWS has limits on the amount of resources you can have in a VPC. You can request to increase these via a process out of bounds of the API. This mechanism is there exactly for these kind of things (and malicious API calls should you get hacked). Maybe someone at your company was thinking to big? Normally these are around 10/50 for each EC2 type.
  - samstave 8y ago
    you are aware that if you have a close enough relationship with AWS you can request and set your own limits? Limits are malleable based on your use case. Speak with your rep. You might even not know how limits came to be... I am. --- I have had a time when git suffered a flaw, and a junior dev also suffered a flaw in checking in secrets.... thousands of instances across the globe were launched... for bitcoin mining... $700,000 in a few hours... We all learned a bit that day.
- nicoburns 8y ago
  What was your resolution to this issue? Did you fix your service to account for the API being down, or did you switch to an entirely different approach?
  - samstave 8y ago
    I can't recall the exact implementation detail, but We then logged the number of running instances in a file, and read the last qty of instances and the delta from when launched - and made the system not get over aggressive if it couldnt read the current set. We also added smart loading across AZs due to spot instances getting whacked when our fleet was outbid and AWS took them back. As well as other monitoring methods to be sure we werent caught with a smart system doing dumb things.
- DVassallo 8y ago
  That's not the right way to do it. You shouldn't monitor how many instances you're running. You just need to determine how many instances you should be running based on your scaling driver (cpu, # of users, database connections, etc). Then you call the Auto Scaling SetDesiredCapacity API with the number, and it is idempotent[1]. If the AWS API is down, your fleet size just won't change. [1] https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_SetDesiredCapacity.html https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API...
  - samstave 8y ago
    Yeah, like I said - this was a few years ago, and the system wasnt designed to be able to scale using ASGs at the time. (Fleet didnt yet exist, amd a bunch of other reasons) - but scaling was based on users and load complexity for the data we were handling -- this wasnt a web service.
  - icelancer 8y ago
    > That's not the right way to do it. The poster is aware of this, which is why they talked specifically what they did wrong.
    - zepolen 8y ago
      While the poster was aware of it, he did not provide a solution whereas DVassallo provided a valuable step by step on how to do it properly. Which may help others in the future. Think long and hard why you felt it necessary to make your comment and what value it actually provided.
      - samstave 8y ago
        If this is still a pitfall for users of AWS ~5 years in.. then its not a fault of my communication... You know what I think its a fault of: Lack of a canonical DevOps "university" stemmed from SV startups. DevOps at this point should not just be a degree -- it should be a freaking field of study and classes offered by YC.... Look at their pedigree of companies at scale. We should all make an effort to give back in this regard...
- AmericanChopper 8y ago
  Another thing to keep in mind is that AWS local capacity can run incredibly close to the wire at times. You might be surprised if you knew how much capacity for your instance type was actually available under the hood. I’ve personally seen insufficient capacity errors.
  - samstave 8y ago
    I've always kept a mental capacity tracker in my head... Would love to hear what others are also tracking. Goes to show what their business numbers are....
cdoxsey 8y ago
This is how the cluster-autoscaler works in kubernetes. It sets the desired capacity based on the number of pods needing to be scheduled. Coupled with a horizontal pod autoscaler (which sets the replica count based on a metric) you get the best of both worlds.
- chrissnell 8y ago
  We do this for all of our Kube clusters here. We have a nifty use of this in our CI cluster, where Job resources are scheduled by a service that monitors our build queue. As builds are initiated by git pushes, single-use Jobs (pods) are created and when there are not enough free capacity to scale them, the build cluster scales up. On the weekends when everyone is gone, it scales back to near-nothing. This is a huuuuge money-saver for us because we use beefy c5 instances for our builds. It also saves lots of time because devs are no longer waiting 45+ minutes for their build to start.
deleted 8y ago
[deleted]

d3sandoval 8y ago

We did the same thing! Glad to hear we weren't too crazy in doing so

teej 8y ago

I don’t think it’s crazy to think you are more capable of predicting the unique demand curves of your business better than a heuristic designed to be good enough for the median Amazon customer.

samstave 8y ago

So we had this cause a spectacular outage a few years ago. We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down. So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually launching instances. The increased scale of instances pummeled the control plane with thousands of instances all trying to come online and pull down their needed data to get operational -- which them killed our DBs, pipeline etc... We had to reboot our entire production environment at peak service time...

robrtsql 8y ago
I don't understand--how were you launching instances if the API(s) was/were down? Your system was unable to determine that there were instances running, but it was able to send RunInstances requests to EC2?
- tudelo 8y ago
  Queue them up?
- samstave 8y ago
  Correct. Couldn't query, but could initiate.
aequitas 8y ago
AWS has limits on the amount of resources you can have in a VPC. You can request to increase these via a process out of bounds of the API. This mechanism is there exactly for these kind of things (and malicious API calls should you get hacked). Maybe someone at your company was thinking to big? Normally these are around 10/50 for each EC2 type.
- samstave 8y ago
  you are aware that if you have a close enough relationship with AWS you can request and set your own limits? Limits are malleable based on your use case. Speak with your rep. You might even not know how limits came to be... I am. --- I have had a time when git suffered a flaw, and a junior dev also suffered a flaw in checking in secrets.... thousands of instances across the globe were launched... for bitcoin mining... $700,000 in a few hours... We all learned a bit that day.
nicoburns 8y ago
What was your resolution to this issue? Did you fix your service to account for the API being down, or did you switch to an entirely different approach?
- samstave 8y ago
  I can't recall the exact implementation detail, but We then logged the number of running instances in a file, and read the last qty of instances and the delta from when launched - and made the system not get over aggressive if it couldnt read the current set. We also added smart loading across AZs due to spot instances getting whacked when our fleet was outbid and AWS took them back. As well as other monitoring methods to be sure we werent caught with a smart system doing dumb things.
DVassallo 8y ago
That's not the right way to do it. You shouldn't monitor how many instances you're running. You just need to determine how many instances you should be running based on your scaling driver (cpu, # of users, database connections, etc). Then you call the Auto Scaling SetDesiredCapacity API with the number, and it is idempotent[1]. If the AWS API is down, your fleet size just won't change. [1] https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_SetDesiredCapacity.html https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API...
- samstave 8y ago
  Yeah, like I said - this was a few years ago, and the system wasnt designed to be able to scale using ASGs at the time. (Fleet didnt yet exist, amd a bunch of other reasons) - but scaling was based on users and load complexity for the data we were handling -- this wasnt a web service.
- icelancer 8y ago
  > That's not the right way to do it. The poster is aware of this, which is why they talked specifically what they did wrong.
  - zepolen 8y ago
    While the poster was aware of it, he did not provide a solution whereas DVassallo provided a valuable step by step on how to do it properly. Which may help others in the future. Think long and hard why you felt it necessary to make your comment and what value it actually provided.
    - samstave 8y ago
      If this is still a pitfall for users of AWS ~5 years in.. then its not a fault of my communication... You know what I think its a fault of: Lack of a canonical DevOps "university" stemmed from SV startups. DevOps at this point should not just be a degree -- it should be a freaking field of study and classes offered by YC.... Look at their pedigree of companies at scale. We should all make an effort to give back in this regard...
AmericanChopper 8y ago
Another thing to keep in mind is that AWS local capacity can run incredibly close to the wire at times. You might be surprised if you knew how much capacity for your instance type was actually available under the hood. I’ve personally seen insufficient capacity errors.
- samstave 8y ago
  I've always kept a mental capacity tracker in my head... Would love to hear what others are also tracking. Goes to show what their business numbers are....

cdoxsey 8y ago

This is how the cluster-autoscaler works in kubernetes. It sets the desired capacity based on the number of pods needing to be scheduled. Coupled with a horizontal pod autoscaler (which sets the replica count based on a metric) you get the best of both worlds.

chrissnell 8y ago
We do this for all of our Kube clusters here. We have a nifty use of this in our CI cluster, where Job resources are scheduled by a service that monitors our build queue. As builds are initiated by git pushes, single-use Jobs (pods) are created and when there are not enough free capacity to scale them, the build cluster scales up. On the weekends when everyone is gone, it scales back to near-nothing. This is a huuuuge money-saver for us because we use beefy c5 instances for our builds. It also saves lots of time because devs are no longer waiting 45+ minutes for their build to start.

deleted 8y ago

[deleted]