9 ms·

I hope people don't go and take this advice and just run everything on Spot as that is a mistake. It is very common for AWS to completely run out of entire cla

I hope people don't go and take this advice and just run everything on Spot as that is a mistake. It is very common for AWS to completely run out of entire classes of instance types e.g. all of R5 or all of M5. And when that happens your cluster will die. What you want to do is split your cluster into minimum two node groups e.g. Core and Task: Core: On-Demand for all of your critical and management apps e.g. monitoring. Task: Spot for your random, ephemeral jobs that aren't a big deal if it needs to be re-ran. So for a Spark cluster for example you would pin your driver to the Core nodes and let the executors run on the Task nodes.

tuananh 7y ago
author here: this is why i suggest to use mix of reserved instances and spot instances. worse case scenario, everything will be on the small number of reserved + on-demand instances. there are other strategies to avoid this as well using multiple type of instance, of different sizes, on different availability zones. because price spike is different for each combinations using bigger size instance than usual because bigger = more stable / less likely to get evicted
ryeguy 7y ago
Why not use spot fleets to solve this? You can specify multiple acceptable instance types.
- tuananh 7y ago
  because at the time when we did this (2016-2017), kops did not support spot fleet
- halbritt 7y ago
  I'm also puzzled why people wouldn't just use spot fleet with on-demand as backup.
  - tuananh 7y ago
    because at the time when we did this (2016-2017), kops did not support spot fleet. worst case scenario, you get mix of reserved + ondemand instances.
- threeseed 7y ago
  I can't speak for the other regions but in ap-southeast-2 we had a couple of incidents in 2019 where there was no Spot capacity for ANY instance types between say 64GB and 512GB of RAM. And so even our instance fleets were failing to provision capacity.
  - tuananh 7y ago
    this is why ability to fall back to on-demand instances is critical.
    - paulddraper 7y ago
      It seems weird that AWS wouldn't just do this automatically, right?
      - llarsson 7y ago
        Yes and no. Because they do not want to disclose how much capacity they have and on demand costs more, customers would be left wondering if AWS unfairly decided to run on demand when spot could have been used instead.
    - londons_explore 7y ago
      When there are no spot instances, it's common that on-demand instances can't be started either. Unless you paid for a reservation, you're SOL.
      - tuananh 7y ago
        that's for you to decide how many percent of reserved instances out of total compute power you need. it's always a trade off. in that case, pct of cost saving is lower but availability is higher.
      - maniktan 7y ago
        Fwiw, Product@SpotInst here. Curious what makes you arrive at this conclusion? At SpotInst we have customers who migrate between Spot <--> RI <--> OD everyday.
- CoolGuySteve 7y ago
  I don't think I've ever seen spotfleet not satisfy a request with all the same instance type. When the instance gets terminated, it seems to fill again with the same instance type as before (and then get killed again in busy periods). For the past few weeks I've been having this exact problem with GPU instances, where on-demand requests would kill my spotfleet instances in the middle of a job, I'd spin up again, get killed again a few minutes later. The only way to fix it without rewriting a bunch of stuff was to blacklist that instance type from my spotfleet requests. It definitely wasted a bunch of my time though which seems to be increasingly more common with newer AWS APIs.
  - maniktan 7y ago
    Fwiw, Product@SpotInst here. Might I suggest that you give us a try. We ward off against this specific scenario by calculating a Spot score based on Instance Type/AZ and deliver flexibility with a choice of different instance types.
someone13 7y ago
Shout-out for "AutoSpotting", which transparently re-launches a regular On-Demand ASG as spot instances, and will fall back to regular instances: https://github.com/AutoSpotting/AutoSpotting/ https://github.com/AutoSpotting/AutoSpotting/ Combined with the fact that you can have an ASG with multiple instance types: https://aws.amazon.com/blogs/aws/new-ec2-auto-scaling-groups-with-multiple-instance-types-purchase-options/ https://aws.amazon.com/blogs/aws/new-ec2-auto-scaling-groups... Means that you can be reasonably certain you'll never run out of capacity unless AWS runs out of every single instance type you have requested, terminates your Spot instances, and you can't launch any more On-Demand ones. (and even so, set a minimum percentage of On-Demand in AutoSpotting to ensure you maintain at least some capacity)
- ignoramous 7y ago
  ASG, per the blog-post you linked to, now supports starting both on-demand and spot instances, so what's the use of AutoSpotting?
  - kondro 7y ago
    It replaces on demand instances in-place. If there’s no spot instances, it will leave them running. If the spot instance gets killed, it will start again as on demand. It sounds a bit hinky, but it tends to leave you with the number of instances you want running without having to determine what percentage of the ASG should be on demand or spot — especially with the possibility of not being able to start new spot instances if they’ve been terminated.
  - alien_ 7y ago
    The author of AutoSpotting here, this is often being asked and I'm happy to clarify it. The mixed capacity ASGs currently run at decreased capacity when failing to launch spot instances. AutoSpotting will automatically failover to on-demand capacity when spot capacity is lost and back to spot once it can launch it again. Another useful feature is that it most often requires no configuration of older on-demand ASGs, because it can just take them over and replace their nodes with compatible spot instances. This makes it very popular for people who run legacy infrastructure that can't be tampered with for whatever reasons, as well as for large-scale rollouts on hundreds of accounts. Someone recently deployed it on infrastructure still running on EC2 Classic started in 2008 or so that wasn't touched for years. Another large company deployed it with the default opt-in configuration against hundreds of AWS accounts owned by as many teams, many with legacy instances running for years. It would normally take them years to coordinate as a mass migration but it just took them a couple of months to migrate to spot. The teams could opt-in and try it out on their application or opt-out known sensitive workloads. A few weeks later then they centrally switched the configuration to opt-out mode, converting most of their infrastructure to spot literally overnight and saving lots of money with very little configuration effort and very few disruption to the teams. If you want to learn more about it have a look at our FAQ at https://autospotting.org/faq/index.html https://autospotting.org/faq/index.html It's also the most prominent open source tool in this space. Most competition consists of closed-source, commercial (and often quite expensive) tools so if you're currently having any issues or missing functionality, anyone skilled enough can submit a fix or improvement pull request.
samrohn 7y ago
Yes. We ran into this early on. Setting your bid price(maximum price you are willing to pay for the resource you are spinning up) higher than the average spot price do not protect you against instance termination. Also even if you place your spot bid price at current on-demand price (you only get charged at the current spot market price) your instane will be terminated if there is no resource availble in the spot instance pool. For example, you have spin up an ec2 instance with 100 GB RAM with type spot instance, with bid price set to the on-demand price. If someone else spin up a 100GB as on-demand instance type- if there is no resource left in spot pool, your instance will be terminated.
croddin 7y ago
Fargate spot is out now and pretty nice, since it is just running a container of a certain size and can decide where it is run it seems like it would be less likely to run out of capacity as well.
- rumanator 7y ago
  The main problem with AWS Fargate, as with most of AWS's offering, is cost. More precisely, granularity. AWS Fargate cost infrastructure, at best, charges you per ceiling of 0.25 vCPU and 0.5GB of RAM (rounded up) per second, with a ceiling of 2GB of RAM. If your containers have some spiky workloads that require more than 2GB of RAM then the coarseness goes up and your are charged per 1GB of RAM (rounded up) per second. That means a spiky workload can lead you to spend more than a month's worth of a EC2 instance with 2GB of RAM (say, T3.small with 2vCPU and 2GB of RAM). The whole service becomes even more ridiculous if a company is already operating EC2 instances or even kubernetes clusters on EC2 spot instances (the topic of this discussion), because more often than not these workloads can be already covered by the cluster's unused capacity.
  - maniktan 7y ago
    Product@SpotInst here. Just fyi, we have customers launch Spark workloads that require 300G RAM and we are able to efficiently deliver this capacity by ensuring that the underlying infrastructure is shared with other pods that are not memory bound. We bin-pack, scale up and down quickly, while allowing you to have warmed-up spare capacity when needed to ensure that your pods/tasks are serviceable immediately.
teddyuk 7y ago
I spent months in the summer of 2018 not able to deploy azure gs4/5’s in north or west Europe - such a pain
hinkley 7y ago
are you using or have you built and dashboards to track this sort of stuff? It seems like deprioritizing certain types of work to save money might warrant some sort of forecasting info. Should I rely on something happening today or not?
maniktan 7y ago
Fwiw, I lead Product for Ocean at SpotInst. Your analysis is _spot_ on :). At SpotInst, we predict interruptions by calculating a Spot Market Score which is based on Instance Family, Type and AZ (using historic data). This allows us to spread pods/workloads across a variety of spot instance types while delivering an SLA. As an additional data point, we have a mix of stateless/stateful workloads running 24x7 on our platform. For critical/management workloads/apps, RI could make a lot of sense and we can dynamically buy/sell those on your behalf, delivering a fully-managed Spot <--> RI <--> OD solution where needed.

tuananh 7y ago

author here: this is why i suggest to use mix of reserved instances and spot instances. worse case scenario, everything will be on the small number of reserved + on-demand instances. there are other strategies to avoid this as well using multiple type of instance, of different sizes, on different availability zones. because price spike is different for each combinations using bigger size instance than usual because bigger = more stable / less likely to get evicted

ryeguy 7y ago

Why not use spot fleets to solve this? You can specify multiple acceptable instance types.

tuananh 7y ago
because at the time when we did this (2016-2017), kops did not support spot fleet
halbritt 7y ago
I'm also puzzled why people wouldn't just use spot fleet with on-demand as backup.
- tuananh 7y ago
  because at the time when we did this (2016-2017), kops did not support spot fleet. worst case scenario, you get mix of reserved + ondemand instances.
threeseed 7y ago
I can't speak for the other regions but in ap-southeast-2 we had a couple of incidents in 2019 where there was no Spot capacity for ANY instance types between say 64GB and 512GB of RAM. And so even our instance fleets were failing to provision capacity.
- tuananh 7y ago
  this is why ability to fall back to on-demand instances is critical.
  - paulddraper 7y ago
    It seems weird that AWS wouldn't just do this automatically, right?
    - llarsson 7y ago
      Yes and no. Because they do not want to disclose how much capacity they have and on demand costs more, customers would be left wondering if AWS unfairly decided to run on demand when spot could have been used instead.
  - londons_explore 7y ago
    When there are no spot instances, it's common that on-demand instances can't be started either. Unless you paid for a reservation, you're SOL.
    - tuananh 7y ago
      that's for you to decide how many percent of reserved instances out of total compute power you need. it's always a trade off. in that case, pct of cost saving is lower but availability is higher.
    - maniktan 7y ago
      Fwiw, Product@SpotInst here. Curious what makes you arrive at this conclusion? At SpotInst we have customers who migrate between Spot <--> RI <--> OD everyday.
CoolGuySteve 7y ago
I don't think I've ever seen spotfleet not satisfy a request with all the same instance type. When the instance gets terminated, it seems to fill again with the same instance type as before (and then get killed again in busy periods). For the past few weeks I've been having this exact problem with GPU instances, where on-demand requests would kill my spotfleet instances in the middle of a job, I'd spin up again, get killed again a few minutes later. The only way to fix it without rewriting a bunch of stuff was to blacklist that instance type from my spotfleet requests. It definitely wasted a bunch of my time though which seems to be increasingly more common with newer AWS APIs.
- maniktan 7y ago
  Fwiw, Product@SpotInst here. Might I suggest that you give us a try. We ward off against this specific scenario by calculating a Spot score based on Instance Type/AZ and deliver flexibility with a choice of different instance types.

someone13 7y ago

Shout-out for "AutoSpotting", which transparently re-launches a regular On-Demand ASG as spot instances, and will fall back to regular instances: https://github.com/AutoSpotting/AutoSpotting/ https://github.com/AutoSpotting/AutoSpotting/ Combined with the fact that you can have an ASG with multiple instance types: https://aws.amazon.com/blogs/aws/new-ec2-auto-scaling-groups-with-multiple-instance-types-purchase-options/ https://aws.amazon.com/blogs/aws/new-ec2-auto-scaling-groups... Means that you can be reasonably certain you'll never run out of capacity unless AWS runs out of every single instance type you have requested, terminates your Spot instances, and you can't launch any more On-Demand ones. (and even so, set a minimum percentage of On-Demand in AutoSpotting to ensure you maintain at least some capacity)

ignoramous 7y ago
ASG, per the blog-post you linked to, now supports starting both on-demand and spot instances, so what's the use of AutoSpotting?
- kondro 7y ago
  It replaces on demand instances in-place. If there’s no spot instances, it will leave them running. If the spot instance gets killed, it will start again as on demand. It sounds a bit hinky, but it tends to leave you with the number of instances you want running without having to determine what percentage of the ASG should be on demand or spot — especially with the possibility of not being able to start new spot instances if they’ve been terminated.
- alien_ 7y ago
  The author of AutoSpotting here, this is often being asked and I'm happy to clarify it. The mixed capacity ASGs currently run at decreased capacity when failing to launch spot instances. AutoSpotting will automatically failover to on-demand capacity when spot capacity is lost and back to spot once it can launch it again. Another useful feature is that it most often requires no configuration of older on-demand ASGs, because it can just take them over and replace their nodes with compatible spot instances. This makes it very popular for people who run legacy infrastructure that can't be tampered with for whatever reasons, as well as for large-scale rollouts on hundreds of accounts. Someone recently deployed it on infrastructure still running on EC2 Classic started in 2008 or so that wasn't touched for years. Another large company deployed it with the default opt-in configuration against hundreds of AWS accounts owned by as many teams, many with legacy instances running for years. It would normally take them years to coordinate as a mass migration but it just took them a couple of months to migrate to spot. The teams could opt-in and try it out on their application or opt-out known sensitive workloads. A few weeks later then they centrally switched the configuration to opt-out mode, converting most of their infrastructure to spot literally overnight and saving lots of money with very little configuration effort and very few disruption to the teams. If you want to learn more about it have a look at our FAQ at https://autospotting.org/faq/index.html https://autospotting.org/faq/index.html It's also the most prominent open source tool in this space. Most competition consists of closed-source, commercial (and often quite expensive) tools so if you're currently having any issues or missing functionality, anyone skilled enough can submit a fix or improvement pull request.

samrohn 7y ago

Yes. We ran into this early on. Setting your bid price(maximum price you are willing to pay for the resource you are spinning up) higher than the average spot price do not protect you against instance termination. Also even if you place your spot bid price at current on-demand price (you only get charged at the current spot market price) your instane will be terminated if there is no resource availble in the spot instance pool. For example, you have spin up an ec2 instance with 100 GB RAM with type spot instance, with bid price set to the on-demand price. If someone else spin up a 100GB as on-demand instance type- if there is no resource left in spot pool, your instance will be terminated.

croddin 7y ago

Fargate spot is out now and pretty nice, since it is just running a container of a certain size and can decide where it is run it seems like it would be less likely to run out of capacity as well.

rumanator 7y ago
The main problem with AWS Fargate, as with most of AWS's offering, is cost. More precisely, granularity. AWS Fargate cost infrastructure, at best, charges you per ceiling of 0.25 vCPU and 0.5GB of RAM (rounded up) per second, with a ceiling of 2GB of RAM. If your containers have some spiky workloads that require more than 2GB of RAM then the coarseness goes up and your are charged per 1GB of RAM (rounded up) per second. That means a spiky workload can lead you to spend more than a month's worth of a EC2 instance with 2GB of RAM (say, T3.small with 2vCPU and 2GB of RAM). The whole service becomes even more ridiculous if a company is already operating EC2 instances or even kubernetes clusters on EC2 spot instances (the topic of this discussion), because more often than not these workloads can be already covered by the cluster's unused capacity.
- maniktan 7y ago
  Product@SpotInst here. Just fyi, we have customers launch Spark workloads that require 300G RAM and we are able to efficiently deliver this capacity by ensuring that the underlying infrastructure is shared with other pods that are not memory bound. We bin-pack, scale up and down quickly, while allowing you to have warmed-up spare capacity when needed to ensure that your pods/tasks are serviceable immediately.

teddyuk 7y ago

I spent months in the summer of 2018 not able to deploy azure gs4/5’s in north or west Europe - such a pain

hinkley 7y ago

are you using or have you built and dashboards to track this sort of stuff? It seems like deprioritizing certain types of work to save money might warrant some sort of forecasting info. Should I rely on something happening today or not?

maniktan 7y ago

Fwiw, I lead Product for Ocean at SpotInst. Your analysis is _spot_ on :). At SpotInst, we predict interruptions by calculating a Spot Market Score which is based on Instance Family, Type and AZ (using historic data). This allows us to spread pods/workloads across a variety of spot instance types while delivering an SLA. As an additional data point, we have a mix of stateless/stateful workloads running 24x7 on our platform. For critical/management workloads/apps, RI could make a lot of sense and we can dynamically buy/sell those on your behalf, delivering a fully-managed Spot <--> RI <--> OD solution where needed.