5 ms·

Why not use spot fleets to solve this? You can specify multiple acceptable instance types.

by ryeguy 7y ago

tuananh 7y ago
because at the time when we did this (2016-2017), kops did not support spot fleet
halbritt 7y ago
I'm also puzzled why people wouldn't just use spot fleet with on-demand as backup.
- tuananh 7y ago
  because at the time when we did this (2016-2017), kops did not support spot fleet. worst case scenario, you get mix of reserved + ondemand instances.
threeseed 7y ago
I can't speak for the other regions but in ap-southeast-2 we had a couple of incidents in 2019 where there was no Spot capacity for ANY instance types between say 64GB and 512GB of RAM. And so even our instance fleets were failing to provision capacity.
- tuananh 7y ago
  this is why ability to fall back to on-demand instances is critical.
  - paulddraper 7y ago
    It seems weird that AWS wouldn't just do this automatically, right?
    - llarsson 7y ago
      Yes and no. Because they do not want to disclose how much capacity they have and on demand costs more, customers would be left wondering if AWS unfairly decided to run on demand when spot could have been used instead.
  - londons_explore 7y ago
    When there are no spot instances, it's common that on-demand instances can't be started either. Unless you paid for a reservation, you're SOL.
    - tuananh 7y ago
      that's for you to decide how many percent of reserved instances out of total compute power you need. it's always a trade off. in that case, pct of cost saving is lower but availability is higher.
    - maniktan 7y ago
      Fwiw, Product@SpotInst here. Curious what makes you arrive at this conclusion? At SpotInst we have customers who migrate between Spot <--> RI <--> OD everyday.
- ramraj07 7y ago
  Requiring a minimum of 64GB RAM seems quite unique computationally, so if it's mission critical that you have at least one instance of this type all the time up, then maybe you could reserve that one instance fully while depending on spots for scaling?
CoolGuySteve 7y ago
I don't think I've ever seen spotfleet not satisfy a request with all the same instance type. When the instance gets terminated, it seems to fill again with the same instance type as before (and then get killed again in busy periods). For the past few weeks I've been having this exact problem with GPU instances, where on-demand requests would kill my spotfleet instances in the middle of a job, I'd spin up again, get killed again a few minutes later. The only way to fix it without rewriting a bunch of stuff was to blacklist that instance type from my spotfleet requests. It definitely wasted a bunch of my time though which seems to be increasingly more common with newer AWS APIs.
- maniktan 7y ago
  Fwiw, Product@SpotInst here. Might I suggest that you give us a try. We ward off against this specific scenario by calculating a Spot score based on Instance Type/AZ and deliver flexibility with a choice of different instance types.

Why not use spot fleets to solve this? You can specify multiple acceptable instance types.

by ryeguy 7y ago

Why not use spot fleets to solve this? You can specify multiple acceptable instance types.

tuananh 7y ago

because at the time when we did this (2016-2017), kops did not support spot fleet

halbritt 7y ago

I'm also puzzled why people wouldn't just use spot fleet with on-demand as backup.

tuananh 7y ago
because at the time when we did this (2016-2017), kops did not support spot fleet. worst case scenario, you get mix of reserved + ondemand instances.

threeseed 7y ago

I can't speak for the other regions but in ap-southeast-2 we had a couple of incidents in 2019 where there was no Spot capacity for ANY instance types between say 64GB and 512GB of RAM. And so even our instance fleets were failing to provision capacity.

tuananh 7y ago
this is why ability to fall back to on-demand instances is critical.
- paulddraper 7y ago
  It seems weird that AWS wouldn't just do this automatically, right?
  - llarsson 7y ago
    Yes and no. Because they do not want to disclose how much capacity they have and on demand costs more, customers would be left wondering if AWS unfairly decided to run on demand when spot could have been used instead.
- londons_explore 7y ago
  When there are no spot instances, it's common that on-demand instances can't be started either. Unless you paid for a reservation, you're SOL.
  - tuananh 7y ago
    that's for you to decide how many percent of reserved instances out of total compute power you need. it's always a trade off. in that case, pct of cost saving is lower but availability is higher.
  - maniktan 7y ago
    Fwiw, Product@SpotInst here. Curious what makes you arrive at this conclusion? At SpotInst we have customers who migrate between Spot <--> RI <--> OD everyday.
ramraj07 7y ago
Requiring a minimum of 64GB RAM seems quite unique computationally, so if it's mission critical that you have at least one instance of this type all the time up, then maybe you could reserve that one instance fully while depending on spots for scaling?

CoolGuySteve 7y ago

I don't think I've ever seen spotfleet not satisfy a request with all the same instance type. When the instance gets terminated, it seems to fill again with the same instance type as before (and then get killed again in busy periods). For the past few weeks I've been having this exact problem with GPU instances, where on-demand requests would kill my spotfleet instances in the middle of a job, I'd spin up again, get killed again a few minutes later. The only way to fix it without rewriting a bunch of stuff was to blacklist that instance type from my spotfleet requests. It definitely wasted a bunch of my time though which seems to be increasingly more common with newer AWS APIs.

maniktan 7y ago
Fwiw, Product@SpotInst here. Might I suggest that you give us a try. We ward off against this specific scenario by calculating a Spot score based on Instance Type/AZ and deliver flexibility with a choice of different instance types.