6 ms·

On the one hand, building fault tolerant infrastructure that can, as a side effect, work painlessly on spot instances is great. On the other hand, you can purc

On the one hand, building fault tolerant infrastructure that can, as a side effect, work painlessly on spot instances is great. On the other hand, you can purchase reserved instances and get ~60% cost savings with zero engineering work. Its worth thinking long and hard about whether the cost of engineering time is worth that next 20%. There's also a lot of useful ground in between "critical state, must never be lost (like a database)" and "can handle being terminated with 2 minutes notice". A service that can be re-created if necessary but takes 10 minutes to start up is really scary if run on spot instances, but can still be pretty useful.

NikolaeVarius 7y ago
Note, AWS savings plans make this even easier https://aws.amazon.com/savingsplans/ https://aws.amazon.com/savingsplans/
- ckdarby 7y ago
  ^ Came here to say the above. Majority of the time the Saving Plan is actually what you want. Unless you have legacy app you know must be around for 3 years and have zero efforts to try to refactor.
- 40acres 7y ago
  Any idea of when this product was introduced?
  - NikolaeVarius 7y ago
    End of 2019 https://aws.amazon.com/blogs/aws-cost-management/reinvent-round-up-savings-plans-cost-categories-and-more/ https://aws.amazon.com/blogs/aws-cost-management/reinvent-ro...
    - 40acres 7y ago
      So is this blog post redundant?
      - QuinnyPig 7y ago
        No. Spot gives deeper discounts than savings plans do.
        chillydawg 7y ago
        And you can stack them.
halbritt 7y ago
I did something like this in GKE with preemptible instances which are guaranteed to go away at least once every 24 hours. I had separate node groups for stateless and stateful work loads. My clusters were roughly a 50/50 mix of each. Worked out pretty well and yielded some decent cost savings.
- tuananh 7y ago
  preemptible instances are worse than spot. we observed there were spot instances that last a year for us.
  - halbritt 7y ago
    That presumes that an instance type that last longer is "better".
    - ody4242 7y ago
      Yes, preemptible instances are expected to be expendable, I don't see this as a bad thing.
      - tuananh 7y ago
        i believe what he meant is by being shutdown more frequent, we will be forced to build a more resilent system. otherwise, we would be more complacent.
tuananh 7y ago
why not both :) we did mix reserved instances and spot instances for our production workload. worst case scenario will be reserved + on-demand
johnmarcus 7y ago
this is why i love google cloud over aws these days - i can get that cost saving with doing none of the work nor making any of the commitment. It's usually that commitment business hate, imho.
rawoke083600 7y ago
Yea were wondering the same thing... How much does this cost in engineering hours ? Its like asking someone who smokes... they always forget/lie about at least 50% of the cigarettes... And even if the engineering hours was super minimal compared to reserved instance... will this be true for other teams ?
- tuananh 7y ago
  i hear all these engineering hours cost but only because engineering cost in US is so high. we're in Southeast Asia where senior engineer only cost $1000 a month back then in 2016.
  - satanspastaroll 7y ago
    Isn't cost of living also much lower in Southeast Asia?
    - tuananh 7y ago
      i believe he was talking how it cost more in engineering cost versus the saving we can get from infrastructure by migrating. eg: if the infra cost down by 10,000 USD per month, they may say it's not worth it because they are paying more for 1 developer in US in a month.
    - mjlee 7y ago
      Sure, but AWS infrastructure costs pretty much the same no matter where you live.
jrockway 7y ago
Back when I used AWS, I went the reserved instances route. The pricing is pretty okay, and you are guaranteed to have machines at busy periods of the day. The problem with spot autoscaling is that if everyone does it, it stops working. Everyone in us-east-1 gets most of their traffic from 9am EST to 5pm EST, because everyone hosts close to their users, their users are human, and humans are diurnal. Most "batch" workloads that people have also follow the same cycle; they're working on something in the office and want results now, not tomorrow morning. So they run their batch jobs during the day. If you can figure out how to get your traffic spikes from 9pm until 5am, spot instances are going to be great -- millions of CPUs are sitting idle waiting for your novel workload. But if your customers are working 9-5 jobs like you, and you care about latency enough to host close to them... you're competing for instances with every other computer user in the region.
- lukasLansky 7y ago
  Most of batch workloads are not that latency sensitive. You can send them to the other side of the world as everyone is sleeping there right now.
  - satanspastaroll 7y ago
    A thing to keep in mind is AWS's major outobound traffic costs, at $90/TB. So taxiing the data oversea might not be affordable either
    - oblio 7y ago
      Even for inter-AWS region transfers?
      - mi100hael 7y ago
        It depends on how the data is being transferred. In general, the answer is yes because you'd be using public IPs. If you're transferring between EC2 instances in peered VPCs meaning you can still use private IPs, it's more like $20/TB. https://datapath.io/resources/blog/what-are-aws-data-transfer-costs-and-how-to-minimize-them/ https://datapath.io/resources/blog/what-are-aws-data-transfe...
ec109685 7y ago
Reserved instances and savings plans are predicated on predicting usage for 1 to 3 years in advance. With spot instances, you don’t need to do this planning, assuming you can fall back to reserved quickly enough.

NikolaeVarius 7y ago

Note, AWS savings plans make this even easier https://aws.amazon.com/savingsplans/ https://aws.amazon.com/savingsplans/

ckdarby 7y ago
^ Came here to say the above. Majority of the time the Saving Plan is actually what you want. Unless you have legacy app you know must be around for 3 years and have zero efforts to try to refactor.
40acres 7y ago
Any idea of when this product was introduced?
- NikolaeVarius 7y ago
  End of 2019 https://aws.amazon.com/blogs/aws-cost-management/reinvent-round-up-savings-plans-cost-categories-and-more/ https://aws.amazon.com/blogs/aws-cost-management/reinvent-ro...
  - 40acres 7y ago
    So is this blog post redundant?
    - QuinnyPig 7y ago
      No. Spot gives deeper discounts than savings plans do.
      - chillydawg 7y ago
        And you can stack them.

halbritt 7y ago

I did something like this in GKE with preemptible instances which are guaranteed to go away at least once every 24 hours. I had separate node groups for stateless and stateful work loads. My clusters were roughly a 50/50 mix of each. Worked out pretty well and yielded some decent cost savings.

tuananh 7y ago
preemptible instances are worse than spot. we observed there were spot instances that last a year for us.
- halbritt 7y ago
  That presumes that an instance type that last longer is "better".
  - ody4242 7y ago
    Yes, preemptible instances are expected to be expendable, I don't see this as a bad thing.
    - tuananh 7y ago
      i believe what he meant is by being shutdown more frequent, we will be forced to build a more resilent system. otherwise, we would be more complacent.

tuananh 7y ago

why not both :) we did mix reserved instances and spot instances for our production workload. worst case scenario will be reserved + on-demand

johnmarcus 7y ago

this is why i love google cloud over aws these days - i can get that cost saving with doing none of the work nor making any of the commitment. It's usually that commitment business hate, imho.

rawoke083600 7y ago

Yea were wondering the same thing... How much does this cost in engineering hours ? Its like asking someone who smokes... they always forget/lie about at least 50% of the cigarettes... And even if the engineering hours was super minimal compared to reserved instance... will this be true for other teams ?

tuananh 7y ago
i hear all these engineering hours cost but only because engineering cost in US is so high. we're in Southeast Asia where senior engineer only cost $1000 a month back then in 2016.
- satanspastaroll 7y ago
  Isn't cost of living also much lower in Southeast Asia?
  - tuananh 7y ago
    i believe he was talking how it cost more in engineering cost versus the saving we can get from infrastructure by migrating. eg: if the infra cost down by 10,000 USD per month, they may say it's not worth it because they are paying more for 1 developer in US in a month.
  - mjlee 7y ago
    Sure, but AWS infrastructure costs pretty much the same no matter where you live.

jrockway 7y ago

Back when I used AWS, I went the reserved instances route. The pricing is pretty okay, and you are guaranteed to have machines at busy periods of the day. The problem with spot autoscaling is that if everyone does it, it stops working. Everyone in us-east-1 gets most of their traffic from 9am EST to 5pm EST, because everyone hosts close to their users, their users are human, and humans are diurnal. Most "batch" workloads that people have also follow the same cycle; they're working on something in the office and want results now, not tomorrow morning. So they run their batch jobs during the day. If you can figure out how to get your traffic spikes from 9pm until 5am, spot instances are going to be great -- millions of CPUs are sitting idle waiting for your novel workload. But if your customers are working 9-5 jobs like you, and you care about latency enough to host close to them... you're competing for instances with every other computer user in the region.

lukasLansky 7y ago
Most of batch workloads are not that latency sensitive. You can send them to the other side of the world as everyone is sleeping there right now.
- satanspastaroll 7y ago
  A thing to keep in mind is AWS's major outobound traffic costs, at $90/TB. So taxiing the data oversea might not be affordable either
  - oblio 7y ago
    Even for inter-AWS region transfers?
    - mi100hael 7y ago
      It depends on how the data is being transferred. In general, the answer is yes because you'd be using public IPs. If you're transferring between EC2 instances in peered VPCs meaning you can still use private IPs, it's more like $20/TB. https://datapath.io/resources/blog/what-are-aws-data-transfer-costs-and-how-to-minimize-them/ https://datapath.io/resources/blog/what-are-aws-data-transfe...

ec109685 7y ago

Reserved instances and savings plans are predicated on predicting usage for 1 to 3 years in advance. With spot instances, you don’t need to do this planning, assuming you can fall back to reserved quickly enough.