3 ms·
> leveraging elastic compute costs and kubernetes "self-healing" The indirect effect of building on a system like this is that the recovery mechanisms get test
by gopalv 7y ago
> leveraging elastic compute costs and kubernetes "self-healing"
The indirect effect of building on a system like this is that the recovery mechanisms get tested on a regular basis instead of just on the odd day when things fail.
Spot instances are like a natural chaosmonkey mode, with money being saved and forcing you to build failure tolerance, retries & circuit breakers early in dev.
- halbritt 7y agoGCP has a similar instance type called "preemptible", they're not quite as cheap as spot, but they don't "dry up" and they're guaranteed to go down every 24 hours. This precludes one from becoming complacent with spot instances that rarely go away.
- tuananh 7y agoYou're right. spot instances are a lot more stable than preemptible. we've seen spot instances that last a year for us.
- halbritt 7y agoThis is the second response of this sort that I'm replying to. "A lot more stable" isn't really a desirable characteristic of ephemeral compute capacity. In my experience, the less frequently the instances went away, the more complacent the operators became. Preemptible instance are stable in the sense that you know they're going away within 24 hours and must be prepared for that.
- tuananh 7y agotrue that. spot used to be that way and the price was very sensitive. but AWS tweaked it so that it's more stable. to the point, after a year or 2 of running spot instances, we don't feel the difference of spot and ondemand that much. we got complacent.
- jedifans 7y agohttps://cloud.google.com/kubernetes-engine/docs/how-to/preemptible-vms#best_practices https://cloud.google.com/kubernetes-engine/docs/how-to/preem... states that preemtible VMs aren't guaranteed to be available.
- raverbashing 7y agoYeah but Kubernetes "self-healing" a lot of times isn't. Or it just trips over itself (stuck pods, stuck cronjobs, etc).