3 ms·

This is really cool! I used to have these same pains. My trick with spot instances has been to set my maximum price to the price of a regular instance of that

This is really cool! I used to have these same pains. My trick with spot instances has been to set my maximum price to the price of a regular instance of that class or higher and to sync weights to S3 in the background on every save. The former is a parameter when starting the instance in the console or terraform, the latter is basically a do...while loop. I've noticed that often one gets booted from a spot instance causing interruption because the market price increases a few cents above the "70% savings price." Increasing the maximum to on demand is basically free money because you don't get booted often, and your max price is the regular on demand price. This has seemed to mitigate about all of the spot downsides (like interruptions or losing data) because you don't easily get kicked out of the spot unless there's a run on machines and the prices rarely fluctuate that much (at least for the higher end p3 instances). This has seemed to prevent data loss and protected the downside risk by setting the instance to a knowable max price. There are instances where the spot price goes higher than the on demand price, so you still get booted once in a while, but it's very infrequent as you can imagine most people don't want to spend more for spot than on demand. Anecdotally I still average out to getting the vast majority of the spot savings with this method with very few interruptions. Looking at the SpotML it seems to be a lot of tooling that achieves these same goals (assuming one would be interrupted when a spot dies and moves to full-freight on-demand with SpotML), which makes SpotML's solution feel very over-engineered to me given that the majority of what SpotML provides can be had with a simple maximum cost parameter change when spinning up a spot instance. I would be very interested in using anything that doesn't have great overhead and saves money. Our bill seems "big" to me (but I realize it may be small to many others) at $200k/mo, so even these small savings add up. Would you compare the potential benefits of SpotML to the method I described above?

a-dub 5y ago
i always envisioned that if i ever did this, i would set the spot price to the equivalent of infinity/max and then monitor it myself, terminating myself if i determined that continuing to run at the spot price was more expensive. why? sure, you don't want to pay more than the on demand price, but iirc spot prices often spike very momentarily. so the question becomes whether the sum over spike time cost at the higher spot price exceeds the time to boot/shutdown/migration overhead at the on demand price. but i've never actually tried it so...
- vishnukool 5y ago
  This is interesting, from what I've read AWS no longer recommends setting spot instance prices so that they manage it themselves. I wonder, if there's an actual advantage of avoiding spot interruptions by setting the spot price even higher than onDemand.
  - a-dub 5y ago
    yeah true. it's an interesting question that involves not only startup/shutdown of instance/os overheads (i suppose they've probably thought about this) but also overhead for checkpoint/restart of your application. i also suppose this way of thinking about it comes from thinking around how to minimize cost from a purely mathematical standpoint. when you think about how and why the spot market is operated, and what those short term spikes may actually be, it may run counter to the intended purpose. (cheap capacity that they may recall at any time, because capacity is actually fixed) funny how market based approaches can gamify things sufficiently that sometimes they obscure the underlying intention or purpose of having a market in the first place.
vishnukool 5y ago
Thanks for the feedback. The biggest upside is, if you have a long running training(say hours, days) and the spot training is interrupted. You probably don't want to manually monitor to check for the next available spot instance and kick start the training. SpotML takes care of that part. Also optionally, you can configure it to resume with an onDemand instance on interruption until the next available spot instance. In essence we try to do it make i) creating buckets/EBS ii) code to save to S3 in loop iii) Monitor for interruptions and resume from checkpoint parts easy.
electroly 5y ago
FWIW your "trick" has been the default for years. Unless you specify otherwise, the default max price is the on-demand price. They changed this back in 2017. Maximum price isn't really used by anyone any more. https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/ https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pri... We spend about $10k/month on spot instances and I don't specify any max price. The way to avoid terminations is just to make sure you spread your workload over a large number of instance types and availability zones.

a-dub 5y ago

i always envisioned that if i ever did this, i would set the spot price to the equivalent of infinity/max and then monitor it myself, terminating myself if i determined that continuing to run at the spot price was more expensive. why? sure, you don't want to pay more than the on demand price, but iirc spot prices often spike very momentarily. so the question becomes whether the sum over spike time cost at the higher spot price exceeds the time to boot/shutdown/migration overhead at the on demand price. but i've never actually tried it so...

vishnukool 5y ago
This is interesting, from what I've read AWS no longer recommends setting spot instance prices so that they manage it themselves. I wonder, if there's an actual advantage of avoiding spot interruptions by setting the spot price even higher than onDemand.
- a-dub 5y ago
  yeah true. it's an interesting question that involves not only startup/shutdown of instance/os overheads (i suppose they've probably thought about this) but also overhead for checkpoint/restart of your application. i also suppose this way of thinking about it comes from thinking around how to minimize cost from a purely mathematical standpoint. when you think about how and why the spot market is operated, and what those short term spikes may actually be, it may run counter to the intended purpose. (cheap capacity that they may recall at any time, because capacity is actually fixed) funny how market based approaches can gamify things sufficiently that sometimes they obscure the underlying intention or purpose of having a market in the first place.

vishnukool 5y ago

Thanks for the feedback. The biggest upside is, if you have a long running training(say hours, days) and the spot training is interrupted. You probably don't want to manually monitor to check for the next available spot instance and kick start the training. SpotML takes care of that part. Also optionally, you can configure it to resume with an onDemand instance on interruption until the next available spot instance. In essence we try to do it make i) creating buckets/EBS ii) code to save to S3 in loop iii) Monitor for interruptions and resume from checkpoint parts easy.

electroly 5y ago

FWIW your "trick" has been the default for years. Unless you specify otherwise, the default max price is the on-demand price. They changed this back in 2017. Maximum price isn't really used by anyone any more. https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/ https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pri... We spend about $10k/month on spot instances and I don't specify any max price. The way to avoid terminations is just to make sure you spread your workload over a large number of instance types and availability zones.