5 ms·

We built a tool called spotML to make training on AWS/GCP cheaper. Spot Instances are 70% cheaper than On-Demand instances but are prone to interruptions. We m

We built a tool called spotML to make training on AWS/GCP cheaper. Spot Instances are 70% cheaper than On-Demand instances but are prone to interruptions. We mitigate the downside of these interruptions through the use of persistence features, including optional fallback to On-Demand instances. So you can optimize workflows according to your budget and time constraints. History: We were working on a neural rendering startup that needed a lot of GAN training which was getting very expensive. We were blowing roughly $1000, to train a single category class. Training on Spot instances was cheaper, but still a mess. It needed lot of hand holding/devops stuff to make it usable. So we built SpotML to automate a lot of things. Posting it here to see if the community finds this helpful, so that we can open it up to the larger community.

cpalumbo 5y ago
Ha! That's interesting. We use AWS instances for model development and costs are definitely an issue. Sending this to my team. Good luck!
oakfr 5y ago
This looks like an obvious approach (in the hindsight of course) to a general problem. I love your pivot. Congrats on the idea and godspeed. You’ll probably have a lot of interest if you execute well.
- vishnukool 5y ago
  Thank you!
zachthewf 5y ago
Really excited to try it out. I've had a heck of a time setting up spot instance training on Sagemaker in the past so simplification efforts are much appreciated.
- vishnukool 5y ago
  Curious, did you eventually start using Sagemaker with spot instances or did you give up on it ? Also what would you say were the biggest pain points with Sagemaker?
talolard 5y ago
This looks useful, I like the pricing of free/$9.99 a month. No one asked, but it's HN so I'll say it anyway. I think it's a questionable "vc business" but a great business for 1-2 people. The road from this to an enterprise sales motion, or even a 10K/year contract is hard for me to imagine. At some point, it becomes cost effective for my org to build this functionality in house. However, as a hobbyist /single dev / small team, $120/Year is a no brainer after the first 2-3K I spend on GPU by mistake. As you know, setting up spots when I just want to get shit done is a pain and I'll gladly pay you (a little) to make that go away. Still no one asked but... One thing that plays to your advantage is that the current price point is something anyone in your user group can buy on their own, and there are a lot of us / enough to make a nice business out of. Good luck!
- vishnukool 5y ago
  Thanks, yeah. At this stage we really just want to validate if this was a real problem in the ML community. Down the line I suppose as we scale to handle multi instance training and other use cases, we could probably charge more, say a % of the cost savings in training.
ignoramous 5y ago
Neat. Congratulations on the launch, Vishnu! Apart from the fact that it could deploy to both GCP and AWS, what does it do differently than AWS Batch [0]? When we had a similar problem, we ran jobs on spots with AWS Batch and it worked nicely enough. Some suggestions (for a later date): 1a. Add built-in support for Ray [1] (you'd essentially be then competing with Anyscale, which is a VC funded startup, just to contrast it with another comment on this thread) and dbt [2]. 1b. Or: Support deploying coin miners (might help widen the product's reach; and stand it up against the likes of consensys). 3. Get in front of the very many cost optimisation consultants out there, like the Duckbill Group. If I may, where are you building this product from? And how many are on the team? Thanks. [0] https://aws.amazon.com/batch/use-cases/ https://aws.amazon.com/batch/use-cases/ [1] https://ray.io/ https://ray.io/ [2] https://getdbt.com/ https://getdbt.com/
- vishnukool 5y ago
  Thank you! Interesting, we actually tried aws batch ourselves. 1) How were you able to handle spot interruptions and resuming from the latest checkpoint ? 2) Not to mention fallback to OnDemand on spot interruptions 3) then switching back to spot from onDemand would also need additional process to be setup. Also i'm not sure how straightforward it is to detach/attach persistent volumes to retain data across different spot interruptions ? The latter can be done but it's just the same rote each time you wanna train something new. Also thanks for the suggestions ! We're a team of 2 right now, I used to be in the bay area but in Mexico temporarily.
  - ignoramous 5y ago
    1. Spot interruptions didn't matter much as AWS Batch looks for spots with low interruption probability. Auto retries kicked-in whenever those did get interrupted. 2. Checkpointing was a pain (we relied mostly on AWS Batch's JobState and S3, not ideal), but the current capability to mount EFS (Elastic Filesystem) looks like it would solve this? 3. No hot swapping on-demand with spot and vice versa. Interestingly, ALB (Application Load Balancer) supports such mixed EC2 configurations (AWS Batch doesn't).
ssivark 5y ago
Kudos; this is likely a very useful tool! So useful (and potentially lucrative) that AWS/GCP would likely take over this functionality eventually -- either by building on top of spot instances like you have, or underneath regular instances or re-use that capacity for some other managed service (thereby reducing the price differential). How do you plan to protect yourselves against that?
rgj 5y ago
You should really mention / give attribution / emphasize more that this is a fork of https://spotty.cloud https://spotty.cloud and you took a lot from https://github.com/nimbo-sh/nimbo https://github.com/nimbo-sh/nimbo as well.

cpalumbo 5y ago

Ha! That's interesting. We use AWS instances for model development and costs are definitely an issue. Sending this to my team. Good luck!

oakfr 5y ago

This looks like an obvious approach (in the hindsight of course) to a general problem. I love your pivot. Congrats on the idea and godspeed. You’ll probably have a lot of interest if you execute well.

vishnukool 5y ago
Thank you!

zachthewf 5y ago

Really excited to try it out. I've had a heck of a time setting up spot instance training on Sagemaker in the past so simplification efforts are much appreciated.

vishnukool 5y ago
Curious, did you eventually start using Sagemaker with spot instances or did you give up on it ? Also what would you say were the biggest pain points with Sagemaker?

talolard 5y ago

This looks useful, I like the pricing of free/$9.99 a month. No one asked, but it's HN so I'll say it anyway. I think it's a questionable "vc business" but a great business for 1-2 people. The road from this to an enterprise sales motion, or even a 10K/year contract is hard for me to imagine. At some point, it becomes cost effective for my org to build this functionality in house. However, as a hobbyist /single dev / small team, $120/Year is a no brainer after the first 2-3K I spend on GPU by mistake. As you know, setting up spots when I just want to get shit done is a pain and I'll gladly pay you (a little) to make that go away. Still no one asked but... One thing that plays to your advantage is that the current price point is something anyone in your user group can buy on their own, and there are a lot of us / enough to make a nice business out of. Good luck!

vishnukool 5y ago
Thanks, yeah. At this stage we really just want to validate if this was a real problem in the ML community. Down the line I suppose as we scale to handle multi instance training and other use cases, we could probably charge more, say a % of the cost savings in training.

ignoramous 5y ago

Neat. Congratulations on the launch, Vishnu! Apart from the fact that it could deploy to both GCP and AWS, what does it do differently than AWS Batch [0]? When we had a similar problem, we ran jobs on spots with AWS Batch and it worked nicely enough. Some suggestions (for a later date): 1a. Add built-in support for Ray [1] (you'd essentially be then competing with Anyscale, which is a VC funded startup, just to contrast it with another comment on this thread) and dbt [2]. 1b. Or: Support deploying coin miners (might help widen the product's reach; and stand it up against the likes of consensys). 3. Get in front of the very many cost optimisation consultants out there, like the Duckbill Group. If I may, where are you building this product from? And how many are on the team? Thanks. [0] https://aws.amazon.com/batch/use-cases/ https://aws.amazon.com/batch/use-cases/ [1] https://ray.io/ https://ray.io/ [2] https://getdbt.com/ https://getdbt.com/

vishnukool 5y ago
Thank you! Interesting, we actually tried aws batch ourselves. 1) How were you able to handle spot interruptions and resuming from the latest checkpoint ? 2) Not to mention fallback to OnDemand on spot interruptions 3) then switching back to spot from onDemand would also need additional process to be setup. Also i'm not sure how straightforward it is to detach/attach persistent volumes to retain data across different spot interruptions ? The latter can be done but it's just the same rote each time you wanna train something new. Also thanks for the suggestions ! We're a team of 2 right now, I used to be in the bay area but in Mexico temporarily.
- ignoramous 5y ago
  1. Spot interruptions didn't matter much as AWS Batch looks for spots with low interruption probability. Auto retries kicked-in whenever those did get interrupted. 2. Checkpointing was a pain (we relied mostly on AWS Batch's JobState and S3, not ideal), but the current capability to mount EFS (Elastic Filesystem) looks like it would solve this? 3. No hot swapping on-demand with spot and vice versa. Interestingly, ALB (Application Load Balancer) supports such mixed EC2 configurations (AWS Batch doesn't).

ssivark 5y ago

Kudos; this is likely a very useful tool! So useful (and potentially lucrative) that AWS/GCP would likely take over this functionality eventually -- either by building on top of spot instances like you have, or underneath regular instances or re-use that capacity for some other managed service (thereby reducing the price differential). How do you plan to protect yourselves against that?

rgj 5y ago

You should really mention / give attribution / emphasize more that this is a fork of https://spotty.cloud https://spotty.cloud and you took a lot from https://github.com/nimbo-sh/nimbo https://github.com/nimbo-sh/nimbo as well.