5 ms·

Aws and Google spot instances are easily comparable price wise to hetzner but it really depends on what you are doing, which you haven't said. For example a g3

Aws and Google spot instances are easily comparable price wise to hetzner but it really depends on what you are doing, which you haven't said. For example a g3 spot instance can be as low as 19 cents per hour $138/month. Google GPU instances I've run for 14 cents an hour. I think these aren't the lowest prices either.

minimaxir 7y ago
For one-time GPU processing tasks that only take a couple hours, AWS spot / GCP preemptible is far more cost efficient than Hetzner. However, if you need to run a GPU 24/7 and/or have massive bandwidth requirements, Hetzner is far more cost effective than AWS/GCP. It's a balance of needs.
- streetcat1 7y ago
  Another option is to over provisioned spot instances, such that it would feel like a 24/7 instance, but cost like spot. However, you would probably need an automatic platform to achieve that.
  - minimaxir 7y ago
    An idea I've had is to use GCP preemptibles + script to automatically run a ML training job on instance start + Google Cloud Function + Cloud Scheduler to attempt to start the instance every minute if it gets preempted. The latter two are effectively free, so you'd get the cost benefits of preemptibles as long as the ML training job is resilient to random shutdowns.
    - streetcat1 7y ago
      Yes, this is basically the idea. However, there are different solutions for training and inference. For training, I would recommend that you add automatic checkpoint, and even consider model migration. For inference (which I think is the original concern), over provisioning is the key (simply because the fact that it would take a long time to load the model. Also, you also want to diversify your node types, etc.
etaioinshrdlu 7y ago
The main use case is on demand neural net inference 24/7 availability. Kind of like hosting a website that must be on 24/7, this neural net must be on 24/7. It runs on the order of 50x faster on an nvidia GPU compared to a CPU. But it will sit idle most of the time. Price per hour per GB of GPU ram is the most important metric when choosing a server.
- somuchtyler 7y ago
  I really like nocix servers, they are out of Kansas City. They got their "i7-6700K 32GB + 2x 480GB SSD + GTX 1080" for $105/mo https://www.nocix.net/cart/?id=338 https://www.nocix.net/cart/?id=338
  - etaioinshrdlu 7y ago
    This sounds great but they are also out of stock!
- rpedela 7y ago
  Would AWS Elastic Inference work for your use case? https://aws.amazon.com/machine-learning/elastic-inference/ https://aws.amazon.com/machine-learning/elastic-inference/
  - etaioinshrdlu 7y ago
    No, I need NVIDIA CUDA support. Also, the most important metric for me is cost per month per GB of GPU RAM, and AWS elastic inference is pretty bad in that metric.
coleca 7y ago
AWS spot instances for gpu can have availability issues as well or sometimes be priced higher than on demand (still trying to understand that one). If you do choose to run spot at aws (gpu or otherwise) be sure to check out the excellent project at autospotting.org and donate if you use it. Makes it super easy to replace on demand nodes in an ASG with spot nodes and always make sure you’re getting a good price.
- pixelwhale 7y ago
  > AWS spot instances for gpu can have availability issues as well or sometimes be priced higher than on demand (still trying to understand that one). You always pay the spot market price, not your bid. Your instance gets killed if somebody outbids you. A higher bid increases the probability that your instances don't get killed while at the same time letting you pay spot market price. By bidding above on-demand price, you are speculating that for the majority of the time, nobody else will bid more than the on-demand price. If you're not the only one doing that, spot price can rise above on-demand price.

minimaxir 7y ago

For one-time GPU processing tasks that only take a couple hours, AWS spot / GCP preemptible is far more cost efficient than Hetzner. However, if you need to run a GPU 24/7 and/or have massive bandwidth requirements, Hetzner is far more cost effective than AWS/GCP. It's a balance of needs.

streetcat1 7y ago
Another option is to over provisioned spot instances, such that it would feel like a 24/7 instance, but cost like spot. However, you would probably need an automatic platform to achieve that.
- minimaxir 7y ago
  An idea I've had is to use GCP preemptibles + script to automatically run a ML training job on instance start + Google Cloud Function + Cloud Scheduler to attempt to start the instance every minute if it gets preempted. The latter two are effectively free, so you'd get the cost benefits of preemptibles as long as the ML training job is resilient to random shutdowns.
  - streetcat1 7y ago
    Yes, this is basically the idea. However, there are different solutions for training and inference. For training, I would recommend that you add automatic checkpoint, and even consider model migration. For inference (which I think is the original concern), over provisioning is the key (simply because the fact that it would take a long time to load the model. Also, you also want to diversify your node types, etc.

etaioinshrdlu 7y ago

The main use case is on demand neural net inference 24/7 availability. Kind of like hosting a website that must be on 24/7, this neural net must be on 24/7. It runs on the order of 50x faster on an nvidia GPU compared to a CPU. But it will sit idle most of the time. Price per hour per GB of GPU ram is the most important metric when choosing a server.

somuchtyler 7y ago
I really like nocix servers, they are out of Kansas City. They got their "i7-6700K 32GB + 2x 480GB SSD + GTX 1080" for $105/mo https://www.nocix.net/cart/?id=338 https://www.nocix.net/cart/?id=338
- etaioinshrdlu 7y ago
  This sounds great but they are also out of stock!
rpedela 7y ago
Would AWS Elastic Inference work for your use case? https://aws.amazon.com/machine-learning/elastic-inference/ https://aws.amazon.com/machine-learning/elastic-inference/
- etaioinshrdlu 7y ago
  No, I need NVIDIA CUDA support. Also, the most important metric for me is cost per month per GB of GPU RAM, and AWS elastic inference is pretty bad in that metric.

coleca 7y ago

AWS spot instances for gpu can have availability issues as well or sometimes be priced higher than on demand (still trying to understand that one). If you do choose to run spot at aws (gpu or otherwise) be sure to check out the excellent project at autospotting.org and donate if you use it. Makes it super easy to replace on demand nodes in an ASG with spot nodes and always make sure you’re getting a good price.

pixelwhale 7y ago
> AWS spot instances for gpu can have availability issues as well or sometimes be priced higher than on demand (still trying to understand that one). You always pay the spot market price, not your bid. Your instance gets killed if somebody outbids you. A higher bid increases the probability that your instances don't get killed while at the same time letting you pay spot market price. By bidding above on-demand price, you are speculating that for the majority of the time, nobody else will bid more than the on-demand price. If you're not the only one doing that, spot price can rise above on-demand price.