5 ms·

I have trouble thinking of a use case that fits this VM type. Is it for batch processing tasks that don't have tight deadlines? The lack of any real guarantees

by titzer 5y ago

I have trouble thinking of a use case that fits this VM type. Is it for batch processing tasks that don't have tight deadlines? The lack of any real guarantees makes it hard to price, as you have no idea what you're actually paying for. Would it make sense to have a model that includes deadlines in its pricing (like give this process 2.5 hours of CPU and 4GB of RAM to complete this task by...next Tuesday)?

wmf 5y ago
You might be misunderstanding how spot works. The VM either runs at full speed or not at all so you know what you're getting.
- titzer 5y ago
  Nevertheless, doesn't that force you to divide your work up into units somehow and manage external storage to track what/how much is done?
  - jiggawatts 5y ago
    Yes, but this is common for "cloud native" applications. Web servers are ideally stateless "cattle", not "pets". Code is pulled down via CI/CD pipelines or via container images. State is in databases or blobs. Logs stream to some sort of analytics system. Similarly, Kubernetes works well with spot VMs. You can have Spot and Regular node pools in one cluster, and then your workloads can happily spread out to the extra capacity at a huge discount.
    - Spivak 5y ago
      It's really easy to "beat" the chaos monkey with this kind of design. You can destroy any VMs in our infra with reckless abandon and everything will be fine. If you destroy all the VMs of particular service obviously it will affect availability but it will still recover by itself in about 60 seconds.
  - wbl 5y ago
    You have to do that anyway for serious batch processing.
  - NavinF 5y ago
    You gotta do that anyway. It really sucks when a multi-day job crashes halfway through because of some silly bug and then you gotta restart from the beginning. The ability to resume from crashes is required regardless of where you're running the code.
  - pushrax 5y ago
    The reality at scale is that all cloud VMs are preemptible and unreliable. You can pay more to increase the probability of uptime but inevitably have to recover from failure sometimes.
    - junon 5y ago
      This is absolutely the correct way of thinking about this problem, and exactly how most people I've seen successfully use pre-empt instances treat them as well.
    - ClumsyPilot 5y ago
      The reality is itselt preemptible and unreliable, you could be terminated without notice
  - pm90 5y ago
    There are tools that enable this (e.g. look at temporal.io). As for external storage: yes, you do generate more metadata, but that's often a tiny fraction of the cost saved by moving to spot machines. In general, these tools are adopted more for increasing reliability of existing systems, but I predict they would be a neat fit to run them on spot machines.
    - orlovs 5y ago
      +1 for Temporal. Using in prod, its great
      - swyx 5y ago
        (i work at Temporal) nice! glad you're happy with it, would love to learn more if you have any compliments/complaints, lets hear it :)
  - dagw 5y ago
    That is probably the primary use case for spot instances and certainly how I tend to used them. I have a few thousand files that need processing and each files takes 30 minutes. So I spin up 100 spot instances and use a small script to copy the file from a control machine to the next available remote machine, start the processing script, and then copy the result back when done. I've also got another script that keeps track of how many spot instances are currently running and spins up some new ones (possibly in a different region) if they fall below a certain level.
zamalek 5y ago
If your code can survive chaos monkey, it can trivially weather this. Netflix, as one example, could run their entire infrastructure on these VMs.
- snewman 5y ago
  Except that (presumably) there is no guarantee that any spot instances are available. You could wind up in a state where you have no instances running. No user-facing production service can tolerate that.
  - roca 5y ago
    Just spin up regular VMs if you can't get spot instances when you need them.
    - deleted 5y ago
      [deleted]
    - xyzzyz 5y ago
      Lack of spot VM means that all spare capacity has been taken up by the regular VMs, meaning that it spots are gone, so are regular ones.
      - roca 5y ago
        Maybe that's true but we haven't seen this happen in practice. Maybe we're just incredibly lucky but we're able to get on-demand instances to recover after spot termination.
      - apendleton 5y ago
        Having worked on infrastructure deployed this way, I can say this oversimplifies a bit. There's virtually never no spot capacity at all, there's no capacity at a given bid price, for a given instance type, in a given region. You can start to scale up on-demand infrastructure using spot price as a signal, before there's literally none left, and you can also run on clusters of heterogeneous instances, and change your instance type mix when one instance type's price goes up. In the worst case you can fail out of a particular region whose spot prices have gotten too high. Lots of options.
whazor 5y ago
With Spark you can indeed run batch jobs on multiple VMs. We are currently working having dynamic allocation and preemption on Kubernetes. This is for us essential for our on premise cluster, as we want fair sharing. But for our cloud GCP jobs we could indeed save money by having a pool of spot instances. If deadlines are important, you could replace them by normal instances.
teraflop 5y ago
That's correct: spot instances are ideal for batch workloads with relatively little persistent state. If your deadline for completing a unit of work isn't extremely short, and you're already distributing jobs across a pool of worker nodes, it's relatively straightforward to dynamically start spot instances if they're available, and on-demand instances otherwise. Technically, you're gambling a bit because whenever a spot instance is terminated, you probably need to spend a bit extra to redo whatever work it was in the middle of. But spot instances have such a huge discount that it's almost always worth taking that risk, as long as you don't mind the extra management complexity.
- deleted 5y ago
  [deleted]
roca 5y ago
All our pernos.co infrastructure uses AWS spot VMs. Batch processing runs on spot and fails over to regular VMs as necessary. CI jobs run on spot. Our front-end server running the Web site and our application also runs on spot and automatically fails over to a regular VM if it gets terminated. Termination is rare.
- technics256 5y ago
  How do you coordinate the failover? Custom scripts or?
  - roca 5y ago
    We use custom logic. There are various ways to do it. One way to do it is to poll for the termination signal on the running instance, as part of your app say, and if you see termination coming, trigger a Lambda that supervises the bringup of an on-demand instance. For critical stuff you probably want a backup system that checks instance health and brings up a new instance if the old one is not available ... but you probably want that even if you always run on-demand.
toast0 5y ago
The 30 second shutdown warning makes this a bit tricky for interactive work, but if your interactive requests will all finish within 20 seconds or so (or you can hand them off), and 5 seconds to get out of load balancing, this could be cost savings for whatever fits; some base number of nodes + a number of spot nodes as demand requires, if no spot nodes are available, serve from somewhat overloaded base nodes + maybe some regular priced nodes (but fewer extra nodes than at spot prices). Or, more likely, for batch processing that's mostly time insensitive. Things that can be resumed without losing much work or lots of small short jobs. Retranscoding media with new settings comes to mind. If you've got deadlines, you'd probably need a mix of on-demand and spot nodes. This is the corner of cloud pricing where if you can optimize use of this, you might be able to do better than traditional hosting. But only if your needs are variable, or you get a lot of high discount spot pricing.
hiyer 5y ago
> Is it for batch processing tasks that don't have tight deadlines? I used to work for a company (since shut down) which provided big data processing systems in the cloud. This was one of the typical use-cases for our customers. Big data systems like Hadoop, Spark, etc are built to handle this kind of disruption where you lose a few nodes once in a while, and we had built in further optimizations to do it even better. This fact, combined with the much cheaper price of Spot Instances (upto 90% less than on-demand) make them a compelling alternative to on-demand instances. In practice - at least on AWS - spot loss used to be quite rare. When it happened, it happened by the truckload, but we used to have spot instances run for several days without termination.
- khuey 5y ago
  I've had AWS spot instances run for months without termination.
bubblethink 5y ago
They are also great if you are paying out of your own pocket and not your employer's. For side projects, academic work, etc., that need GPU instances or large memory instances, the regular pricing is quite out of whack. Spot pricing makes it tolerable.
ramoz 5y ago
My team runs fairly large GKE clusters on preemptible machines, and with a dedicated footprint that can scale if needed. Business critical workloads, we ensure a lot of these https://cloud.google.com/kubernetes-engine/docs/concepts/spot-vms#best_practices https://cloud.google.com/kubernetes-engine/docs/concepts/spo... cost savings.
Thristle 5y ago
1. You only pay for compute time you actaully get 2. Any stateless process such as web servers are ideal for spots 3. Also k8s nodes are very good to run in spot since pods are "natively" crash "resistant"

by titzer 5y ago

wmf 5y ago

You might be misunderstanding how spot works. The VM either runs at full speed or not at all so you know what you're getting.

titzer 5y ago
Nevertheless, doesn't that force you to divide your work up into units somehow and manage external storage to track what/how much is done?
- jiggawatts 5y ago
  Yes, but this is common for "cloud native" applications. Web servers are ideally stateless "cattle", not "pets". Code is pulled down via CI/CD pipelines or via container images. State is in databases or blobs. Logs stream to some sort of analytics system. Similarly, Kubernetes works well with spot VMs. You can have Spot and Regular node pools in one cluster, and then your workloads can happily spread out to the extra capacity at a huge discount.
  - Spivak 5y ago
    It's really easy to "beat" the chaos monkey with this kind of design. You can destroy any VMs in our infra with reckless abandon and everything will be fine. If you destroy all the VMs of particular service obviously it will affect availability but it will still recover by itself in about 60 seconds.
- wbl 5y ago
  You have to do that anyway for serious batch processing.
- NavinF 5y ago
  You gotta do that anyway. It really sucks when a multi-day job crashes halfway through because of some silly bug and then you gotta restart from the beginning. The ability to resume from crashes is required regardless of where you're running the code.
- pushrax 5y ago
  The reality at scale is that all cloud VMs are preemptible and unreliable. You can pay more to increase the probability of uptime but inevitably have to recover from failure sometimes.
  - junon 5y ago
    This is absolutely the correct way of thinking about this problem, and exactly how most people I've seen successfully use pre-empt instances treat them as well.
  - ClumsyPilot 5y ago
    The reality is itselt preemptible and unreliable, you could be terminated without notice
- pm90 5y ago
  There are tools that enable this (e.g. look at temporal.io). As for external storage: yes, you do generate more metadata, but that's often a tiny fraction of the cost saved by moving to spot machines. In general, these tools are adopted more for increasing reliability of existing systems, but I predict they would be a neat fit to run them on spot machines.
  - orlovs 5y ago
    +1 for Temporal. Using in prod, its great
    - swyx 5y ago
      (i work at Temporal) nice! glad you're happy with it, would love to learn more if you have any compliments/complaints, lets hear it :)
- dagw 5y ago
  That is probably the primary use case for spot instances and certainly how I tend to used them. I have a few thousand files that need processing and each files takes 30 minutes. So I spin up 100 spot instances and use a small script to copy the file from a control machine to the next available remote machine, start the processing script, and then copy the result back when done. I've also got another script that keeps track of how many spot instances are currently running and spins up some new ones (possibly in a different region) if they fall below a certain level.

zamalek 5y ago

If your code can survive chaos monkey, it can trivially weather this. Netflix, as one example, could run their entire infrastructure on these VMs.

snewman 5y ago
Except that (presumably) there is no guarantee that any spot instances are available. You could wind up in a state where you have no instances running. No user-facing production service can tolerate that.
- roca 5y ago
  Just spin up regular VMs if you can't get spot instances when you need them.
  - deleted 5y ago
    [deleted]
  - xyzzyz 5y ago
    Lack of spot VM means that all spare capacity has been taken up by the regular VMs, meaning that it spots are gone, so are regular ones.
    - roca 5y ago
      Maybe that's true but we haven't seen this happen in practice. Maybe we're just incredibly lucky but we're able to get on-demand instances to recover after spot termination.
    - apendleton 5y ago
      Having worked on infrastructure deployed this way, I can say this oversimplifies a bit. There's virtually never no spot capacity at all, there's no capacity at a given bid price, for a given instance type, in a given region. You can start to scale up on-demand infrastructure using spot price as a signal, before there's literally none left, and you can also run on clusters of heterogeneous instances, and change your instance type mix when one instance type's price goes up. In the worst case you can fail out of a particular region whose spot prices have gotten too high. Lots of options.

whazor 5y ago

With Spark you can indeed run batch jobs on multiple VMs. We are currently working having dynamic allocation and preemption on Kubernetes. This is for us essential for our on premise cluster, as we want fair sharing. But for our cloud GCP jobs we could indeed save money by having a pool of spot instances. If deadlines are important, you could replace them by normal instances.

teraflop 5y ago

That's correct: spot instances are ideal for batch workloads with relatively little persistent state. If your deadline for completing a unit of work isn't extremely short, and you're already distributing jobs across a pool of worker nodes, it's relatively straightforward to dynamically start spot instances if they're available, and on-demand instances otherwise. Technically, you're gambling a bit because whenever a spot instance is terminated, you probably need to spend a bit extra to redo whatever work it was in the middle of. But spot instances have such a huge discount that it's almost always worth taking that risk, as long as you don't mind the extra management complexity.

deleted 5y ago
[deleted]

roca 5y ago

All our pernos.co infrastructure uses AWS spot VMs. Batch processing runs on spot and fails over to regular VMs as necessary. CI jobs run on spot. Our front-end server running the Web site and our application also runs on spot and automatically fails over to a regular VM if it gets terminated. Termination is rare.

technics256 5y ago
How do you coordinate the failover? Custom scripts or?
- roca 5y ago
  We use custom logic. There are various ways to do it. One way to do it is to poll for the termination signal on the running instance, as part of your app say, and if you see termination coming, trigger a Lambda that supervises the bringup of an on-demand instance. For critical stuff you probably want a backup system that checks instance health and brings up a new instance if the old one is not available ... but you probably want that even if you always run on-demand.

toast0 5y ago

The 30 second shutdown warning makes this a bit tricky for interactive work, but if your interactive requests will all finish within 20 seconds or so (or you can hand them off), and 5 seconds to get out of load balancing, this could be cost savings for whatever fits; some base number of nodes + a number of spot nodes as demand requires, if no spot nodes are available, serve from somewhat overloaded base nodes + maybe some regular priced nodes (but fewer extra nodes than at spot prices). Or, more likely, for batch processing that's mostly time insensitive. Things that can be resumed without losing much work or lots of small short jobs. Retranscoding media with new settings comes to mind. If you've got deadlines, you'd probably need a mix of on-demand and spot nodes. This is the corner of cloud pricing where if you can optimize use of this, you might be able to do better than traditional hosting. But only if your needs are variable, or you get a lot of high discount spot pricing.

hiyer 5y ago

> Is it for batch processing tasks that don't have tight deadlines? I used to work for a company (since shut down) which provided big data processing systems in the cloud. This was one of the typical use-cases for our customers. Big data systems like Hadoop, Spark, etc are built to handle this kind of disruption where you lose a few nodes once in a while, and we had built in further optimizations to do it even better. This fact, combined with the much cheaper price of Spot Instances (upto 90% less than on-demand) make them a compelling alternative to on-demand instances. In practice - at least on AWS - spot loss used to be quite rare. When it happened, it happened by the truckload, but we used to have spot instances run for several days without termination.

khuey 5y ago
I've had AWS spot instances run for months without termination.

bubblethink 5y ago

They are also great if you are paying out of your own pocket and not your employer's. For side projects, academic work, etc., that need GPU instances or large memory instances, the regular pricing is quite out of whack. Spot pricing makes it tolerable.

ramoz 5y ago

My team runs fairly large GKE clusters on preemptible machines, and with a dedicated footprint that can scale if needed. Business critical workloads, we ensure a lot of these https://cloud.google.com/kubernetes-engine/docs/concepts/spot-vms#best_practices https://cloud.google.com/kubernetes-engine/docs/concepts/spo... cost savings.

Thristle 5y ago

1. You only pay for compute time you actaully get 2. Any stateless process such as web servers are ideal for spots 3. Also k8s nodes are very good to run in spot since pods are "natively" crash "resistant"