4 ms·

Nevertheless, doesn't that force you to divide your work up into units somehow and manage external storage to track what/how much is done?

by titzer 5y ago

jiggawatts 5y ago
Yes, but this is common for "cloud native" applications. Web servers are ideally stateless "cattle", not "pets". Code is pulled down via CI/CD pipelines or via container images. State is in databases or blobs. Logs stream to some sort of analytics system. Similarly, Kubernetes works well with spot VMs. You can have Spot and Regular node pools in one cluster, and then your workloads can happily spread out to the extra capacity at a huge discount.
- Spivak 5y ago
  It's really easy to "beat" the chaos monkey with this kind of design. You can destroy any VMs in our infra with reckless abandon and everything will be fine. If you destroy all the VMs of particular service obviously it will affect availability but it will still recover by itself in about 60 seconds.
wbl 5y ago
You have to do that anyway for serious batch processing.
NavinF 5y ago
You gotta do that anyway. It really sucks when a multi-day job crashes halfway through because of some silly bug and then you gotta restart from the beginning. The ability to resume from crashes is required regardless of where you're running the code.
pushrax 5y ago
The reality at scale is that all cloud VMs are preemptible and unreliable. You can pay more to increase the probability of uptime but inevitably have to recover from failure sometimes.
- junon 5y ago
  This is absolutely the correct way of thinking about this problem, and exactly how most people I've seen successfully use pre-empt instances treat them as well.
- ClumsyPilot 5y ago
  The reality is itselt preemptible and unreliable, you could be terminated without notice
pm90 5y ago
There are tools that enable this (e.g. look at temporal.io). As for external storage: yes, you do generate more metadata, but that's often a tiny fraction of the cost saved by moving to spot machines. In general, these tools are adopted more for increasing reliability of existing systems, but I predict they would be a neat fit to run them on spot machines.
- orlovs 5y ago
  +1 for Temporal. Using in prod, its great
  - swyx 5y ago
    (i work at Temporal) nice! glad you're happy with it, would love to learn more if you have any compliments/complaints, lets hear it :)
dagw 5y ago
That is probably the primary use case for spot instances and certainly how I tend to used them. I have a few thousand files that need processing and each files takes 30 minutes. So I spin up 100 spot instances and use a small script to copy the file from a control machine to the next available remote machine, start the processing script, and then copy the result back when done. I've also got another script that keeps track of how many spot instances are currently running and spins up some new ones (possibly in a different region) if they fall below a certain level.

Nevertheless, doesn't that force you to divide your work up into units somehow and manage external storage to track what/how much is done?

by titzer 5y ago

Nevertheless, doesn't that force you to divide your work up into units somehow and manage external storage to track what/how much is done?

jiggawatts 5y ago

Yes, but this is common for "cloud native" applications. Web servers are ideally stateless "cattle", not "pets". Code is pulled down via CI/CD pipelines or via container images. State is in databases or blobs. Logs stream to some sort of analytics system. Similarly, Kubernetes works well with spot VMs. You can have Spot and Regular node pools in one cluster, and then your workloads can happily spread out to the extra capacity at a huge discount.

Spivak 5y ago
It's really easy to "beat" the chaos monkey with this kind of design. You can destroy any VMs in our infra with reckless abandon and everything will be fine. If you destroy all the VMs of particular service obviously it will affect availability but it will still recover by itself in about 60 seconds.

wbl 5y ago

You have to do that anyway for serious batch processing.

NavinF 5y ago

You gotta do that anyway. It really sucks when a multi-day job crashes halfway through because of some silly bug and then you gotta restart from the beginning. The ability to resume from crashes is required regardless of where you're running the code.

pushrax 5y ago

The reality at scale is that all cloud VMs are preemptible and unreliable. You can pay more to increase the probability of uptime but inevitably have to recover from failure sometimes.

junon 5y ago
This is absolutely the correct way of thinking about this problem, and exactly how most people I've seen successfully use pre-empt instances treat them as well.
ClumsyPilot 5y ago
The reality is itselt preemptible and unreliable, you could be terminated without notice

pm90 5y ago

There are tools that enable this (e.g. look at temporal.io). As for external storage: yes, you do generate more metadata, but that's often a tiny fraction of the cost saved by moving to spot machines. In general, these tools are adopted more for increasing reliability of existing systems, but I predict they would be a neat fit to run them on spot machines.

orlovs 5y ago
+1 for Temporal. Using in prod, its great
- swyx 5y ago
  (i work at Temporal) nice! glad you're happy with it, would love to learn more if you have any compliments/complaints, lets hear it :)

dagw 5y ago

That is probably the primary use case for spot instances and certainly how I tend to used them. I have a few thousand files that need processing and each files takes 30 minutes. So I spin up 100 spot instances and use a small script to copy the file from a control machine to the next available remote machine, start the processing script, and then copy the result back when done. I've also got another script that keeps track of how many spot instances are currently running and spins up some new ones (possibly in a different region) if they fall below a certain level.