3 ms·

So, probably a very naive question here, but how his the "continuity" across pre-emptions working? Say I am running a long batch process on one of these spot V

So, probably a very naive question here, but how his the "continuity" across pre-emptions working? Say I am running a long batch process on one of these spot VMs and I get pre-empted ... Does my job restart where it was stopped automatically and everything is transparent save how long the job takes to complete, or do I have do to checkpointing myself and deal with the fact that my jobs may be killed at anytime? Also, if the restarts are indeed transparent to my job, what of stateful network connections? Finally, what kind of guarantee do I have that my job will ever complete if pre-emption can happen at an arbitrary frequency for arbitrarily long?

tecleandor 5y ago
Restarting the job is on you, the instance will be destroyed. So you'll need to adapt your process to be resumable (or partially resumable through checkpoints) and/or idempotent, so nothing goes wrong if you run the job (or parts of the job) twice.
- ur-whale 5y ago
  OK, thanks, that answers my question: there is not continuity whatsoever, you have to take care of everything yourself.
belval 5y ago
Spot instances are the equivalent of stopping an instance while it's running. You get a signal (30 seconds prior for Google, 2 mins for AWS) and you are free to try and handoff the work to a different machine in your cluster . Basically this is good for running short jobs and not long running ones. If you have a service that process chunks of data from a queue it's probably the ideal scenario. In practice, with AWS your instances don't get killed that often. Some in the comments are claiming months but I was using mine with AWS Batch and they usually lived 1-2 weeks.