you can set your max spot price high, and/or you can periodically save the trained weights (to an EBS volume) and continue from where you left off, same as reco

you can set your max spot price high, and/or you can periodically save the trained weights (to an EBS volume) and continue from where you left off, same as recovering from a crash. Probably not a huge deal. Here's how you do it with Tensorflow: https://www.tensorflow.org/versions/r0.10/how_tos/variables/index.html#checkpoint-files https://www.tensorflow.org/versions/r0.10/how_tos/variables/... The sweet spot would be updated AWS GPU instances at spot prices. You could spin up XX instances for a relatively short time to train fast. Needs some Kubernetes etc cluster setup for distributed training but should be doable. If the spot price is $0.1-$0.2/hr that's pretty attractive, considering that is close to the price of electricity alone