4 ms·
What about comparing against spot market prices? Those AWS GPU instances are frequently $0.10 or $0.20 rather than $0.66. Not clearly a better deal anymore.
by gcr 10y ago
What about comparing against spot market prices? Those AWS GPU instances are frequently $0.10 or $0.20 rather than $0.66. Not clearly a better deal anymore.
- visarga 10y agoYes, if you are ok with having your neural net suddenly killed mid-training.
- dharma1 10y agoyou can set your max spot price high, and/or you can periodically save the trained weights (to an EBS volume) and continue from where you left off, same as recovering from a crash. Probably not a huge deal. Here's how you do it with Tensorflow: https://www.tensorflow.org/versions/r0.10/how_tos/variables/index.html#checkpoint-files https://www.tensorflow.org/versions/r0.10/how_tos/variables/... The sweet spot would be updated AWS GPU instances at spot prices. You could spin up XX instances for a relatively short time to train fast. Needs some Kubernetes etc cluster setup for distributed training but should be doable. If the spot price is $0.1-$0.2/hr that's pretty attractive, considering that is close to the price of electricity alone