3 ms·
The 30 second shut-off notice is pretty bad for deep learning training, IMO. You often will not be able to finish a mini-batch of training and save a model to d
by argonaut 9y ago
The 30 second shut-off notice is pretty bad for deep learning training, IMO. You often will not be able to finish a mini-batch of training and save a model to disk in 30 seconds. There are probably ways to interrupt processing of a mini-batch, but it'll require some custom code (it probably will be hard to do in Keras, for example).
- gputhrowaway 9y agoSince you are paying 50% discount and if you save at every 5 minutes you still come ahead, not to forget that if you get pre-empted within first 10 minutes (assuming 4 minutes to spin up the training) you got those batches for free!
- argonaut 9y agoGreat! Depending on the framework you use this will not be trivial to implement correctly.
- boulos 9y agoDisclosure: I work on Google Cloud (and helped launch this). Actually both Keras and raw TF have a check point thing built in (as does Torch). I believe it can be setup to do so every epoch, but unless you're talking about many many GBs of parameters, you can also stream lots of data to GCS in 30s.
- argonaut 9y agoYou're talking about support for checkpointing every epoch, but a 30 second shutoff would require checkpointing every N mini-batches, which is not a typical use for checkpointing.
- boulos 9y agoLosing some work in exchange for a large discount is still a good trade though. It's a non-goal to provide 100% efficiency :).
- argonaut 9y agoMy point is, you would have to write a nontrivial amount of code to get this working. It would not work out of the box with Keras right now, for example.
- boulos 9y agoI think you skipped over my first comment. This is built into Keras (and was the second result on Google for 'keras checkpoint'): https://keras.io/callbacks/#example-model-checkpoints https://keras.io/callbacks/#example-model-checkpoints I don't consider that a nontrivial amount of code to write. Am I missing something?
- argonaut 9y agoI am well aware of Keras' checkpoint code. It only supports checkpointing every epoch. So no, it does not work out of the box with Keras.