3 ms·
Disclosure: I work on Google Cloud (and helped launch this). Actually both Keras and raw TF have a check point thing built in (as does Torch). I believe it can
by boulos 9y ago
Disclosure: I work on Google Cloud (and helped launch this).
Actually both Keras and raw TF have a check point thing built in (as does Torch). I believe it can be setup to do so every epoch, but unless you're talking about many many GBs of parameters, you can also stream lots of data to GCS in 30s.
- argonaut 9y agoYou're talking about support for checkpointing every epoch, but a 30 second shutoff would require checkpointing every N mini-batches, which is not a typical use for checkpointing.
- boulos 9y agoLosing some work in exchange for a large discount is still a good trade though. It's a non-goal to provide 100% efficiency :).
- argonaut 9y agoMy point is, you would have to write a nontrivial amount of code to get this working. It would not work out of the box with Keras right now, for example.
- boulos 9y agoI think you skipped over my first comment. This is built into Keras (and was the second result on Google for 'keras checkpoint'): https://keras.io/callbacks/#example-model-checkpoints https://keras.io/callbacks/#example-model-checkpoints I don't consider that a nontrivial amount of code to write. Am I missing something?
- argonaut 9y agoI am well aware of Keras' checkpoint code. It only supports checkpointing every epoch. So no, it does not work out of the box with Keras.