4 ms·
Have model training make checkpoints every five minutes. When the training gets interrupted, just restart it on a new instance from the last checkpoint. Repeat
by PeterisP 7y ago
Have model training make checkpoints every five minutes. When the training gets interrupted, just restart it on a new instance from the last checkpoint. Repeat as necessary until hundreds of hours of training is completed.
- eru 7y agoYes, exactly. To add one more detail: mount your Google Drive and save the checkpoint there.
- PeterisP 7y agoCloud storage bucket works better than Google Drive for both training data and checkpoints; and if you're doing training on TPUs, then I believe that you must do the data exchange via cloud storage.
- eru 7y agoOK, that also works. Just don't rely solely on the storage that comes with your notebook instance.