4 ms·

Can you elaborate on this?

• PeterisP 7y ago
  Have model training make checkpoints every five minutes. When the training gets interrupted, just restart it on a new instance from the last checkpoint. Repeat as necessary until hundreds of hours of training is completed.
  • eru 7y ago
    Yes, exactly. To add one more detail: mount your Google Drive and save the checkpoint there.
    • PeterisP 7y ago
      Cloud storage bucket works better than Google Drive for both training data and checkpoints; and if you're doing training on TPUs, then I believe that you must do the data exchange via cloud storage.
      • eru 7y ago
        OK, that also works. Just don't rely solely on the storage that comes with your notebook instance.