7 ms·
To clarify, does that mean you can continuously train a model for e.g. hundred of hours or are you still limited by the 24 hours notebook limit?
by ampdepolymerase 7y ago
To clarify, does that mean you can continuously train a model for e.g. hundred of hours or are you still limited by the 24 hours notebook limit?
- anidh 7y agoThe page said "Longer running notebooks and fewer idle timeouts mean you disconnect less often." I'm guessing there is a limit but it has been increased for the pro customers. In contrast on GCE you can train for 22 hours if you want.
- piadodjanho 7y agoThe page says its twice the traditional Cloud.
- eru 7y agoYou can work around that by saving your model and reloading.
- ampdepolymerase 7y agoCan you elaborate on this?
- PeterisP 7y agoHave model training make checkpoints every five minutes. When the training gets interrupted, just restart it on a new instance from the last checkpoint. Repeat as necessary until hundreds of hours of training is completed.
- eru 7y agoYes, exactly. To add one more detail: mount your Google Drive and save the checkpoint there.
- PeterisP 7y agoCloud storage bucket works better than Google Drive for both training data and checkpoints; and if you're doing training on TPUs, then I believe that you must do the data exchange via cloud storage.
- eru 7y agoOK, that also works. Just don't rely solely on the storage that comes with your notebook instance.