3 ms·
A preemptible P100 + VM on Google Compute Engine is about ~$0.45/hr, so to exceed that value with Colaboratory Pro (ignoring conveience factors) you'd need to t
by minimaxir 7y ago
A preemptible P100 + VM on Google Compute Engine is about ~$0.45/hr, so to exceed that value with Colaboratory Pro (ignoring conveience factors) you'd need to train for more than 22 hours in a month. Which, for deep learning, is not too unreasonable.
Reading between the lines of both the signup page and up-to-date FAQ, it seems like the free TPU in Colab notebooks will be depreciated, which isn't too surprising.
- ampdepolymerase 7y agoTo clarify, does that mean you can continuously train a model for e.g. hundred of hours or are you still limited by the 24 hours notebook limit?
- anidh 7y agoThe page said "Longer running notebooks and fewer idle timeouts mean you disconnect less often." I'm guessing there is a limit but it has been increased for the pro customers. In contrast on GCE you can train for 22 hours if you want.
- piadodjanho 7y agoThe page says its twice the traditional Cloud.
- eru 7y agoYou can work around that by saving your model and reloading.
- ampdepolymerase 7y agoCan you elaborate on this?
- PeterisP 7y agoHave model training make checkpoints every five minutes. When the training gets interrupted, just restart it on a new instance from the last checkpoint. Repeat as necessary until hundreds of hours of training is completed.
- eru 7y agoYes, exactly. To add one more detail: mount your Google Drive and save the checkpoint there.
- PeterisP 7y agoCloud storage bucket works better than Google Drive for both training data and checkpoints; and if you're doing training on TPUs, then I believe that you must do the data exchange via cloud storage.
- eru 7y agoOK, that also works. Just don't rely solely on the storage that comes with your notebook instance.
- sillysaurusx 7y agoHas anyone here actually tried to use preemptible GPUs on GCE? You're lucky if you can get the VM to stay up for more than 10 minutes. Litany of failure here: https://twitter.com/theshawwn/status/1174480402779648000 https://twitter.com/theshawwn/status/1174480402779648000 Perhaps I just got unlucky and the situation is improved now. But it was a waste of a day for me.
- minimaxir 7y agoI haven't in awhile; when I used them more heavily, I got the full 24 hours. It's possible that it's region-dependent; IIRC I had better luck with us-east1 than us-central1. I should test more at some point.
- boulos 7y agoDisclosure: I work on Google Cloud. Sorry about that. We (I) had screwed up and allowed users to smash themselves into the wall of "Huh, you keep getting preempted, but there's technically a slot right here". We've re-enabled some backpressure to keep you from getting hit over and over again, and instead see a stockout error when you should probably try another (less full) zone.
- sillysaurusx 7y agoHey, neat! Thanks for your work. I was hoping that this was the case, and I haven't tried since September. But I'll give it another shot now.
- boulos 7y agoFeel free to send me an email (contact info in profile), if you want to go back and forth.
- dwiel 7y agoI killed a lot of time with this exact issue a couple months ago as well. Quite frustrating, and it was even worse trying to include the preemptible GPUs in a kubernetes cluster. It would often take so long to add the GPU to the cluster that it would get killed before it even ran any real workload. This cycle would repeat for an hour before a machine stayed up long enough to actually schedule kubernetes pods on them.