3 ms·
Has anyone here actually tried to use preemptible GPUs on GCE? You're lucky if you can get the VM to stay up for more than 10 minutes. Litany of failure here:
by sillysaurusx 7y ago
Has anyone here actually tried to use preemptible GPUs on GCE? You're lucky if you can get the VM to stay up for more than 10 minutes.
Litany of failure here: https://twitter.com/theshawwn/status/1174480402779648000 https://twitter.com/theshawwn/status/1174480402779648000
Perhaps I just got unlucky and the situation is improved now. But it was a waste of a day for me.
- minimaxir 7y agoI haven't in awhile; when I used them more heavily, I got the full 24 hours. It's possible that it's region-dependent; IIRC I had better luck with us-east1 than us-central1. I should test more at some point.
- boulos 7y agoDisclosure: I work on Google Cloud. Sorry about that. We (I) had screwed up and allowed users to smash themselves into the wall of "Huh, you keep getting preempted, but there's technically a slot right here". We've re-enabled some backpressure to keep you from getting hit over and over again, and instead see a stockout error when you should probably try another (less full) zone.
- sillysaurusx 7y agoHey, neat! Thanks for your work. I was hoping that this was the case, and I haven't tried since September. But I'll give it another shot now.
- boulos 7y agoFeel free to send me an email (contact info in profile), if you want to go back and forth.
- dwiel 7y agoI killed a lot of time with this exact issue a couple months ago as well. Quite frustrating, and it was even worse trying to include the preemptible GPUs in a kubernetes cluster. It would often take so long to add the GPU to the cluster that it would get killed before it even ran any real workload. This cycle would repeat for an hour before a machine stayed up long enough to actually schedule kubernetes pods on them.
- dannyw 7y agoIt would be nice of GCP didn't bill for pre-emptible instances that got system killed within 10 minutes of starting; or something reasonable like that. I doubt it makes a material difference to their (or our) bottom line, but it's just a nice gesture and makes you never feel like "damn, I literally wasted money".
- dannyw 7y agoWhile this is pretty frustrating and seemed to be the result of a bug, I would personally only use preemptible instances are when I have everything scripted, and save the state of my work when you get a termination signal. To me, they're not really sold as "cheaper, just not 100% guaranteed"; they're sold as "cheap, it will be killed at any moments notice so".