3 ms·
Disclosure: I work on Google Cloud (and know the OpenAI folks). Fwiw, I think 2500 "V100-days" here is an extrapolation of ~1 2048-TPUv3 day. So approximately
by boulos 6y ago
Disclosure: I work on Google Cloud (and know the OpenAI folks).
Fwiw, I think 2500 "V100-days" here is an extrapolation of ~1 2048-TPUv3 day. So approximately $8/hr * 24 hours * 2048 => $400k, if you know, you could make use of that TPU Pod for much of the rest of the days of the year :).
But yes, this is still the realm of "Do you have millions of dollars of ML infrastructure" (if you want it quickly). I'm kind of hoping gwern et al. will use Colab to slowly train a variant of this for free :).
- forgingahead 6y agoIs there a way to guarantee a specific GPU for Colab, and also have it run in the background without the runtime disconnecting? Even Colab Pro doesn't seem to guarantee a specific resource without the disconnects.
- bjourne 6y agoNope. You need to have it open in a browser tab and to keep it from disconnecting you run this script: function ConnectButton(){document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click(); console.log("Connect pushed"); } setInterval(ConnectButton,60000); :) You still get booted every 6 hour or so.
- gwern 6y agoWe've moved up in the world! These days we'd train on one of our TFRC TPU pods. Colab is a lot of trouble and better avoided... (We actually can create a TPU-2048 pod. For a short while, anyway, before it preempts. But we wouldn't need to if we used lucidrain's efficient attention implementations I mention in my other comment.)
- boulos 6y agoOh, I missed that! Fwiw, I’m a big proponent of 1/4-pod slices anyway :).