5 ms·

Apparently this takes "2500-V100 days" which is an insane amount of resources for images of this resolution. For context, this is equivalent to 100 $10,000 GPU

Apparently this takes "2500-V100 days" which is an insane amount of resources for images of this resolution. For context, this is equivalent to 100 $10,000 GPUs running for 25 days, 24/7. https://twitter.com/jm_alexia/status/1273349716915470340?s=20 https://twitter.com/jm_alexia/status/1273349716915470340?s=2...

fxtentacle 6y ago
... for 64x64 px resolution. This is truly a brute force approach.
- akx 6y ago
  But add an AI superresolution model after that and you're off to the races!
  - gwern 6y ago
    The problem with superresolution is that by design, they focus on details and textures for upscaling, and avoid adding in any large-scale structure. So if you train it separately (ie there is no backpropagation from the final highres 224px image through the superresolution model all the way back through iGPT), you cripple the original model. There is only so much about objects and the world you can learn from 64px thumbnails, and they mention that this is probably a reason iGPT wins on the tiny images like CIFAR but loses to the semi-supervised CNNs on ImageNet: because the CNNs are compute-efficient enough that they can train at the standard 224px and see all of the details and structure that disappears at iGPT's 64px, and learn end-to-end.
gowld 6y ago
Cost to train the model (which can be used for many images, and images can be upscaled using AI): $10000 * 2500days / 5yr = $15000 hardware cost 200W * 2500day * (0.10 USD / Whr) = $1.2M in electricity
- indiandennis 6y ago
  Electricity rates are kWh, so I think it would be 120k for electricity, not 1.2 mil.
  - Scaevolus 6y ago
    $1200, off by a factor of 1000.
- sillysaurusx 6y ago
  I’ll have to do a writeup sometime explaining why these numbers aren’t as surprising as they seem. They’re also not as wasteful as they seem. Roughly, the boxes would be turned on anyway. Might as well put them to work. And yeah, it draws more power when in use (by a lot). But the data center probably isn’t paying a huge premium on top of what they would already pay for electricity. So all that’s left is feeling generally queasy about using so much electricity, rather than a practical feeling of “this is bad because it costs a lot.” Economies of scale have some dramatic effects at the high ends: I doubt the datacenter’s costs are anywhere near linear for the amount of Whr they’re consuming. And even if they are, it’s still a rounding error considering the profitability of GoogSoftBook.
  - tinco 6y ago
    At some point it becomes linear though, there is a base cost to electricity. Google builds datacenters near coal power plants specifically to get (literally) dirt cheap electricity.
    - sillysaurusx 6y ago
      This is true. But researchers are divorced from those realities. The day-to-day researchers almost never have to think like "If I run this, will it cost a ton?" Those kinds of agreements are usually made on a company basis, i.e. there's no cost to just leave a model running and see what happens. (Which is a huge advantage, by the way.)
    - 6y ago
boulos 6y ago
Disclosure: I work on Google Cloud (and know the OpenAI folks). Fwiw, I think 2500 "V100-days" here is an extrapolation of ~1 2048-TPUv3 day. So approximately $8/hr * 24 hours * 2048 => $400k, if you know, you could make use of that TPU Pod for much of the rest of the days of the year :). But yes, this is still the realm of "Do you have millions of dollars of ML infrastructure" (if you want it quickly). I'm kind of hoping gwern et al. will use Colab to slowly train a variant of this for free :).
- forgingahead 6y ago
  Is there a way to guarantee a specific GPU for Colab, and also have it run in the background without the runtime disconnecting? Even Colab Pro doesn't seem to guarantee a specific resource without the disconnects.
  - bjourne 6y ago
    Nope. You need to have it open in a browser tab and to keep it from disconnecting you run this script: function ConnectButton(){document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click(); console.log("Connect pushed"); } setInterval(ConnectButton,60000); :) You still get booted every 6 hour or so.
- gwern 6y ago
  We've moved up in the world! These days we'd train on one of our TFRC TPU pods. Colab is a lot of trouble and better avoided... (We actually can create a TPU-2048 pod. For a short while, anyway, before it preempts. But we wouldn't need to if we used lucidrain's efficient attention implementations I mention in my other comment.)
  - boulos 6y ago
    Oh, I missed that! Fwiw, I’m a big proponent of 1/4-pod slices anyway :).

fxtentacle 6y ago

... for 64x64 px resolution. This is truly a brute force approach.

akx 6y ago
But add an AI superresolution model after that and you're off to the races!
- gwern 6y ago
  The problem with superresolution is that by design, they focus on details and textures for upscaling, and avoid adding in any large-scale structure. So if you train it separately (ie there is no backpropagation from the final highres 224px image through the superresolution model all the way back through iGPT), you cripple the original model. There is only so much about objects and the world you can learn from 64px thumbnails, and they mention that this is probably a reason iGPT wins on the tiny images like CIFAR but loses to the semi-supervised CNNs on ImageNet: because the CNNs are compute-efficient enough that they can train at the standard 224px and see all of the details and structure that disappears at iGPT's 64px, and learn end-to-end.

gowld 6y ago

Cost to train the model (which can be used for many images, and images can be upscaled using AI): $10000 * 2500days / 5yr = $15000 hardware cost 200W * 2500day * (0.10 USD / Whr) = $1.2M in electricity

indiandennis 6y ago
Electricity rates are kWh, so I think it would be 120k for electricity, not 1.2 mil.
- Scaevolus 6y ago
  $1200, off by a factor of 1000.
sillysaurusx 6y ago
I’ll have to do a writeup sometime explaining why these numbers aren’t as surprising as they seem. They’re also not as wasteful as they seem. Roughly, the boxes would be turned on anyway. Might as well put them to work. And yeah, it draws more power when in use (by a lot). But the data center probably isn’t paying a huge premium on top of what they would already pay for electricity. So all that’s left is feeling generally queasy about using so much electricity, rather than a practical feeling of “this is bad because it costs a lot.” Economies of scale have some dramatic effects at the high ends: I doubt the datacenter’s costs are anywhere near linear for the amount of Whr they’re consuming. And even if they are, it’s still a rounding error considering the profitability of GoogSoftBook.
- tinco 6y ago
  At some point it becomes linear though, there is a base cost to electricity. Google builds datacenters near coal power plants specifically to get (literally) dirt cheap electricity.
  - sillysaurusx 6y ago
    This is true. But researchers are divorced from those realities. The day-to-day researchers almost never have to think like "If I run this, will it cost a ton?" Those kinds of agreements are usually made on a company basis, i.e. there's no cost to just leave a model running and see what happens. (Which is a huge advantage, by the way.)
  - 6y ago

boulos 6y ago

Disclosure: I work on Google Cloud (and know the OpenAI folks). Fwiw, I think 2500 "V100-days" here is an extrapolation of ~1 2048-TPUv3 day. So approximately $8/hr * 24 hours * 2048 => $400k, if you know, you could make use of that TPU Pod for much of the rest of the days of the year :). But yes, this is still the realm of "Do you have millions of dollars of ML infrastructure" (if you want it quickly). I'm kind of hoping gwern et al. will use Colab to slowly train a variant of this for free :).

forgingahead 6y ago
Is there a way to guarantee a specific GPU for Colab, and also have it run in the background without the runtime disconnecting? Even Colab Pro doesn't seem to guarantee a specific resource without the disconnects.
- bjourne 6y ago
  Nope. You need to have it open in a browser tab and to keep it from disconnecting you run this script: function ConnectButton(){document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click(); console.log("Connect pushed"); } setInterval(ConnectButton,60000); :) You still get booted every 6 hour or so.
gwern 6y ago
We've moved up in the world! These days we'd train on one of our TFRC TPU pods. Colab is a lot of trouble and better avoided... (We actually can create a TPU-2048 pod. For a short while, anyway, before it preempts. But we wouldn't need to if we used lucidrain's efficient attention implementations I mention in my other comment.)
- boulos 6y ago
  Oh, I missed that! Fwiw, I’m a big proponent of 1/4-pod slices anyway :).