8 ms·

> Google CEO Sundar Pichai said the new TPU is eight times more powerful than last year Are we sure about this? He specifically said that a "pod" would be 8 ti

> Google CEO Sundar Pichai said the new TPU is eight times more powerful than last year Are we sure about this? He specifically said that a "pod" would be 8 times faster than last year, not the TPU itself. And the picture in the background showed what looked like to be 8 racks of 64 TPUs (or maybe 32?). Until now a "pod" was a single rack of 64 TPUs. So if the new definition for a "pod" is 8 times as many TPUs as it was before, the result is less impressive... Is there any actual spec released?

twtw 8y ago
Google started playing this game with the tpuv2, wherein they define a "TPU" as whatever they want to make it sound suitably impressive (I.e. Cloud TPU is four chips). This in turn led to nvidia calling the DGX-2 "the world's largest GPU
- jacksmith21006 8y ago
  What does it matter? What we care about is the cost. The TPU gen 2 are about half as much as using Nvidia in the Amazon cloud for the same work. To me that is what matters. If the TPU 3 further drops the cost fantastic. How many chips sounds a lot more like a pissing contest. What if it was a giant chip with actually a bunch of chips inside? Who cares?
  - deepnotderp 8y ago
    You care from a technical standpoint. But, if you insist on going by the cost metric, you've already lost because you can buy nVidia GPUs and that's a lot cheaper than any of the TPU instances :)
    - jacksmith21006 8y ago
      Cost is to do some task.
      - p1esk 8y ago
        So how does cost compare to run a task 24/7 for a year if I buy Nvidia GPUs vs rent TPUs?
        jacksmith21006 8y ago
        Would expect Nvidia GPUs run 24/7 would be far cheaper. But that is an unusual use case. Even more so with ML training. It is much more bumpy. But you have a lot of other issues. Buying is going to hurt cash flow versus renting is a big one for small companies or startups not well funded. Then also responsible for updating and stuck with old silicon. So for example we just got the TPU 3.0. So the cost will decrease. But your cost of running what you buy is static. You are not be rewarded by improvements and they are happening quickly with the TPUs for example. Just one year and we have the 3.0.
        p1esk 8y ago
        Let's see, last year I bought a 4x 1080Ti workstation for ~$6k. I agree that I haven't run it 24/7 all year long, the typical usage pattern has been training a model for 2-8 days, then it's 0-3 days idle time. The longest it ever sat idle was a week. Let's say 50% utilization rate for the last 12 months. That would be 180x24 = 4,300 hours. In a small startup, typically such workstations would be shared by multiple people, so the utilization would be higher. When I interned in a small startup people always waited to launch their experiments, so the utilization was actually close to 24/7. So, how much would 4k hours of 4x 1080Ti compute time cost me to rent? Or, to put another way, how many hours do I need to use to justify buy vs rent? Or, how many hours of 4x 1080Ti compute time can I rent for $6k?
        jacksmith21006 8y ago
        The thing is ML for training is much more bumpy. You need further capacity at certain times. But the other issue is you are stuck with the hardware. Google just put out the TPU 3.0 and you can use them without losing all your investment if you bought your own hardware. The other is buying the hardware is much harder from a cash flow standpoint.
  - Voloskaya 8y ago
    I care because assuming the cost of a single TPUv3 is roughly similar to the cost of a single TPUv2 (at launch), then paying for a "pod" of 256 or 512 (whatever a pod is this year) will be nowhere near the cost of 64 TPUv2 "pod". So I care because I want to know if I will actually see this 8x speed improvement or if this is just marketing BS. If NVIDIA tells me a 2080 is 8x time faster than a 1080, I know that a 2080 when released will be roughly the same price as the 1080 when it was released, and so I can expect to actually see a 8x cost/perf ratio improvement (if we put aside the fact that this are always best case scenarios)
    - jacksmith21006 8y ago
      You can already hook together TPUs. So, IMO, it is here or there what is happening in the data center. What I care about is how much is the cost. Time to train is more of a function of how much resources I want to use.
      - Voloskaya 8y ago
        So what? I just want to know if the 8x speed increase was achieve at roughly the same cost (i.e same numbers of TPUs) or not. If the 8x speed increase was achieve with 4x more TPUs (and roughly 4x the cost) like it seems it was, then this is just marketing bullshit, and we should really expect a 2x improvement in cost/perf ratio not 8x.
        jacksmith21006 8y ago
        I am also curious. But really it comes down to cost. Right now we see about 1/2 the cost of using TPUs versus Nvidia on the AWS cloud. Will be interesting to see if the difference grows further with the TPU 3.0 or Google will just take larger margins.
- sanxiyn 8y ago
  While DGX-2 includes 16 V100, you can't replicate DGX-2 performance with 16 V100. Much of its performance is due to NVSwitch/NVLink. I think it's same with TPU.
  - shaklee3 8y ago
    Except a DGX-2 you can buy and have a tangible item sitting next to you.
sanxiyn 8y ago
This is about scaling. Since you usually can't achieve 8x speed with 8 chips (there will be issues with interconnect, algorithm, etc.), 8x speed with 8 chips is noteworthy. Although I wouldn't describe that scaling work as "8 times more powerful" myself.
- Voloskaya 8y ago
  ~8x speed with 8 chips of the same generation would obviously be very impressive. But 8x speed with 8 chips compared to a last year's single chip is definitely not the same thing. Pretty sure you can get more than (or around) 8x speed up when comparing 8 V100 vs a single P100 for example.
- whazor 8y ago
  More chips on a board requires more cooling or less power consumption.
forgot-my-pw 8y ago
I haven't seen much info yet on TPU 3, but TPU 2 was: - 64 TPUs per pod (11.5 petaflops) - 4 chips per TPU (180 teraflops)
shaklee3 8y ago
It's 2x faster per area: https://twitter.com/soumithchintala/status/993916560472461313 https://twitter.com/soumithchintala/status/99391656047246131...

twtw 8y ago

Google started playing this game with the tpuv2, wherein they define a "TPU" as whatever they want to make it sound suitably impressive (I.e. Cloud TPU is four chips). This in turn led to nvidia calling the DGX-2 "the world's largest GPU

jacksmith21006 8y ago
What does it matter? What we care about is the cost. The TPU gen 2 are about half as much as using Nvidia in the Amazon cloud for the same work. To me that is what matters. If the TPU 3 further drops the cost fantastic. How many chips sounds a lot more like a pissing contest. What if it was a giant chip with actually a bunch of chips inside? Who cares?
- deepnotderp 8y ago
  You care from a technical standpoint. But, if you insist on going by the cost metric, you've already lost because you can buy nVidia GPUs and that's a lot cheaper than any of the TPU instances :)
  - jacksmith21006 8y ago
    Cost is to do some task.
    - p1esk 8y ago
      So how does cost compare to run a task 24/7 for a year if I buy Nvidia GPUs vs rent TPUs?
      - jacksmith21006 8y ago
        Would expect Nvidia GPUs run 24/7 would be far cheaper. But that is an unusual use case. Even more so with ML training. It is much more bumpy. But you have a lot of other issues. Buying is going to hurt cash flow versus renting is a big one for small companies or startups not well funded. Then also responsible for updating and stuck with old silicon. So for example we just got the TPU 3.0. So the cost will decrease. But your cost of running what you buy is static. You are not be rewarded by improvements and they are happening quickly with the TPUs for example. Just one year and we have the 3.0.
        p1esk 8y ago
        Let's see, last year I bought a 4x 1080Ti workstation for ~$6k. I agree that I haven't run it 24/7 all year long, the typical usage pattern has been training a model for 2-8 days, then it's 0-3 days idle time. The longest it ever sat idle was a week. Let's say 50% utilization rate for the last 12 months. That would be 180x24 = 4,300 hours. In a small startup, typically such workstations would be shared by multiple people, so the utilization would be higher. When I interned in a small startup people always waited to launch their experiments, so the utilization was actually close to 24/7. So, how much would 4k hours of 4x 1080Ti compute time cost me to rent? Or, to put another way, how many hours do I need to use to justify buy vs rent? Or, how many hours of 4x 1080Ti compute time can I rent for $6k?
        jacksmith21006 8y ago
        The thing is ML for training is much more bumpy. You need further capacity at certain times. But the other issue is you are stuck with the hardware. Google just put out the TPU 3.0 and you can use them without losing all your investment if you bought your own hardware. The other is buying the hardware is much harder from a cash flow standpoint.
- Voloskaya 8y ago
  I care because assuming the cost of a single TPUv3 is roughly similar to the cost of a single TPUv2 (at launch), then paying for a "pod" of 256 or 512 (whatever a pod is this year) will be nowhere near the cost of 64 TPUv2 "pod". So I care because I want to know if I will actually see this 8x speed improvement or if this is just marketing BS. If NVIDIA tells me a 2080 is 8x time faster than a 1080, I know that a 2080 when released will be roughly the same price as the 1080 when it was released, and so I can expect to actually see a 8x cost/perf ratio improvement (if we put aside the fact that this are always best case scenarios)
  - jacksmith21006 8y ago
    You can already hook together TPUs. So, IMO, it is here or there what is happening in the data center. What I care about is how much is the cost. Time to train is more of a function of how much resources I want to use.
    - Voloskaya 8y ago
      So what? I just want to know if the 8x speed increase was achieve at roughly the same cost (i.e same numbers of TPUs) or not. If the 8x speed increase was achieve with 4x more TPUs (and roughly 4x the cost) like it seems it was, then this is just marketing bullshit, and we should really expect a 2x improvement in cost/perf ratio not 8x.
      - jacksmith21006 8y ago
        I am also curious. But really it comes down to cost. Right now we see about 1/2 the cost of using TPUs versus Nvidia on the AWS cloud. Will be interesting to see if the difference grows further with the TPU 3.0 or Google will just take larger margins.
sanxiyn 8y ago
While DGX-2 includes 16 V100, you can't replicate DGX-2 performance with 16 V100. Much of its performance is due to NVSwitch/NVLink. I think it's same with TPU.
- shaklee3 8y ago
  Except a DGX-2 you can buy and have a tangible item sitting next to you.

sanxiyn 8y ago

This is about scaling. Since you usually can't achieve 8x speed with 8 chips (there will be issues with interconnect, algorithm, etc.), 8x speed with 8 chips is noteworthy. Although I wouldn't describe that scaling work as "8 times more powerful" myself.

Voloskaya 8y ago
~8x speed with 8 chips of the same generation would obviously be very impressive. But 8x speed with 8 chips compared to a last year's single chip is definitely not the same thing. Pretty sure you can get more than (or around) 8x speed up when comparing 8 V100 vs a single P100 for example.
whazor 8y ago
More chips on a board requires more cooling or less power consumption.

forgot-my-pw 8y ago

I haven't seen much info yet on TPU 3, but TPU 2 was: - 64 TPUs per pod (11.5 petaflops) - 4 chips per TPU (180 teraflops)

shaklee3 8y ago

It's 2x faster per area: https://twitter.com/soumithchintala/status/993916560472461313 https://twitter.com/soumithchintala/status/99391656047246131...