3 ms·

I really don't get how they came up with those numbers comparing CPUs to GPUs. They claim to have 3.5x as much on-chip memory as a GPU, but the R9 Fury X has 1

I really don't get how they came up with those numbers comparing CPUs to GPUs. They claim to have 3.5x as much on-chip memory as a GPU, but the R9 Fury X has 16.7 MiB of register memory compared to their 28MiB. And then of course there's caches on top of that (which funnily add up to less than the register memory, I believe). I also don't get how they come up with those MAC numbers. An RX Vega 64 can do 27 TFlop/s of half-precision arithmetic, which is way more than 1/25x the 92 TOp/s they claim for the TPU. In fact, it makes the GPU look pretty damn good, considering the TPU only does 8-bit ops. Of course I'd expect the TPU to beat a GPU in terms of perf/watt, but that's not what they're comparing on that particular slide. There's the whole question of how you manage latency in inference, but then I'd expect them to talk about the utilization of the GPU resources relative to the theoretical peak.

aub3bhat 9y ago
Floating point operations (TFlop) != Tensor Operations (TOp)
- nhaehnle 9y ago
  Sure, but read the slide. They have 64k multiply-accumulate units, running at 700 MHz. That means ~46T/s multiply-accumulates, which means ~92T/s individual arithmetic ops. It's a standard way to measure this. I think it's fair to say that 92T/s 8-bit arithmetic ops is much less than 25x the 27T/s half-float operations of a GPU.
  - deleted 9y ago
    [deleted]
  - Symmetry 9y ago
    If an 8-bit integer is sufficient for a problem there isn't anything to be gained by moving to 32 bit floats. You can't just use a 32-bit floating point operation to emulate 4 8-bit integer operations for free (or vice versa) so you just can't compare the two the way you're trying to. Especially since moving to larger precision values would balloon memory and bandwidth requirements. For an honest comparison find out what the 8-bit integer performance of the GPU is.
    - Dylan16807 9y ago
      > you just can't compare the two the way you're trying to I don't think that accusation is justified. The part about float operations being better is only a side note. The core of the comment is that they are not inferior. If you needed to, you could snip wires to turn that half-float unit into an 8 bit unit. So treat the numbers as if they were the same thing. 27 vs. 92. That's not a 25x increase. Not even close. Something about this comparison seems either unfair or misleading. For example if a GPU doesn't engage most of its ALUs for certain sizes of input (cough GP10x cough), that's not a point in favor of the google design, that's just the GPU being broken.
      - Symmetry 9y ago
        They aren't inferior but you can't just multiply by 4 here either. You could turn a int32 adder into 4 int8 addres if the larger adder works on a ripple-carry principle but really everybody uses ripple-carry or carry-bypass. And a float32 is more complex and you could get 3 int8s out of it but you'd have lots of transistors left over in the execution logic. But simple quantity of execution logic is almost never an interesting constraint in a design. But the actually important part here is that the register and bypass networks to pass 4 bits of int8 data around are way more complicated than those required to pass a single float32 around and that's where Google's decision to restrict the flexibility of its TPU pays big dividends. NVidia's GPUs do not have broken designs. They're just making compromises based on the need to handle a wider variety of use cases.
        Dylan16807 9y ago
        > They aren't inferior but you can't just multiply by 4 here either. Yeah but there wasn't a suggestion to do so. Just by raw count there are issues with 25x. > NVidia's GPUs do not have broken designs. The part where the current generation sticks in a single FP16x2 unit per 128 FP32 units, so that if your code triggers them it runs 64x slower on FP16 while leaving all the FP32 units idle? That's broken as far as I can see, there to upsell you the pro cards. Anything that would make 8 bit math slower than 32 bit math is just a fundamental lack of forethought. It's not preferred by GPU design, and shouldn't be used as a point against GPUs in general.
jsnell 9y ago
It's an old chip, it needs to be compared to the 2015 competition rather than the current one. While it's not on the slides, the notes suggest that the CPU is a Haswell and the GPU a Tesla K80. Also, is that 25x claim really about the rate of operations? It reads to me like they're talking about the number of execution units.
puzzle 9y ago
They compared TPU v1 with server hardware available at the time (2015, i.e. K80 and Haswell). Are the cards you mention from 2015? Are they for gaming or servers? Do they use ECC? If not, that rules them out right away. You can find the paper with the methodology, theoretical peaks and latency management at https://arxiv.org/abs/1704.04760 https://arxiv.org/abs/1704.04760
- buildbot 9y ago
  I would argue ECC is completely pointless during neural network inference, the amount of change a single bit flip, even a sign change, is likely to generate is minimal.
  - vomjom 9y ago
    An exponent bit flip, especially at later layers, would completely break inference.
    - puzzle 9y ago
      I'd be concerned about code, too. It's not unheard of for a single bit flip to make petabytes go away (true story).
- nhaehnle 9y ago
  Fair enough. The first one, for the memory comparison, is indeed from 2015. The second one isn't -- 2015 desktop/server GPUs didn't have good half-float operations yet, as that hadn't really been a market. However, the first-mentioned GPU from 2015 has 8.6 single-precision TFlop/s, which is also more than 2x higher than their comparison baseline for GPUs. The gaming/server and especially the ECC thing is pretty moot. First, while I'm not sure what kind of server SKUs were available at the time, it hardly affects architectural results. Second, even market availability shouldn't matter much. They're Google. If they had wanted different SKUs in volume, they almost certainly could have gotten them. I mean, it's clear that a special-purpose chip is going to beat anything else at a task like this. It's just odd that they apparently felt the need to make themselves look better than they really are when the result is impressive enough with a real comparison.

aub3bhat 9y ago

Floating point operations (TFlop) != Tensor Operations (TOp)

nhaehnle 9y ago
Sure, but read the slide. They have 64k multiply-accumulate units, running at 700 MHz. That means ~46T/s multiply-accumulates, which means ~92T/s individual arithmetic ops. It's a standard way to measure this. I think it's fair to say that 92T/s 8-bit arithmetic ops is much less than 25x the 27T/s half-float operations of a GPU.
- deleted 9y ago
  [deleted]
- Symmetry 9y ago
  If an 8-bit integer is sufficient for a problem there isn't anything to be gained by moving to 32 bit floats. You can't just use a 32-bit floating point operation to emulate 4 8-bit integer operations for free (or vice versa) so you just can't compare the two the way you're trying to. Especially since moving to larger precision values would balloon memory and bandwidth requirements. For an honest comparison find out what the 8-bit integer performance of the GPU is.
  - Dylan16807 9y ago
    > you just can't compare the two the way you're trying to I don't think that accusation is justified. The part about float operations being better is only a side note. The core of the comment is that they are not inferior. If you needed to, you could snip wires to turn that half-float unit into an 8 bit unit. So treat the numbers as if they were the same thing. 27 vs. 92. That's not a 25x increase. Not even close. Something about this comparison seems either unfair or misleading. For example if a GPU doesn't engage most of its ALUs for certain sizes of input (cough GP10x cough), that's not a point in favor of the google design, that's just the GPU being broken.
    - Symmetry 9y ago
      They aren't inferior but you can't just multiply by 4 here either. You could turn a int32 adder into 4 int8 addres if the larger adder works on a ripple-carry principle but really everybody uses ripple-carry or carry-bypass. And a float32 is more complex and you could get 3 int8s out of it but you'd have lots of transistors left over in the execution logic. But simple quantity of execution logic is almost never an interesting constraint in a design. But the actually important part here is that the register and bypass networks to pass 4 bits of int8 data around are way more complicated than those required to pass a single float32 around and that's where Google's decision to restrict the flexibility of its TPU pays big dividends. NVidia's GPUs do not have broken designs. They're just making compromises based on the need to handle a wider variety of use cases.
      - Dylan16807 9y ago
        > They aren't inferior but you can't just multiply by 4 here either. Yeah but there wasn't a suggestion to do so. Just by raw count there are issues with 25x. > NVidia's GPUs do not have broken designs. The part where the current generation sticks in a single FP16x2 unit per 128 FP32 units, so that if your code triggers them it runs 64x slower on FP16 while leaving all the FP32 units idle? That's broken as far as I can see, there to upsell you the pro cards. Anything that would make 8 bit math slower than 32 bit math is just a fundamental lack of forethought. It's not preferred by GPU design, and shouldn't be used as a point against GPUs in general.

jsnell 9y ago

It's an old chip, it needs to be compared to the 2015 competition rather than the current one. While it's not on the slides, the notes suggest that the CPU is a Haswell and the GPU a Tesla K80. Also, is that 25x claim really about the rate of operations? It reads to me like they're talking about the number of execution units.

puzzle 9y ago

They compared TPU v1 with server hardware available at the time (2015, i.e. K80 and Haswell). Are the cards you mention from 2015? Are they for gaming or servers? Do they use ECC? If not, that rules them out right away. You can find the paper with the methodology, theoretical peaks and latency management at https://arxiv.org/abs/1704.04760 https://arxiv.org/abs/1704.04760

buildbot 9y ago
I would argue ECC is completely pointless during neural network inference, the amount of change a single bit flip, even a sign change, is likely to generate is minimal.
- vomjom 9y ago
  An exponent bit flip, especially at later layers, would completely break inference.
  - puzzle 9y ago
    I'd be concerned about code, too. It's not unheard of for a single bit flip to make petabytes go away (true story).
nhaehnle 9y ago
Fair enough. The first one, for the memory comparison, is indeed from 2015. The second one isn't -- 2015 desktop/server GPUs didn't have good half-float operations yet, as that hadn't really been a market. However, the first-mentioned GPU from 2015 has 8.6 single-precision TFlop/s, which is also more than 2x higher than their comparison baseline for GPUs. The gaming/server and especially the ECC thing is pretty moot. First, while I'm not sure what kind of server SKUs were available at the time, it hardly affects architectural results. Second, even market availability shouldn't matter much. They're Google. If they had wanted different SKUs in volume, they almost certainly could have gotten them. I mean, it's clear that a special-purpose chip is going to beat anything else at a task like this. It's just odd that they apparently felt the need to make themselves look better than they really are when the result is impressive enough with a real comparison.