4 ms·

How does this compare to Nvidia GPUs on AWS price/perf-wise? The article makes it sound like this is a new thing...

bloudermilk 9y ago
Google claims[0] the TPU is many times faster for the workloads they've designed it for. > On our production AI workloads that utilize neural network inference, the TPU is 15x to 30x faster than contemporary GPUs and CPUs. As far as I know this will be the first opportunity for the public to prove those claims, as until now they've not been available on GCP. I don't mean to sound skeptical–I'm quite confident they're not exaggerating. [0]: https://cloudplatform.googleblog.com/2017/04/quantifying-the-performance-of-the-TPU-our-first-machine-learning-chip.html https://cloudplatform.googleblog.com/2017/04/quantifying-the...
- phamilton 9y ago
  I wonder how these would compare with Amazon's FPGA instances with a comparable core running.
  - deleted 9y ago
    [deleted]
- indescions_2018 9y ago
  The reserve TPU button has been available on the dashboard for the last few months. But I assume instances have been prioritized for large customers such as Two Sigma. From the paper: "Despite low utilization for some applications, the TPU is on average about 15X - 30X faster than its contemporary GPU or CPU, with TOPS/Watt about 30X - 80X higher. Moreover, using the GPU's GDDR5 memory in the TPU would triple achieved TOPS and raise TOPS/Watt to nearly 70X the GPU and 200X the CPU." In-Datacenter Performance Analysis of a Tensor Processing Unit https://arxiv.org/abs/1704.04760 https://arxiv.org/abs/1704.04760 Price is about 5x cloud nvidia gpu instance on an hourly basis.
- twtw 9y ago
  It will be interesting to see some benchmarks that compare TPUs to V100, since all previously published comparisons from Google compare TPU to K80 (3 GPU architectures ago).
- xiphias 9y ago
  One thing to keep in mind that Google was using Tensorflow for comparision, which is heavily optimized for TPU, and GPU is just a second class citizen. Of course it was a great strategy on Google's side, and TPUs perform better than GPUs, but this is a little bit of cheating in the benchmarks.
- vomjom 9y ago
  Keep in mind that what you linked refers to TPUv1, which is built for quantized 8-bit inference. The TPUv2, which was announced in this blog post, is for general purpose training and uses 32-bit weights, activations, and gradients. It will have very different performance characteristics.
  - bloudermilk 9y ago
    Thanks for pointing that out!
- ebikelaw 9y ago
  Perf per watt matters to Google but not you. You should only think of it on a perf/$ basis, right?
  - gh02t 9y ago
    They're closely related though, since if the perf per watt is lower then Google can charge you less doller per perf. The price they charge you is ultimately tied to the operating cost.
headmelted 9y ago
I would imagine that (by design) they're not directly comparable. I suspect that we'll see more information about the ASICs over time, but it'll take time to really understand their characteristics vs a Nvidia GPU - which are at least right now a bit better understood.
perfmode 9y ago
It is. TPUs perform calculations on weights using low-precision floating point and integer types. This saves a ton of computation, but doesn't matter much for training models.
- bicubic 9y ago
  But GPU is also able to use lower resolution types. There must be more to the TPU advantage.
  - Cyph0n 9y ago
    GPUs are much more complex (general-purpose) and therefore cannot be optimized beyond a certain point due to timing requirements and PVT (process, temperature, voltage) variations. In other words, the more stuff you have on an ASIC, the more careful you have to be ensure a margin of tolerance for variations.
    - saguro 9y ago
      So the only advantage of the TPU is it's a simpler and more specialized asic? Google didn't break any new ground in terms of training perf?
      - Cyph0n 9y ago
        > So the only advantage of the TPU is it's a simpler and more specialized asic And everything that entails: lower energy consumption, higher throughput, lower cost at volume, higher profits for GCP, etc. > Google didn't break any new ground in terms of training perf? Relative to GPUs, sure, but I can't say how well they stack up against other custom ASICs for DL applications.
      - dgacmu 9y ago
        So, a way to think of this is: The speed (and therefore, cost) of training a machine learning model depends on (a) the ML techniques (how rapidly the model converges and to what accuracy); and (b) how quickly the processor executes the operations involved in the ML techniques. The TPU is only an improvement in (b). It's not going to result in a big-O style speedup, because the same training algorithms and architectures will run on it that we run on CPUs & GPUs today. I'm not sure what counts as "breaking new ground" - is that 10%? 100%? 1000? :-) The things to watch out for in benchmarks will be: (a) Perf/$. This is actually a big deal - one of my students recently blew through $5000 of Google Cloud credits running Imagenet experiments, in a week. And we didn't finish them! As this cost really drops, it enables things like Neural Architecture Search, which uses tons of compute capability to explore architectural variants automatically. (b) Absolute perf. (c) Performance scaling. To what degree will the fast, 2D torroidal mesh allow a full pod of Cloud TPUs to scale nearly-linearly? Absolute training times matter from a user productivity standpoint. Waiting 30 minutes for a result is very different from waiting 12 hours (you can do one of these while you sneak out to go running! :-). The NIPS'17 slides have more technical context for some of this: https://supercomputersfordl2017.github.io/Presentations/ImageNetNewMNIST.pdf https://supercomputersfordl2017.github.io/Presentations/Imag...
briffle 9y ago
This is a new thing. Google also has Nvidia GPUs. these are new custom designed ASICs google has designed for certain ML tasks.

Google claims[0] the TPU is many times faster for the workloads they've designed it for. > On our production AI workloads that utilize neural network inference, the TPU is 15x to 30x faster than contemporary GPUs and CPUs. As far as I know this will be the first opportunity for the public to prove those claims, as until now they've not been available on GCP. I don't mean to sound skeptical–I'm quite confident they're not exaggerating. [0]: https://cloudplatform.googleblog.com/2017/04/quantifying-the-performance-of-the-TPU-our-first-machine-learning-chip.html https://cloudplatform.googleblog.com/2017/04/quantifying-the...

phamilton 9y ago
I wonder how these would compare with Amazon's FPGA instances with a comparable core running.
- deleted 9y ago
  [deleted]
indescions_2018 9y ago
The reserve TPU button has been available on the dashboard for the last few months. But I assume instances have been prioritized for large customers such as Two Sigma. From the paper: "Despite low utilization for some applications, the TPU is on average about 15X - 30X faster than its contemporary GPU or CPU, with TOPS/Watt about 30X - 80X higher. Moreover, using the GPU's GDDR5 memory in the TPU would triple achieved TOPS and raise TOPS/Watt to nearly 70X the GPU and 200X the CPU." In-Datacenter Performance Analysis of a Tensor Processing Unit https://arxiv.org/abs/1704.04760 https://arxiv.org/abs/1704.04760 Price is about 5x cloud nvidia gpu instance on an hourly basis.
twtw 9y ago
It will be interesting to see some benchmarks that compare TPUs to V100, since all previously published comparisons from Google compare TPU to K80 (3 GPU architectures ago).
xiphias 9y ago
One thing to keep in mind that Google was using Tensorflow for comparision, which is heavily optimized for TPU, and GPU is just a second class citizen. Of course it was a great strategy on Google's side, and TPUs perform better than GPUs, but this is a little bit of cheating in the benchmarks.
vomjom 9y ago
Keep in mind that what you linked refers to TPUv1, which is built for quantized 8-bit inference. The TPUv2, which was announced in this blog post, is for general purpose training and uses 32-bit weights, activations, and gradients. It will have very different performance characteristics.
- bloudermilk 9y ago
  Thanks for pointing that out!
ebikelaw 9y ago
Perf per watt matters to Google but not you. You should only think of it on a perf/$ basis, right?
- gh02t 9y ago
  They're closely related though, since if the perf per watt is lower then Google can charge you less doller per perf. The price they charge you is ultimately tied to the operating cost.

headmelted 9y ago

I would imagine that (by design) they're not directly comparable. I suspect that we'll see more information about the ASICs over time, but it'll take time to really understand their characteristics vs a Nvidia GPU - which are at least right now a bit better understood.

perfmode 9y ago

It is. TPUs perform calculations on weights using low-precision floating point and integer types. This saves a ton of computation, but doesn't matter much for training models.

bicubic 9y ago
But GPU is also able to use lower resolution types. There must be more to the TPU advantage.
- Cyph0n 9y ago
  GPUs are much more complex (general-purpose) and therefore cannot be optimized beyond a certain point due to timing requirements and PVT (process, temperature, voltage) variations. In other words, the more stuff you have on an ASIC, the more careful you have to be ensure a margin of tolerance for variations.
  - saguro 9y ago
    So the only advantage of the TPU is it's a simpler and more specialized asic? Google didn't break any new ground in terms of training perf?
    - Cyph0n 9y ago
      > So the only advantage of the TPU is it's a simpler and more specialized asic And everything that entails: lower energy consumption, higher throughput, lower cost at volume, higher profits for GCP, etc. > Google didn't break any new ground in terms of training perf? Relative to GPUs, sure, but I can't say how well they stack up against other custom ASICs for DL applications.
    - dgacmu 9y ago
      So, a way to think of this is: The speed (and therefore, cost) of training a machine learning model depends on (a) the ML techniques (how rapidly the model converges and to what accuracy); and (b) how quickly the processor executes the operations involved in the ML techniques. The TPU is only an improvement in (b). It's not going to result in a big-O style speedup, because the same training algorithms and architectures will run on it that we run on CPUs & GPUs today. I'm not sure what counts as "breaking new ground" - is that 10%? 100%? 1000? :-) The things to watch out for in benchmarks will be: (a) Perf/$. This is actually a big deal - one of my students recently blew through $5000 of Google Cloud credits running Imagenet experiments, in a week. And we didn't finish them! As this cost really drops, it enables things like Neural Architecture Search, which uses tons of compute capability to explore architectural variants automatically. (b) Absolute perf. (c) Performance scaling. To what degree will the fast, 2D torroidal mesh allow a full pod of Cloud TPUs to scale nearly-linearly? Absolute training times matter from a user productivity standpoint. Waiting 30 minutes for a result is very different from waiting 12 hours (you can do one of these while you sneak out to go running! :-). The NIPS'17 slides have more technical context for some of this: https://supercomputersfordl2017.github.io/Presentations/ImageNetNewMNIST.pdf https://supercomputersfordl2017.github.io/Presentations/Imag...

briffle 9y ago

This is a new thing. Google also has Nvidia GPUs. these are new custom designed ASICs google has designed for certain ML tasks.