4 ms·

Perf/W, the official metric of slow but efficient processors. How many times must we go down this road? Let's see this sucker train AlexNet...

dgacmu 10y ago
Wearing my CMU hat for a moment (but keeping in mind Google's paying me this year): Google's always been cautious about the balance of speed and efficiency, out of concerns about programmer productivity, parallelization, and generality. See, for example, Urs's article in response to my and a few other people's crazy-academic research on using "Wimpy" nodes: http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36448.pdf http://static.googleusercontent.com/media/research.google.co... - vs http://www.cs.cmu.edu/~fawnproj/ http://www.cs.cmu.edu/~fawnproj/ There's a big difference between just cranking down the GHz and going for ASIC specialization. GPUs, for example, already represent a point on this spectrum -- it's true that they run at reduced GHz compared to high-end CPUs, but they're arithmetic monsters. The blog post notes, in fact, that the use of TPUs in AlphaGo let them do more searching. So why would you assume automatically that they're slow?
- LogicFailsMe 10y ago
  Because there's an absolute sippy straw of bandwidth to the thing if that's a 1x pci-e connection. For if it were delivering performance on par with a $1000 Maxwell class GPU, why wouldn't you guys crow about it? That would be a really big deal wouldn't it? TitanX for 20W? That'd be awesome. And having suffered through multiple pitches for us to buy various FPGA and boutique processors, I have yet to see someone who produced perf per watt numbers first, subsequently produce an impressive performance number. In fact, it took nearly yelling at one vendor for them to finally admit perf was abysmal. Finally, training does not equal inference. Training requires strong scaling, but inference need only weak scale. So I suspect that Urs had to bite his tongue and buy a bunch of gpus for training networks. Am I missing something?
  - dgacmu 10y ago
    That wasn't really my point - I'm simply noting that Urs is one of the last people I'd think to hop on the wimpy crazy train. His published articles suggest that he's got a very good grasp of the tradeoffs involved in "real" TCO -- i.e., taking a fairly global view of both the human, capital, and operating expenses involved in a technology decision such as using wimpies (no) or fabricating a custom ASIC for machine learning (yes). That doesn't mean a TPU is faster or slower than anything in particular, it just means that quite likely that it's good for some machine learning tasks that Google cares enough about to spend the whatever dollars it cost to make the thing. The WSJ article has a few more quotes from Norm Jouppi, btw.: http://www.wsj.com/articles/google-isnt-playing-games-with-new-chip-1463597820 http://www.wsj.com/articles/google-isnt-playing-games-with-n... (Sorry if that gets paywalled. Googling "wall street journal google tensor processing unit" got me there.)
    - LogicFailsMe 10y ago
      There's a huge gap between wimpy and GPU, no? I'd guess this chip was right-sized to run inference on a 10 Gbit/s feed, optimized for perf/W.
  - Symmetry 10y ago
    Hmm, I'd expect that once you've got your weights loaded machine learning would need much less bandwidth/flop than graphics does. Is that incorrect?
    - LogicFailsMe 10y ago
      Apples and oranges. And as it turned out, it's an 8-bit fixed point processor. Useless for training without heroic handholding, limited use for inference. Google's PR machine does it again.

dgacmu 10y ago

Wearing my CMU hat for a moment (but keeping in mind Google's paying me this year): Google's always been cautious about the balance of speed and efficiency, out of concerns about programmer productivity, parallelization, and generality. See, for example, Urs's article in response to my and a few other people's crazy-academic research on using "Wimpy" nodes: http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36448.pdf http://static.googleusercontent.com/media/research.google.co... - vs http://www.cs.cmu.edu/~fawnproj/ http://www.cs.cmu.edu/~fawnproj/ There's a big difference between just cranking down the GHz and going for ASIC specialization. GPUs, for example, already represent a point on this spectrum -- it's true that they run at reduced GHz compared to high-end CPUs, but they're arithmetic monsters. The blog post notes, in fact, that the use of TPUs in AlphaGo let them do more searching. So why would you assume automatically that they're slow?

LogicFailsMe 10y ago
Because there's an absolute sippy straw of bandwidth to the thing if that's a 1x pci-e connection. For if it were delivering performance on par with a $1000 Maxwell class GPU, why wouldn't you guys crow about it? That would be a really big deal wouldn't it? TitanX for 20W? That'd be awesome. And having suffered through multiple pitches for us to buy various FPGA and boutique processors, I have yet to see someone who produced perf per watt numbers first, subsequently produce an impressive performance number. In fact, it took nearly yelling at one vendor for them to finally admit perf was abysmal. Finally, training does not equal inference. Training requires strong scaling, but inference need only weak scale. So I suspect that Urs had to bite his tongue and buy a bunch of gpus for training networks. Am I missing something?
- dgacmu 10y ago
  That wasn't really my point - I'm simply noting that Urs is one of the last people I'd think to hop on the wimpy crazy train. His published articles suggest that he's got a very good grasp of the tradeoffs involved in "real" TCO -- i.e., taking a fairly global view of both the human, capital, and operating expenses involved in a technology decision such as using wimpies (no) or fabricating a custom ASIC for machine learning (yes). That doesn't mean a TPU is faster or slower than anything in particular, it just means that quite likely that it's good for some machine learning tasks that Google cares enough about to spend the whatever dollars it cost to make the thing. The WSJ article has a few more quotes from Norm Jouppi, btw.: http://www.wsj.com/articles/google-isnt-playing-games-with-new-chip-1463597820 http://www.wsj.com/articles/google-isnt-playing-games-with-n... (Sorry if that gets paywalled. Googling "wall street journal google tensor processing unit" got me there.)
  - LogicFailsMe 10y ago
    There's a huge gap between wimpy and GPU, no? I'd guess this chip was right-sized to run inference on a 10 Gbit/s feed, optimized for perf/W.
- Symmetry 10y ago
  Hmm, I'd expect that once you've got your weights loaded machine learning would need much less bandwidth/flop than graphics does. Is that incorrect?
  - LogicFailsMe 10y ago
    Apples and oranges. And as it turned out, it's an 8-bit fixed point processor. Useless for training without heroic handholding, limited use for inference. Google's PR machine does it again.