3 ms·
There's nothing about the TPU architecture that prevents use for training. Training is also done as tensor operations, and training can be done at limited preci
by paulsutter 10y ago
There's nothing about the TPU architecture that prevents use for training. Training is also done as tensor operations, and training can be done at limited precision[1]. Just because the first TPU was memory-limited doesn't mean TPUs can never be used for training.
[1] https://arxiv.org/abs/1606.06160 https://arxiv.org/abs/1606.06160
- shaklee3 10y agoSure, but it's a garbage in, garbage out thing. If you're using those in production in something like, say, google images, you wouldn't want to lose accuracy on a one-time pass. Anything can be quantized lower, but you start making the tradeoff of losing information that may affect the quality of your product later.
- pjscott 10y agoThe key problem with the TPU architecture for training is that its on-die buffer space is fairly limited and its memory bandwidth is currently nowhere near that of a GPU. This matters if you've got a lot of intermediate buffers that need to be alive at the same time for backpropagation. Also, there's the less fundamental matter of convenience: while there are some clever ways to train with limited precision gradients, it's not yet as easy as not doing that. One style of training that would be a good fit for the current TPU architecture is some of the recent work on reinforcement learning without gradients: https://arxiv.org/abs/1703.03864 https://arxiv.org/abs/1703.03864