The key problem with the TPU architecture for training is that its on-die buffer space is fairly limited and its memory bandwidth is currently nowhere near that

The key problem with the TPU architecture for training is that its on-die buffer space is fairly limited and its memory bandwidth is currently nowhere near that of a GPU. This matters if you've got a lot of intermediate buffers that need to be alive at the same time for backpropagation. Also, there's the less fundamental matter of convenience: while there are some clever ways to train with limited precision gradients, it's not yet as easy as not doing that. One style of training that would be a good fit for the current TPU architecture is some of the recent work on reinforcement learning without gradients: https://arxiv.org/abs/1703.03864 https://arxiv.org/abs/1703.03864