Meh, not sure how much more there is to do to specialize for transformer specifically. TPU and GPU are mainly just fantastic matrix multipliers. And transformer

Meh, not sure how much more there is to do to specialize for transformer specifically. TPU and GPU are mainly just fantastic matrix multipliers. And transformer is partly designed with this hardware in mind: the operations are basically the same operations you see in a CNN. And in fact, one of the nice parts of the transformer is that you can run it without RNN, making it even better optimized for the matrix multipliers. Furthermore, tpus are a moving target themselves: as ML needs change, the team build new operations and optimizations into the next generation of chips.