Since TPUv2, announced in 2017: https://arstechnica.com/information-technology/2017/05/google-brings-45-teraflops-tensor-flow-processors-to-its-compute-cloud/ h

Since TPUv2, announced in 2017: https://arstechnica.com/information-technology/2017/05/google-brings-45-teraflops-tensor-flow-processors-to-its-compute-cloud/ https://arstechnica.com/information-technology/2017/05/googl... The superscalers are all working on this. https://aws.amazon.com/ai/machine-learning/trainium/ https://aws.amazon.com/ai/machine-learning/trainium/