> Btw, as far as I can tell, TPUs are really just a variant of GPUs with an emphasis on lower precision? GPUs and TPUs have substantially different architectur

> Btw, as far as I can tell, TPUs are really just a variant of GPUs with an emphasis on lower precision? GPUs and TPUs have substantially different architectures, it’s not accurate to consider a TPU an a variant of a GPU. See https://dl.acm.org/doi/pdf/10.1145/3360307 https://dl.acm.org/doi/pdf/10.1145/3360307, particularly section “Contrasting GPU and TPU Architectures” looking only at the matrix multiply units (and ignoring the hardware multithreading in GPUs that was deliberately not part of the TPU architecture): TPUv3: 2 cores, 2 128x128 matrix multiply units per core V100: 80 cores, 8 4x4 matrix multiply units per core