I believe that Nvidia uses the same chip from A16 up to the A100, and maybe for some of the Quadro chips. That easily puts the unit count into the several mill

I believe that Nvidia uses the same chip from A16 up to the A100, and maybe for some of the Quadro chips. That easily puts the unit count into the several millions. The picture in this article shows 8 racks with (according to the paper on arxiv) has 16 TPU sleds each, 4 TPUs per sled. that's only 512 chips. According to the paper, it is one of eight in a 4096-chip supercomputer. If you give them 10-100 of those around the world, you get 40,000-400,000 chips. That's enough for reasonable scale. Nvidia should still have 100x (or more) their scale.