They seem roughly comprable. If you can get 85% scale out efficiency, based on the numbers in this post, 256 V100s would be at parity with 256 TPUs. In practice

They seem roughly comprable. If you can get 85% scale out efficiency, based on the numbers in this post, 256 V100s would be at parity with 256 TPUs. In practice, you can get 90% scale out on Resnet with last year’s GPU and networking hardware. Network bandwidth is the bottleneck, and with better network hardware that could be improved. Maybe that’s why Google didn’t publish a benchmark for 256 V100s. I’m also curious what the is the performance of the network interconnects in the TPU pod. I couldn’t find it documented, though I didn’t look super hard. [1] https://github.com/uber/horovod/blob/master/docs/benchmarks.md https://github.com/uber/horovod/blob/master/docs/benchmarks....