6 ms·
The authors trained ResNet-50 in 7.3 minutes at 75.3% accuracy. As a comparison, a Google TPUv3 pod with 1024 chips got to 75.2% accuracy with ResNet-50 in 1.8
by frankchn 8y ago
The authors trained ResNet-50 in 7.3 minutes at 75.3% accuracy.
As a comparison, a Google TPUv3 pod with 1024 chips got to 75.2% accuracy with ResNet-50 in 1.8 minutes, and 76.2% accuracy in 2.2 minutes with an optimizer change and distributed batch normalization [1].
[1]: https://arxiv.org/abs/1811.06992 https://arxiv.org/abs/1811.06992
- gpm 8y agoA TPUv3 pod is ~107 petaflops (Googles number from your paper). 512 Volta GPUs is ~64 petaflops (Nvidias number from [1]). v3 pods don't seem to be publicly available. A 256 chip 11.5 petaflop v2 pod is $384 per hour, $3.366 million per year. [2] Meanwhile Google Cloud Volta GPU prices (which are probably inflated over building your own cluster, but are hopefully close enough to a reasonable ballpark) are $1.736 per hour, would be $7.791 million per year for 512. Unless Google GPU prices are really inflated, clusters are legitimately substantially cheaper than cloud GPUs, or these researchers did a poor job, it seems like this is a good advertisement for TPUs. [1] https://www.nvidia.com/en-us/data-center/volta-gpu-architecture/ https://www.nvidia.com/en-us/data-center/volta-gpu-architect... [2] Pod availability / performance / pricing information here: https://cloud.google.com/tpu/ https://cloud.google.com/tpu/ [3] GPU pricing info: https://cloud.google.com/gpu/ https://cloud.google.com/gpu/