7 ms·
Cloud TPU Pods Break AI Training Records
- ArthurBrussee 7y agoIs it me or are those results somewhat underwhelming if anything? Dedicated hardware for a 2x speedup at best, tossup for most results, and only competes in some categories. Not to be just a NVIDIA fan here, surely there is value in dedicated training hardware, but just surprising that benefit isn't bigger!
- totoglazer 7y agoI think it’s probably because the benchmark isn’t optimized for TPU Pods. Check out the BERT in 76 minutes paper for how you need to rethink the training regime to take advantage of pods.
- zak 7y agoYes, Cloud TPU Pods are designed to train much larger models on much larger datasets. And, as you mention, if you are willing to adjust your model architectures and training algorithms to take full advantage of the hardware, you can sometimes achieve substantial gains.
- boulos 7y agoDisclosure: I work on Google Cloud (even with Zak sometimes). The DGX-2h is a beast! Don’t parse this as “huh, TPU Pods are about the same as just a few V100s”. The data sheet [1] is probably the easiest to follow, but their writeup is more informative [2]. These are souped up V100s, with awesome networking, which is pretty similar in style to a TPU Pod. So I’d say that they’re both purpose built systems for distributed ML training. The name for the NVIDIA system is even “DGX SuperPOD” :). [1] https://www.nvidia.com/content/dam/en-zz/es_em/Solutions/Data-Center/dgx-2/nvidia-dgx-2h-datasheet.pdf https://www.nvidia.com/content/dam/en-zz/es_em/Solutions/Dat... [2] https://devblogs.nvidia.com/dgx-superpod-world-record-supercomputing-enterprise/ https://devblogs.nvidia.com/dgx-superpod-world-record-superc...
- zak 7y agoAuthor of the blog post here. As mentioned in other comments, I'd recommend doing a performance-per-dollar comparison in addition to looking at this pure performance comparison at maximum scale.
- daxfohl 7y agoWhat's in the picture? Can't figure out the scale. Are those like server racks or breadboards or ...?
- smartinspereira 7y agoServer racks. The colored things are all cables.
- Klinky 7y agoHere's a higher resolution version: https://storage.googleapis.com/gweb-cloudblog-publish/original_images/TPU_V3_POD_FULLFRONT_FORWEBONLY_FINAL.jpg https://storage.googleapis.com/gweb-cloudblog-publish/origin...
- morphle 7y agoAll you see is 8 server racks with colored network cables, switches and power supplies. The TPU ASICs themselves make up just a tiny part of this datacenter, you also have the printed circuit boards, cooling fins, 8x48 metal boxes, power and network cables, DC/DC or AC/DC converters at the bottom, fans or water tubes for cooling and airgaps. My startup is trying to develop wafer scale integration where you collapse 2 racks of the network, metal boxes, power and cooling into a 300mm wafer immersed a 100 mm x 400mm box with a few fibers and three power cables coming out. That can save around 900% of the capital cost and orders of magnitude of power (especially if you put the box in a building where the waste heat is not wasted but used to heat water for showering and space heating). As it is now, these datacenter customers and hyperscalers don't seem to care about the enormous waste and cost of paying for inefficient hardware. Considering the enormous cost and carbon emmission savings a wafer scale integration would bring (and the competitive advantage), you would be suprised how hard it is to get funding from them to develop it. I suspect is also the main reason they don't care to publish normalized benchmarks for $/performance/joule, as it would demonstrate how wasteful it all is.
- rrss 7y ago> would be suprised how hard it is to get funding from them to develop it. That's probably because there has been no evidence that wafer scale integration can actually work.
- rrss 7y agoIt's disappointing there is no attempt at normalizing results. It's not perf/$, or perf/W, or perf/chip, or anything that might be useful - it seems to just be perf/(the largest machine google/nvidia could afford to put together for a given benchmark). Seriously. Transformer: 1024 TPUs are twice as fast as 480 GPUs. Resnet50: 1536 GPUs are about as fast as 1024 TPUs. SSD: 1024 TPUs are twice as fast as 240 GPUs. Great.
- aey 7y agoResults are disappointing. 2x speed up over general purpose GPUs doesn’t justify a whole new hardware architecture.
- anonuser123456 7y agoWhen Moores law is finally dead and buried (e.g. 5nm), new architectures will be all that's left. Seems like a great time to start down the new architecture path.
- deleted 7y ago[deleted]
- rrss 7y agoHave semiconductor companies not been on the new architecture path already? It's not like they've just been doing die shrinks this whole time.
- anonuser123456 7y agoYes, but the emphasis now is much greater than before. The old rules of turn the scaling crank that defined the industry for decades are no longer helping.
- zak 7y agoAuthor of the blog post here. I'd recommend doing a performance-per-dollar comparison before drawing this conclusion.
- gok 7y agoI'm curious how (absolutely) efficient the Transformer training is even on the TPUs. The results from self-attention models are really impressive but unfortunately their topology makes them very difficult to implement efficiently in silicon. It often becomes purely a question of memory bandwidth, because you're not doing much math per weight on each iteration. I wonder if the speedup is from the use of on-chip HBM in the TPUs.
- rrss 7y agoYou mean this speedup? > 1024 TPUs are twice as fast as 480 GPUs Might it be because there are twice as many?
- zak 7y agoAuthor of the blog post here. We submitted multiple results using various Cloud TPU v3 Pod slice sizes to show the current achievable Transformer training efficiency at several scales: https://mlperf.org/training-results-0-6 https://mlperf.org/training-results-0-6 We're actively improving the whole TPU software stack, so training efficiency is likely to continue to increase over time.
- gok 7y agoWhat I'm really asking is: how much effective compute throughput are you able to get during Transformer training relative to the amount of theoretical raw compute available?