3 ms·
I'm curious how (absolutely) efficient the Transformer training is even on the TPUs. The results from self-attention models are really impressive but unfortunat
by gok 7y ago
I'm curious how (absolutely) efficient the Transformer training is even on the TPUs. The results from self-attention models are really impressive but unfortunately their topology makes them very difficult to implement efficiently in silicon. It often becomes purely a question of memory bandwidth, because you're not doing much math per weight on each iteration. I wonder if the speedup is from the use of on-chip HBM in the TPUs.
- rrss 7y agoYou mean this speedup? > 1024 TPUs are twice as fast as 480 GPUs Might it be because there are twice as many?
- zak 7y agoAuthor of the blog post here. We submitted multiple results using various Cloud TPU v3 Pod slice sizes to show the current achievable Transformer training efficiency at several scales: https://mlperf.org/training-results-0-6 https://mlperf.org/training-results-0-6 We're actively improving the whole TPU software stack, so training efficiency is likely to continue to increase over time.
- gok 7y agoWhat I'm really asking is: how much effective compute throughput are you able to get during Transformer training relative to the amount of theoretical raw compute available?