3 ms·

You mean this speedup? > 1024 TPUs are twice as fast as 480 GPUs Might it be because there are twice as many?

• zak 7y ago
  Author of the blog post here. We submitted multiple results using various Cloud TPU v3 Pod slice sizes to show the current achievable Transformer training efficiency at several scales: https://mlperf.org/training-results-0-6 https://mlperf.org/training-results-0-6 We're actively improving the whole TPU software stack, so training efficiency is likely to continue to increase over time.
  • gok 7y ago
    What I'm really asking is: how much effective compute throughput are you able to get during Transformer training relative to the amount of theoretical raw compute available?
    • zak 7y ago
      That's a great question. I don't have that analysis handy, but it would definitely be worth doing.
      • gok 7y ago
        What about energy? (watt-hours?)