3 ms·

>>> 8. DNNs grow ~1.5x annually in memory and compute Wow! That's a massive growth rate for ML. TPUv3 was already faster than A100 in MLPerf. But this suggests

>>> 8. DNNs grow ~1.5x annually in memory and compute Wow! That's a massive growth rate for ML. TPUv3 was already faster than A100 in MLPerf. But this suggests a real breakthrough is needed to keep pace with future requirements. Each MXU already handles 16k ops per tick. And with the additional constraint of optimizing per watt rather than dollar, its quite the challange ;)

omegalulw 5y ago
> But this suggests a real breakthrough is needed to keep pace with future requirements. Not necessarily. Look at papers like the lottery ticket hypothesis - big ML models may be doing better simply because gradient descent just isn't doing a good enough job. Better optimizers would go a long way than just throwing compute at the the problem. Even if you can, it's impractical to use something like GPT-3 all the time.