3 ms·
It's not clear. Scaling laws still seem to hold AFAICT. Right now the bottleneck is "how big a model can you fit on an H100 TPU". It's possible that in a few y
by PoignardAzur 3y ago
It's not clear. Scaling laws still seem to hold AFAICT.
Right now the bottleneck is "how big a model can you fit on an H100 TPU". It's possible that in a few years, when bigger cards come out and/or we get better at compressing models, we'll get even better models just by increasing the scale.