4 ms·
Surely there are diminishing returns for the AI computing though? I mean, is a model with 10x the parameter count 10x better? I think it is still possible that
by thifhi 3y ago
Surely there are diminishing returns for the AI computing though? I mean, is a model with 10x the parameter count 10x better? I think it is still possible that the training costs will be irrelevant for all players at some point with this non-linear scale. Access to data is another story
- PoignardAzur 3y agoIt's not clear. Scaling laws still seem to hold AFAICT. Right now the bottleneck is "how big a model can you fit on an H100 TPU". It's possible that in a few years, when bigger cards come out and/or we get better at compressing models, we'll get even better models just by increasing the scale.
- swalsh 3y ago10x the parameters? Maybe not in a single model, but maybe 10x the expert models has 10x the value. I'm sure there are diminishing returns eventually, but we're probably not close to that.
- mvkel 3y agoIt's still SO early. We are in the "640K [of memory] ought to be enough for anybody" phase of LLMs. So much more to go.