5 ms·
Over the past year there have been advances in making models smaller while keeping performance high. So if that continues then he is wrong unless he is definin
by gitfan86 3y ago
Over the past year there have been advances in making models smaller while keeping performance high.
So if that continues then he is wrong unless he is defining LLMs in a strict way that does not include new improvement in the future
- viraptor 3y agoFor an example, the diagrams in the post compare the big gpts, but looking at the number of tokens PHI-2 sits below gpt3. And it still beats it in Humaneval and a few other benchmarks.
- Xelynega 3y agoIt's not about the size of the models, it's about the size of the training data. Humans are able to begin to generalize with a single persons experiences over less than a year, so the fact that LLMs cannot with billions of person-years of information could be an indicator of their inability to generalize no matter how much training data you throw at it.