4 ms·
Yes and no. The performance does scale up with parameters, though it’s not linear. As discovered by Google in their work on their Chinchilla LLM, performance
by dougmwne 4y ago
Yes and no.
The performance does scale up with parameters, though it’s not linear.
As discovered by Google in their work on their Chinchilla LLM, performance also scales up with the size of your training set. They were able to do work to define the optimal amount of training for a model of a given size to get the most out of your budget.
So even if we don’t find any better model architectures, which we probably will, if we increase the size of our models, training corpus and budget, we should continue to get more performant models.
- freeqaz 4y agoHow does that performance scale though? If you double the inputs you train on, do you get twice the performance? Less? More? I've heard conflicting things and I'm curious to hear your reasoning!
- peteradio 4y agoI asked ChatGPT, it said: "Uhhh it depends."
- ivalm 4y agoChinchilla paper proposes the following empirical scaling power law Loss= (N0/N)^a+(D0/D)^b Where N is number of parameters, D is number of examples, a,b are exponents. N0,D0 are scaling factors.
- 152334H 4y agoWhere A,B,C are empirical constants, Loss = A/parameters + B/dataset_size + C Just read the chinchilla paper.
- stevenhuang 4y agoThe scaling appears linear in relationship, though on some performance metrics it is super linear. See Emergent Abilities of Large Language Models https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682 Scaling Laws for Generative Mixed-Modal Language Models https://arxiv.org/abs/2301.03728 https://arxiv.org/abs/2301.03728 All this demonstrates further the "bitter lesson" http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html that surprising ability emerges from just throwing more data and compute at the problem and is seen more fruitful an endeavor than trying to find a deeper/analytic solution to cognition (like Yann Lecun's research, who has basically become a meme now for how much he poopoos LLMs and insists what we're seeing are LLMs only pretending to reason).
- fbdab103 4y agoThe corpus part is the one that gets me. Presumably Google, Facebook, Microsoft, etc have a mirror of essentially the entire public (and their private) internet communications. An enormous amount of published English literature has been digitized. What more corpus is even available? If the corpus is a matter of human annotation, what exactly is being annotated in say the Illiad? A tweet that says, "You up?". The lyrics to 8 Mile. So on and so forth.
- falcor84 4y agoI would assume that video and audio are the next frontier. Talking about Google, training on the entirety of YouTube would probably be next, if they haven't yet used it. And the frontier after is probably having a fleet of actual robots capture additional information, as per this paper.