3 ms·

How does that performance scale though? If you double the inputs you train on, do you get twice the performance? Less? More? I've heard conflicting things and

• peteradio 4y ago
  I asked ChatGPT, it said: "Uhhh it depends."
• ivalm 4y ago
  Chinchilla paper proposes the following empirical scaling power law Loss= (N0/N)^a+(D0/D)^b Where N is number of parameters, D is number of examples, a,b are exponents. N0,D0 are scaling factors.
• 152334H 4y ago
  Where A,B,C are empirical constants, Loss = A/parameters + B/dataset_size + C Just read the chinchilla paper.
• stevenhuang 4y ago
  The scaling appears linear in relationship, though on some performance metrics it is super linear. See Emergent Abilities of Large Language Models https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682 Scaling Laws for Generative Mixed-Modal Language Models https://arxiv.org/abs/2301.03728 https://arxiv.org/abs/2301.03728 All this demonstrates further the "bitter lesson" http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html that surprising ability emerges from just throwing more data and compute at the problem and is seen more fruitful an endeavor than trying to find a deeper/analytic solution to cognition (like Yann Lecun's research, who has basically become a meme now for how much he poopoos LLMs and insists what we're seeing are LLMs only pretending to reason).