3 ms·
The original paper on emergent behaviors in LLMs is from Google: https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682 The difference between OpenA
by lern_too_spel 3y ago
The original paper on emergent behaviors in LLMs is from Google: https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682
The difference between OpenAI and Google isn't due to different research directions between the two firms. It is a difference in ability to execute.
- babl-yc 3y agoGoogle certainly had influential papers on language models having emergent behaviors, but this isn't the one that inspired scaling up GPT. It was published August 2022 and GPTs at OpenAI were getting scaled up long before this.
- pama 3y agoThis paper was interesting at the time but the main sentence in it was wrong: “Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models.” Such behaviors can totally be predicted if one looks at the evolution of the likelihoods for these behaviors by scaling and extrapolating from the small models.
- benlivengood 3y agoI don't think many-shot prompting or chain-of-thought were predictable from model size. They just showed up at a particular model+data size.
- canjobear 3y agoMaybe you can predict emergent abilities post hoc, but I recall no one predicting beforehand that, for example, a pure language model could do translation simply be giving a prompt that said “translate to French”
- _t89y 3y agoFrom the GPT-2 paper: ``Also thanks to all the Googlers who helped us with training infrastructure.''