4 ms·
Agreed -- given the talk is titled "Exciting trends in Machine Learning" it feels pretty incomplete to gloss over ChatGPT blowing up. OpenAI bet big on (1) eme
by babl-yc 3y ago
Agreed -- given the talk is titled "Exciting trends in Machine Learning" it feels pretty incomplete to gloss over ChatGPT blowing up.
OpenAI bet big on (1) emergent behaviors as you scale language models and (2) RLHF / fine-tuning to follow instructions.
Both those topics were lightly covered but very much from the "Google did this" perspective (word2vec, step-by-step reasoning).
- lern_too_spel 3y agoThe original paper on emergent behaviors in LLMs is from Google: https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682 The difference between OpenAI and Google isn't due to different research directions between the two firms. It is a difference in ability to execute.
- babl-yc 3y agoGoogle certainly had influential papers on language models having emergent behaviors, but this isn't the one that inspired scaling up GPT. It was published August 2022 and GPTs at OpenAI were getting scaled up long before this.
- pama 3y agoThis paper was interesting at the time but the main sentence in it was wrong: “Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models.” Such behaviors can totally be predicted if one looks at the evolution of the likelihoods for these behaviors by scaling and extrapolating from the small models.
- benlivengood 3y agoI don't think many-shot prompting or chain-of-thought were predictable from model size. They just showed up at a particular model+data size.
- canjobear 3y agoMaybe you can predict emergent abilities post hoc, but I recall no one predicting beforehand that, for example, a pure language model could do translation simply be giving a prompt that said “translate to French”
- _t89y 3y agoFrom the GPT-2 paper: ``Also thanks to all the Googlers who helped us with training infrastructure.''
- elevatedastalt 3y agoThe research breakthroughs all happened at Google, so it's not surprising.
- janalsncm 3y agoRLHF was invented by OpenAI. DPO was invented at Stanford.
- deleted 3y ago[deleted]
- _t89y 3y agoExactly. Google.
- candiodari 3y agoAnd all the people that actually did the inventing promptly left Google. (yes I get there's 2 exceptions)