4 ms·
This paper was interesting at the time but the main sentence in it was wrong: “Thus, emergent abilities cannot be predicted simply by extrapolating the performa
by pama 3y ago
This paper was interesting at the time but the main sentence in it was wrong: “Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models.” Such behaviors can totally be predicted if one looks at the evolution of the likelihoods for these behaviors by scaling and extrapolating from the small models.
- benlivengood 3y agoI don't think many-shot prompting or chain-of-thought were predictable from model size. They just showed up at a particular model+data size.
- canjobear 3y agoMaybe you can predict emergent abilities post hoc, but I recall no one predicting beforehand that, for example, a pure language model could do translation simply be giving a prompt that said “translate to French”