2 ms·
My personal excitement about language models is based on what they can do today. I'm a big believer in the "capability overhang" idea, which is that the existi
by simonw 4y ago
My personal excitement about language models is based on what they can do today.
I'm a big believer in the "capability overhang" idea, which is that the existing language models still have a huge array of capabilities that we haven't discovered yet.
That theory seems to be proved correct on a constant basis. Even the classic "let's think about this step by step" paper came out less than a year ago: https://arxiv.org/abs/2205.11916 https://arxiv.org/abs/2205.11916 - May 2022.
- iliane5 4y agoCouldn't agree more. This paper (https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682) also touches on a pretty fascinating phenomenon - that when scaling up large language models they seem to "naturally" obtain new emergent abilities that do not exist on smaller models.