2 ms·

The magic is this: more is different. [1] Parrots just repeat simple phrases, these language models recombine words to fit new contexts in ways that make sense

The magic is this: more is different. [1] Parrots just repeat simple phrases, these language models recombine words to fit new contexts in ways that make sense. They memorise facts and common sense knowledge and use it to successfully solve hundreds of language based tasks. They are problem solvers, not parrots. This term was invented by an anti language model activist to discredit the big language model approach. Fortunately people still forge ahead with research and we get the likes of GPT-3, PaLM and DALL-E. [1] Sutton's "Bitter Lesson" - the importance of scale in AI models. Even AI researchers don't like to accept this fact. No wonder people are incredulous. http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html Richard Sutton is one of the top AI researchers with 115,498 paper citations. https://en.wikipedia.org/wiki/Richard_S._Sutton https://en.wikipedia.org/wiki/Richard_S._Sutton

mdp2021 4y ago
> Sutton's "Bitter Lesson" The page from Sutton makes a statement that it is an error to implement imitations of human mind, especially hardcoding in models of it. The implementation of Intelligence is not an imitation of human mind, nor it needs hardcoding in simplified models. > problem solvers, not parrots They may solve language based tasks, yet Intelligence is a different matter. Parroting was used here in the sense of "repeating acritically inputs", not in the sense you used it. > invented by an anti language model activist And others, as you can see from this case, coincidentally decided for that term independently and unaware that others used it (as is normal). Probably driven to that having seen somebody call "intelligence" what is described as a system of consolidated associations - and noting that Intelligence is the opposite.

mdp2021 4y ago

> Sutton's "Bitter Lesson" The page from Sutton makes a statement that it is an error to implement imitations of human mind, especially hardcoding in models of it. The implementation of Intelligence is not an imitation of human mind, nor it needs hardcoding in simplified models. > problem solvers, not parrots They may solve language based tasks, yet Intelligence is a different matter. Parroting was used here in the sense of "repeating acritically inputs", not in the sense you used it. > invented by an anti language model activist And others, as you can see from this case, coincidentally decided for that term independently and unaware that others used it (as is normal). Probably driven to that having seen somebody call "intelligence" what is described as a system of consolidated associations - and noting that Intelligence is the opposite.