5 ms·

I'm not reducing GPT-3 to the extent that you're suggesting. I'm pointing out (and so does LeCun in his post) that it's a language model designed to continue a

by st1x7 6y ago

I'm not reducing GPT-3 to the extent that you're suggesting. I'm pointing out (and so does LeCun in his post) that it's a language model designed to continue a sequence of words. It has no understanding of the world and is no particularly suited for knowledge extraction or conversation. > GPT-3 is a really impressive milestone towards AGI We really don't know this. It's a big step for the field of language models, that's for sure. But we're so far from AGI that nobody knows which direction it's in and whether it exists at all.

moultano 6y ago
> it's a language model designed to continue a sequence of words. If a language model were able to do this task perfectly, it would be indistinguishable from intelligence, because continuing a sequence of words requires reasoning. You cannot conclude that has no understanding based solely on what it is trained to do when the task it is trained on would be sufficient to demonstrate understanding were it to fully succeed. There are lots of reasons to be skeptical of its potential, but this isn't one of them.
- uryga 6y ago
  > you cannot conclude that [a model] has no understanding based solely on what it is trained to do agreed. but that's not everything we're basing our conclusions on – we also know that GPT-3 was trained purely on text, and i (and presumably GP) don't think that's a path towards "understanding". in other words, i think being a language model [trained only using a text corpus] is a valid reason to be skeptical of its potential :)
  - Veedrac 6y ago
    There's this really weird idea that you need high-bandwidth highly-multimodal sensory data to ‘truly’ understand things, but this makes no sense to me. 1. It's all just data in the end. The signals from your eye are not any more real because they are in response to photons. 2. Deafblind people exist, and can even learn to speak vocally. They might have misconceptions about the visual world, but it's certainly not true that they lack general intelligence. 3. As an objection to GPT-3 as a pathway to AGI, the only thing stopping us training these models on high-bandwidth highly-multimodal data is scale, anyway, so the objection doesn't work. 4. GPT-3 is clearly capable of reasoning in ways that can only be explained with more sophisticated world models than at smaller scales. People asserted reasoning was at a limit with GPT-2, and it just clearly isn't.
ralfn 6y ago
>It has no understanding of the world What does this mean? The world with countries, borders, celebrities, hypes, newsletters and ideologies? Because that world is as virtual and not grounded in reality as the world of data points fed into this model. Culture is made up. Language is made up. It may express itself in reality but so do the algorithms of social network sites. If something can only live in a reality of data points on the internet this thing would live mostly in the same reality as we do.

by st1x7 6y ago

moultano 6y ago

> it's a language model designed to continue a sequence of words. If a language model were able to do this task perfectly, it would be indistinguishable from intelligence, because continuing a sequence of words requires reasoning. You cannot conclude that has no understanding based solely on what it is trained to do when the task it is trained on would be sufficient to demonstrate understanding were it to fully succeed. There are lots of reasons to be skeptical of its potential, but this isn't one of them.

uryga 6y ago
> you cannot conclude that [a model] has no understanding based solely on what it is trained to do agreed. but that's not everything we're basing our conclusions on – we also know that GPT-3 was trained purely on text, and i (and presumably GP) don't think that's a path towards "understanding". in other words, i think being a language model [trained only using a text corpus] is a valid reason to be skeptical of its potential :)
- Veedrac 6y ago
  There's this really weird idea that you need high-bandwidth highly-multimodal sensory data to ‘truly’ understand things, but this makes no sense to me. 1. It's all just data in the end. The signals from your eye are not any more real because they are in response to photons. 2. Deafblind people exist, and can even learn to speak vocally. They might have misconceptions about the visual world, but it's certainly not true that they lack general intelligence. 3. As an objection to GPT-3 as a pathway to AGI, the only thing stopping us training these models on high-bandwidth highly-multimodal data is scale, anyway, so the objection doesn't work. 4. GPT-3 is clearly capable of reasoning in ways that can only be explained with more sophisticated world models than at smaller scales. People asserted reasoning was at a limit with GPT-2, and it just clearly isn't.

ralfn 6y ago

>It has no understanding of the world What does this mean? The world with countries, borders, celebrities, hypes, newsletters and ideologies? Because that world is as virtual and not grounded in reality as the world of data points fed into this model. Culture is made up. Language is made up. It may express itself in reality but so do the algorithms of social network sites. If something can only live in a reality of data points on the internet this thing would live mostly in the same reality as we do.