3 ms·

the thing has no fixed world-model What is "world-model"? What makes you think GPT-3 does not have some kind of a world model? It's clearly not a very good one

by p1esk 6y ago

the thing has no fixed world-model What is "world-model"? What makes you think GPT-3 does not have some kind of a world model? It's clearly not a very good one, but at the same time it does not mean it can't get better. A 3 year old also does not have a very good world model, what's the difference between his world model and one of GPT-3? Again, clearly there's a big difference, I'm just not sure we know enough about what's going on inside 175B model to make any dismissive statements about it. I'm also not sure what will happen if you train a much bigger model on much bigger data. Things might start to emerge at some scale.

inimino 6y ago
We know much more about the world-model of GPT-3 (which is none, but it has a language model) than we do about the three year old (which we can't build or reproduce in silico) but we know plenty well enough to say that no, things are not going to magically "emerge" at some scale. You need something fundamentally different from endless cloze deletion exercises to learn what a drawer is.
wokwokwok 6y ago
> What if I finetune GPT-3 ... > if you train a much bigger model on much bigger data... Yep, no one is denying that the potential exists for something to be created that can do something useful. However, the point being made is that GPT-3 doesn't do anything particularly useful. The generated text from is not consistent, and apparently fine tuning can result in poor performance (ie. generates random crap). It scored better at a bunch of metrics, which is great, but of questionable practical value. "world model" or not, it's currently interesting, amazing... probably not useful unless you're writing a spam bot. What is unclear, is if bigger models will actually solve that or not.
HarHarVeryFunny 6y ago
Well, GPT-3 isn't any kind of general intelligence - it's explicitly architected as a language model - something that learns to pay attention to prior context to predict what word comes next. The only kind of world model it has is a statistical model of what word is most likely to come next based on the corpus it has been trained on. You could argue that general intelligence is also based on prediction, and that a human's world model therefore isn't so different in nature, but there are some very significant differences ... 1) GPT-3's model is based only on a corpus of text (facts, lies, errors, etc) it was fed... there is no grounding in reality. 2) GPT-3 is only a passive model - it's not an agent that can act or in any way attempt to validate or augment it's world model. 3) GPT-3 is architecturally a language model .. it can get better with better or more data, but it's never going to be more than a language model. The difference between a 3-year old's brain and GPT-3 is that the 3-year old's brain is not a one-trick pony ... it's a complex cognitive architecture, honed by millions of years evolution, capable of performing a whole range of tasks, not just language modelling. The 3-year old's brain also has the massive advantage of being embedded in a 3-year old autonomous agent able to explore and interact with the world it's world model is representing... It you tell GPT-3 pigs can fly then as far as it is concerned pigs can fly, whereas the 3-year old can go seek out pigs and see that, in fact, they can't.

by p1esk 6y ago

inimino 6y ago

We know much more about the world-model of GPT-3 (which is none, but it has a language model) than we do about the three year old (which we can't build or reproduce in silico) but we know plenty well enough to say that no, things are not going to magically "emerge" at some scale. You need something fundamentally different from endless cloze deletion exercises to learn what a drawer is.

wokwokwok 6y ago

> What if I finetune GPT-3 ... > if you train a much bigger model on much bigger data... Yep, no one is denying that the potential exists for something to be created that can do something useful. However, the point being made is that GPT-3 doesn't do anything particularly useful. The generated text from is not consistent, and apparently fine tuning can result in poor performance (ie. generates random crap). It scored better at a bunch of metrics, which is great, but of questionable practical value. "world model" or not, it's currently interesting, amazing... probably not useful unless you're writing a spam bot. What is unclear, is if bigger models will actually solve that or not.

HarHarVeryFunny 6y ago

Well, GPT-3 isn't any kind of general intelligence - it's explicitly architected as a language model - something that learns to pay attention to prior context to predict what word comes next. The only kind of world model it has is a statistical model of what word is most likely to come next based on the corpus it has been trained on. You could argue that general intelligence is also based on prediction, and that a human's world model therefore isn't so different in nature, but there are some very significant differences ... 1) GPT-3's model is based only on a corpus of text (facts, lies, errors, etc) it was fed... there is no grounding in reality. 2) GPT-3 is only a passive model - it's not an agent that can act or in any way attempt to validate or augment it's world model. 3) GPT-3 is architecturally a language model .. it can get better with better or more data, but it's never going to be more than a language model. The difference between a 3-year old's brain and GPT-3 is that the 3-year old's brain is not a one-trick pony ... it's a complex cognitive architecture, honed by millions of years evolution, capable of performing a whole range of tasks, not just language modelling. The 3-year old's brain also has the massive advantage of being embedded in a 3-year old autonomous agent able to explore and interact with the world it's world model is representing... It you tell GPT-3 pigs can fly then as far as it is concerned pigs can fly, whereas the 3-year old can go seek out pigs and see that, in fact, they can't.