3 ms·
> In case this is not clear, the human mind obviously doesn't do stochastic prediction on phrases. Are you sure? Because this is exactly what you and I are doi
by blackbear_ 6y ago
> In case this is not clear, the human mind obviously doesn't do stochastic prediction on phrases.
Are you sure? Because this is exactly what you and I are doing right now. A language model is a very appropriate description for how we see each other: you produce some text, I produce some more, and you respond based on that. And I inject some stochasticity by re-typing this paragraph five times trying to make it sound like natural English and a cohesive text, a bit like beam search if you are familiar with that.
What I am trying to say is that language models are a good interface (in the programming sense) to describe human interactions on the internet: text in, text out. So while I agree that GPT-3 is not a realistic "human-like" implementation of this interface, I don't see why a priori a neural network cannot eventually incorporate world models with agents and so on and reach actual understanding (whatever that means) of text.
- tsimionescu 6y agoWe may be agreed in fact, to some extent at least. My claim, and I believe the article's as well, is that this model of the world and agents and everything will not probably be learned just by consuming ever larger quantities of text and adding ever more parameters to the model. But I would also claim that our conversation now is unlikely to be predictable without some model of the world. It's not 'you produce some text, I produce some text', it's 'you produce some text, I run it through my model of the world, my inference engine produces some other model of the world, and then that gets translated to some text in reply'.