4 ms·

We all know what the trick is, but none of us know why the trick actually works. Especially the people now shouting that the models "do not understand", but in

We all know what the trick is, but none of us know why the trick actually works. Especially the people now shouting that the models "do not understand", but instead are just doing next token prediction, would for sure not have foreseen that next token prediction apparently simulates understanding. I mean, they still don't see it, although it is there for everyone to try out.

roughly 3y ago
There's three things at play here - First is humans are bad at grokking complex systems - simple rules applied at large scale can create enormously complex systems, and we're bad at predicting what those things are going to be. The second is these systems are "self-tuned" - we explicitly designed the basic structure, but the interactions between different parts of the structure are derived, not created, and so we can't say exactly why those parameters are picked. The final one is that they're simulating language, and we're a species constitutionally prone to anthropomorphizing - if something looks like it's acting with an intent we recognize, we assume it is. That's helpful in an evolutionary sense, but it means it's really easy to fool us into thinking something is sentient. We've been doing this to ourselves since Eliza. The LLMs look impressive, but it takes very little time or effort to notice there's nobody home - not least of which is you need to keep feeding the model everything it's said before and if you tap out of the context window, it forgets. It's a very neat parlor trick, but there's nobody home there, at least not how they're currently constructed and run. I think there's something being captured in how the models are created and trained, but at best, the run model does not allow for consciousness, because there's no continuity and no updating of the model over time.
jprete 3y ago
“Doing next-token prediction” isn’t a contradiction of “understanding” any more than it would be for a SO/friend who can complete your sentences. But it’s useful to remember that autocompletion of sequences is at the bottom of LLMs, and a hefty dose of RLHF of whatever the RLHF raters thought was good output.
- auggierose 3y ago
  I agree, it is not a contradiction. But usually those who bring up doing next-token prediction imply that it is a contradiction. Also, there is often some confusion between consciousness and understanding. LLMs clearly understand on some level, within certain intrinsic and extrinsic constraints, but of course there is no consciousness there.

roughly 3y ago

There's three things at play here - First is humans are bad at grokking complex systems - simple rules applied at large scale can create enormously complex systems, and we're bad at predicting what those things are going to be. The second is these systems are "self-tuned" - we explicitly designed the basic structure, but the interactions between different parts of the structure are derived, not created, and so we can't say exactly why those parameters are picked. The final one is that they're simulating language, and we're a species constitutionally prone to anthropomorphizing - if something looks like it's acting with an intent we recognize, we assume it is. That's helpful in an evolutionary sense, but it means it's really easy to fool us into thinking something is sentient. We've been doing this to ourselves since Eliza. The LLMs look impressive, but it takes very little time or effort to notice there's nobody home - not least of which is you need to keep feeding the model everything it's said before and if you tap out of the context window, it forgets. It's a very neat parlor trick, but there's nobody home there, at least not how they're currently constructed and run. I think there's something being captured in how the models are created and trained, but at best, the run model does not allow for consciousness, because there's no continuity and no updating of the model over time.

jprete 3y ago

“Doing next-token prediction” isn’t a contradiction of “understanding” any more than it would be for a SO/friend who can complete your sentences. But it’s useful to remember that autocompletion of sequences is at the bottom of LLMs, and a hefty dose of RLHF of whatever the RLHF raters thought was good output.

auggierose 3y ago
I agree, it is not a contradiction. But usually those who bring up doing next-token prediction imply that it is a contradiction. Also, there is often some confusion between consciousness and understanding. LLMs clearly understand on some level, within certain intrinsic and extrinsic constraints, but of course there is no consciousness there.