4 ms·
That is because LLMs don’t have any agency. They learn an underlying probability distribution over sequences of characters / tokens, and output the next token
by pfisherman 4y ago
That is because LLMs don’t have any agency. They learn an underlying probability distribution over sequences of characters / tokens, and output the next token in a sequence according to some sampling policy.
They have no goal / intent other than that. They have no agency. If left unprompted, they just sit there like a toy train that has been switched off. If promoted they just append tokens to the input text according to their sampling policy.
Here is how one should think about the current state of the art in NLP. They are to verbal reasoning what calculators are to quantitative reasoning. Powerful tools that represent a step change in capability and will revolutionize how we work. But let’s not anthropomorphise them.
- Kamq 4y ago> If left unprompted, they just sit there like a toy train that has been switched off. This is entirely fair, but how about a hypothetical where one gets to prompt itself? Or, possibly more likely, two models are allowed to invoke each other?
- pfisherman 4y agoThe same thing that happens when you let Stockfish play itself or another chess bot. Except instead of trying to win a game, the objective is to append text with more characters. In fact stockfish and alpha zero can be thought of as appending chess moves to a sequence by sampling from the space of legal moves by sampling from some policy. No one thinks stockfish is alive or worries that it will break out and start WW3.