3 ms·

They are absolutely planning ahead inasmuch as what they are outputting is setting up a continuation. They’re not even word predictors remember - they are token

They are absolutely planning ahead inasmuch as what they are outputting is setting up a continuation. They’re not even word predictors remember - they are token predictors. Are you really saying that when you prompt an LLM with ‘name a large grey land animal’ and it outputs ‘ele’, it isn’t ‘planning’ that the next token will likely be ‘phant’? The ‘decision’ to output ‘elephant’ is being made further up the neural network than final token selection - after all, it might want to output ‘Ele’ or ‘an’ (with a view to ultimately outputting ‘an elephant’) or ‘a’ (with a view to ultimately outputting ‘a common large grey land animal is an elephant’), or maybe it has been LoRA trained to output all responses as JSON so the first token it needs to output is ‘{‘… but surely the neural activations for that prompt are firing off ‘elephanty’ messages somewhere in the network, right? So if there’s some sort of symbol activation ahead of token selection, why would it be hard to believe that a large neural network is forming more complex decisions about what it intends to output, in an abstract way, before it selects how to express itself? And in what way is that distinct from ‘planning ahead’?

HarHarVeryFunny 3y ago
> Are you really saying that when you prompt an LLM with ‘name a large grey land animal’ and it outputs ‘ele’, it isn’t ‘planning’ that the next token will likely be ‘phant’? The model outputs words, not tokens, so that is not a great example. Any prompt will have multiple possible (predict next word) continuations, which you can think of as branching futures. Many possible next words, each of which have many possible following words, etc, etc. The model is essentially predicting over all these possible futures. You can call it planning if you like, but remember that the model has no idea of which of these branching futures it is going to follow - it literally doesn't even know which word it is going to output next - it is just providing a bunch of probabilities (predictions) of next word, and the sampling process is then picking one - not necessarily the most confident next word prediction. The model really is winging it word by word, even if those (multiple alternative) next words are only probable because they are part of coherent following sentences in the training data.
- microtonal 3y ago
  The model outputs words, not tokens, so that is not a great example. Virtually all modern transformer models use pieces, which may be words, but also subwords. Theoretically, they could be longer units, but in most cases some characters (like whitespace) are used as piece boundaries when training the piece vocabulary. If they didn’t use pieces, they’d work terribly on languages where e.g. compounds are a single word. In most realistic piece vocabs, ‘elephant’ will be a single piece, since it’s a fairly frequent word. But it’s totally possible in a small vocab that it would be split like the parent said and conversely, it would generate elephant by first predicting one piece. Some piecing methods, like BBPE have bytes as the smallest unit, so theoretically an unknown token could be split up (and generated) as pieces consisting of bytes.
- jameshart 3y ago
  Why so adamant that models work on ‘words’? ChatGPT3.4/4 tokens: “Elephant”: 46439, 28022 - “Ele” “phant” “elephant”: 10274, 28022 - “ele” “phant” “ Elephant”: 79189 “ elephant”: 46840 “ elephantine”: 46840, 483 - “ elephant” “ine” Tokens are tokens. If it was limited to words it wouldn’t be able to produce non-words, but GPT and other LLMs are quite capable of inventing words, outputting nonsense words, and modifying words. Regarding the ‘no idea which future it is going to follow’ - sure, it doesn’t know which future; indeed the sampler phase is going to pick an output merely based on the probabilities it’s outputting. But it’s outputting higher probabilities for some tokens because they are good tokens to use to lead to probable futures. It’s suggesting taking steps down certain paths because those paths are likely to lead to useful places.
  - HarHarVeryFunny 3y ago
    I didn't say WORK on words, I said OUTPUT words. But, it doesn't make any difference whether you are considering tokens or words. There are multiple possible continuations of the prompt, and the next word (or token) output does not - in general - force the word (or token) after that ... Your "large grey mammal" could be an "elected official in a grey suit".
    - 3y ago

HarHarVeryFunny 3y ago

> Are you really saying that when you prompt an LLM with ‘name a large grey land animal’ and it outputs ‘ele’, it isn’t ‘planning’ that the next token will likely be ‘phant’? The model outputs words, not tokens, so that is not a great example. Any prompt will have multiple possible (predict next word) continuations, which you can think of as branching futures. Many possible next words, each of which have many possible following words, etc, etc. The model is essentially predicting over all these possible futures. You can call it planning if you like, but remember that the model has no idea of which of these branching futures it is going to follow - it literally doesn't even know which word it is going to output next - it is just providing a bunch of probabilities (predictions) of next word, and the sampling process is then picking one - not necessarily the most confident next word prediction. The model really is winging it word by word, even if those (multiple alternative) next words are only probable because they are part of coherent following sentences in the training data.

microtonal 3y ago
The model outputs words, not tokens, so that is not a great example. Virtually all modern transformer models use pieces, which may be words, but also subwords. Theoretically, they could be longer units, but in most cases some characters (like whitespace) are used as piece boundaries when training the piece vocabulary. If they didn’t use pieces, they’d work terribly on languages where e.g. compounds are a single word. In most realistic piece vocabs, ‘elephant’ will be a single piece, since it’s a fairly frequent word. But it’s totally possible in a small vocab that it would be split like the parent said and conversely, it would generate elephant by first predicting one piece. Some piecing methods, like BBPE have bytes as the smallest unit, so theoretically an unknown token could be split up (and generated) as pieces consisting of bytes.
jameshart 3y ago
Why so adamant that models work on ‘words’? ChatGPT3.4/4 tokens: “Elephant”: 46439, 28022 - “Ele” “phant” “elephant”: 10274, 28022 - “ele” “phant” “ Elephant”: 79189 “ elephant”: 46840 “ elephantine”: 46840, 483 - “ elephant” “ine” Tokens are tokens. If it was limited to words it wouldn’t be able to produce non-words, but GPT and other LLMs are quite capable of inventing words, outputting nonsense words, and modifying words. Regarding the ‘no idea which future it is going to follow’ - sure, it doesn’t know which future; indeed the sampler phase is going to pick an output merely based on the probabilities it’s outputting. But it’s outputting higher probabilities for some tokens because they are good tokens to use to lead to probable futures. It’s suggesting taking steps down certain paths because those paths are likely to lead to useful places.
- HarHarVeryFunny 3y ago
  I didn't say WORK on words, I said OUTPUT words. But, it doesn't make any difference whether you are considering tokens or words. There are multiple possible continuations of the prompt, and the next word (or token) output does not - in general - force the word (or token) after that ... Your "large grey mammal" could be an "elected official in a grey suit".
  - 3y ago