3 ms·

Let's not be ridiculous. Of course LLMs are planning ahead, not generating words one at a time. You can't even generate correct indefinite articles (a/an) witho

Let's not be ridiculous. Of course LLMs are planning ahead, not generating words one at a time. You can't even generate correct indefinite articles (a/an) without planning one word (or token) ahead. Surely you are not suggesting LLMs pick a/an according to statistical distribution and then choose vowel starting/non-starting words, not vice versa? This is testable: let it complete "I climbed up the apple/pear tree and picked". It always picks "an" for apple and "a" for pear, and never reaches a dead end by picking the wrong article so that it either fails syntax (wrong article) or semantics (pear growing on apple tree). We actually have fairly good idea how LLMs pick a/an. Have a look at https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-an-neuron-in-gpt-2 https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-a....

HarHarVeryFunny 4y ago
I'm not sure that this apple/pear example proves anything. In the context of "apple tree and picked" in the input, it's enough simply that the word "an" received positive feedback during training as the next word, and having generated "an" then "apple" becomes almost certain as the next word after that. For sure there has to be some internal high level representation of the entire input, and I would expect also some representation of possible continuations at a higher level than just alternate word choices, but I wouldn't exactly characterize this as a plan since it's so statistical in nature, and the model has to continually adapt to what is actually generated (sampled). At a low temperature setting "an" might always be picked, making "apple" more likely to follow, but at a higher temperature maybe another word such as "the" might be chosen, perhaps with a continuation of "biggest apple i could find", etc. In a 350B token training set I'm sure there are a number of other alternatives too. So, does the model plan ahead ? Well, sort of, but this "plan" has to represent multiple branching alternate futures, and continually adapt as the output is sampled, so it's a very fluid kind of plan. It's also not clear how far ahead the plan needs to represent .. maybe one word only (like a state machine), even if the reason those alternate one-word continuations are there is because they were seen as part of specific longer continuations. The lack of any internal state preservation from one word generated to the next puts a huge limit on what can be planned, as does the extent of how complex a plan can be generated from the flow of a single input sequence forwards thru the layers of the model. You can see the limitations of this with prompts such as "pick two primes > 200, and tell me their product, but not the numbers themselves"... if you play with it and analyze the mistakes, it makes it clear how much the output really is generated word by word, adapting to what came before, much of the time.

HarHarVeryFunny 4y ago

I'm not sure that this apple/pear example proves anything. In the context of "apple tree and picked" in the input, it's enough simply that the word "an" received positive feedback during training as the next word, and having generated "an" then "apple" becomes almost certain as the next word after that. For sure there has to be some internal high level representation of the entire input, and I would expect also some representation of possible continuations at a higher level than just alternate word choices, but I wouldn't exactly characterize this as a plan since it's so statistical in nature, and the model has to continually adapt to what is actually generated (sampled). At a low temperature setting "an" might always be picked, making "apple" more likely to follow, but at a higher temperature maybe another word such as "the" might be chosen, perhaps with a continuation of "biggest apple i could find", etc. In a 350B token training set I'm sure there are a number of other alternatives too. So, does the model plan ahead ? Well, sort of, but this "plan" has to represent multiple branching alternate futures, and continually adapt as the output is sampled, so it's a very fluid kind of plan. It's also not clear how far ahead the plan needs to represent .. maybe one word only (like a state machine), even if the reason those alternate one-word continuations are there is because they were seen as part of specific longer continuations. The lack of any internal state preservation from one word generated to the next puts a huge limit on what can be planned, as does the extent of how complex a plan can be generated from the flow of a single input sequence forwards thru the layers of the model. You can see the limitations of this with prompts such as "pick two primes > 200, and tell me their product, but not the numbers themselves"... if you play with it and analyze the mistakes, it makes it clear how much the output really is generated word by word, adapting to what came before, much of the time.