4 ms·

There has to be more going on somewhere in the system. You can ask GPT4 to describe something (Vermont) and ask it to end to description with a chosen word (hou

There has to be more going on somewhere in the system. You can ask GPT4 to describe something (Vermont) and ask it to end to description with a chosen word (house). It will then usually be able to write out a descriptive paragraph that ultimately lands on your chosen word pretty seamlessly - "...embodying a serene and rural charm that culminates in the warm, welcoming feel of a cozy house." Models do struggle with these tests, for sure, but from an analytical standpoint, a "next token predictor" should not be able to ever correctly land on the right token 100 tokens in the future. Edit: Thinking about it, I suppose it is possible that the model can encode a "destination" in the first token. Like a pool shot that is artfully bounced off many bumpers to hit a ball, perhaps the LLM can encode a "path" to a destination token in the first token generated. Which might be even crazier as it suggests that the model is playing a meta-game with being able to precisely manipulate the individual layers of output, even though those layers are disparate from token to token.

HarHarVeryFunny 3y ago
The "next token predictor" description is a bit too literal, and anyways incomplete. A transformer has a lot of layers (e.g. 96 for GPT-2) and it's only at the input that it has pure token embeddings (ignoring positional encoding). At the output it's a built-up embedding that's decodable into a token. One way to think of what all the intermediate layers are doing is to consider them as levels of a linguistic parse tree with the leaves (words) at the bottom and trunk ("sentence") at the top, except in the transformer the evolving embeddings at each level contain semantic as well as syntactic information. This largely hierarchical view of language was the motivation for the transformer design. It seems we should really think of each layer of the transformer as an independent predictor, with increasingly abstract and more semantically complete information available as we ascend the transformer layers towards the output. Predict next token is only what the transformer is being trained to do at the output layer. At the inner layers (i.e. the bulk of what the transformer is doing), it will be predicting at these higher levels of representation held at those layers. These models do struggle (although getting better) at ending on a given word rather than starting on it, and understandably so since random sampling and continual resetting after each output token (= new input sequence) means that planning ahead at level of word specificity is simply not an option. They have to continually adapt to next sampled token, and take it from there. I'm guessing that ability to end on a chosen word is due to continued salience of that word during generation, prediction of sentence fragments using/ending with that word, and opportunistic stopping when it has been emitted and the sentence is complete. Kind of the same way you might do it yourself if you just started talking immediately without planning, while trying to end on a given word.

HarHarVeryFunny 3y ago

The "next token predictor" description is a bit too literal, and anyways incomplete. A transformer has a lot of layers (e.g. 96 for GPT-2) and it's only at the input that it has pure token embeddings (ignoring positional encoding). At the output it's a built-up embedding that's decodable into a token. One way to think of what all the intermediate layers are doing is to consider them as levels of a linguistic parse tree with the leaves (words) at the bottom and trunk ("sentence") at the top, except in the transformer the evolving embeddings at each level contain semantic as well as syntactic information. This largely hierarchical view of language was the motivation for the transformer design. It seems we should really think of each layer of the transformer as an independent predictor, with increasingly abstract and more semantically complete information available as we ascend the transformer layers towards the output. Predict next token is only what the transformer is being trained to do at the output layer. At the inner layers (i.e. the bulk of what the transformer is doing), it will be predicting at these higher levels of representation held at those layers. These models do struggle (although getting better) at ending on a given word rather than starting on it, and understandably so since random sampling and continual resetting after each output token (= new input sequence) means that planning ahead at level of word specificity is simply not an option. They have to continually adapt to next sampled token, and take it from there. I'm guessing that ability to end on a chosen word is due to continued salience of that word during generation, prediction of sentence fragments using/ending with that word, and opportunistic stopping when it has been emitted and the sentence is complete. Kind of the same way you might do it yourself if you just started talking immediately without planning, while trying to end on a given word.