The "next token predictor" description is a bit too literal, and anyways incomplete. A transformer has a lot of layers (e.g. 96 for GPT-2) and it's only at the

The "next token predictor" description is a bit too literal, and anyways incomplete. A transformer has a lot of layers (e.g. 96 for GPT-2) and it's only at the input that it has pure token embeddings (ignoring positional encoding). At the output it's a built-up embedding that's decodable into a token. One way to think of what all the intermediate layers are doing is to consider them as levels of a linguistic parse tree with the leaves (words) at the bottom and trunk ("sentence") at the top, except in the transformer the evolving embeddings at each level contain semantic as well as syntactic information. This largely hierarchical view of language was the motivation for the transformer design. It seems we should really think of each layer of the transformer as an independent predictor, with increasingly abstract and more semantically complete information available as we ascend the transformer layers towards the output. Predict next token is only what the transformer is being trained to do at the output layer. At the inner layers (i.e. the bulk of what the transformer is doing), it will be predicting at these higher levels of representation held at those layers. These models do struggle (although getting better) at ending on a given word rather than starting on it, and understandably so since random sampling and continual resetting after each output token (= new input sequence) means that planning ahead at level of word specificity is simply not an option. They have to continually adapt to next sampled token, and take it from there. I'm guessing that ability to end on a chosen word is due to continued salience of that word during generation, prediction of sentence fragments using/ending with that word, and opportunistic stopping when it has been emitted and the sentence is complete. Kind of the same way you might do it yourself if you just started talking immediately without planning, while trying to end on a given word.