This model isn't especially helpful. "Text prediction" is correct, but glosses right over the fact that it's closer to "concept prediction", where a token is ju

This model isn't especially helpful. "Text prediction" is correct, but glosses right over the fact that it's closer to "concept prediction", where a token is just a forced choice given the model's conceptual representation. Given the way the embedding layer learns, various tokens are likely to be chosen even if they've never been seen in the data in that sequence. This isn't a minor detail.