I don't think you quite understand how tokenization works. Try typing "strawberry" in here: https://platform.openai.com/tokenizer https://platform.openai.com/t

I don't think you quite understand how tokenization works. Try typing "strawberry" in here: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer Tokens aren't just individual parts of compound words, they're sliced up in a way that's convenient statistically. The tokenizer has each individual character as a token, so it could be purely character-based if desired, it's just easier to compute when some common sequences like "berry" are represented by a single token. Try typing "strawberry" into the tokenizer and see it tokenized as "str", "aw", and "berry". Also, next token prediction is not stupid. A "sufficiently advanced" next token predictor must be at least as intelligent as a human, if it could predict any humans' next token in any scenario. Obviously, we're not there yet, but there's no reason to think right now that next token prediction will face any sort of limitation. Especially with new models coming out that are seeing better perfomance purely from training them much longer on the same datasets.