It's a tradeoff - compared to character level tokens, larger tokens allow you to get much longer context windows with the same amount of computation and slightl

It's a tradeoff - compared to character level tokens, larger tokens allow you to get much longer context windows with the same amount of computation and slightly faster transfer from tokens to concepts (perhaps extra 1-2 transformer layers on the character model would make it equivalent), so getting the same outcome from a character level model would require much more computation, perhaps something like a 3x increase just because you'd have so much more tokens for the same amount of text. Since all the letter counting/spelling/rhyming issues are pretty much irrelevant for the vast majority of practical text understanding tasks as they pretty much trigger on toy puzzle problems, it makes all sense to spend any available computing power on increasing model size (which helps all tasks) instead of switching to character models; when some niche use case does need character analysis, that can get a specialized model, but we don't want to pay so large overhead cost for every task.