> it's more efficient to assign single tokens to entire English words Not entirely true, it's just that English has relatively limited inflection so words are

> it's more efficient to assign single tokens to entire English words Not entirely true, it's just that English has relatively limited inflection so words are not modified that much. In the weather example Spanish uses one more token than in English just because of the difference in sentence structure. The eight tokens are essentially "what weather will-be the week that come/s", i.e. the same sentence would use nine tokens in English. For a simpler example, Italian will use two tokens "legg/o" where English will also use two tokens but they will be entire words "I read". But "he read/s" may be three tokens where Italian uses two for "legg/e", because English in this case has some redundancy from its remaining vestiges of inflection. The fact that the tokenizers explored in the article aren't as efficient for non Latin alphabets is a different story.