Tokenizers used to be 1 character per token. Then Google implemented Subword encoding[1] on their early neural translation work and found it was much better. S

Tokenizers used to be 1 character per token. Then Google implemented Subword encoding[1] on their early neural translation work and found it was much better. Subword units are genuinely meaningful in most languages. You do need to tune the vocabulary size though. [1] https://aclanthology.org/P16-1162/ https://aclanthology.org/P16-1162/