This is the correct take. Much has been written about how tokenization hurts tasks that the LLM providers literally market their model on (Anthropic Hiaku, Son

This is the correct take. Much has been written about how tokenization hurts tasks that the LLM providers literally market their model on (Anthropic Hiaku, Sonnet): https://aclanthology.org/2022.cai-1.2/ https://aclanthology.org/2022.cai-1.2/