One of the quoted blogposts [1] has a comparison of various tokenizers. It seems like the GPT-2/3 tokenizer is performing especially bad, while tokenizers of mo

One of the quoted blogposts [1] has a comparison of various tokenizers. It seems like the GPT-2/3 tokenizer is performing especially bad, while tokenizers of models with the explicit goal of modeling many languages (like google's mt5 [2] or xlm-roberta [3]) have a much smaller disparity. Maybe OpenAI trains their tokenizers on predominately English data, and treats other languages more like a nice bonus? 1: https://denyslinkov.medium.com/why-is-gpt-3-15-77x-more-expensive-for-certain-languages-2b19a4adc4bc https://denyslinkov.medium.com/why-is-gpt-3-15-77x-more-expe... 2: https://huggingface.co/google/mt5-base https://huggingface.co/google/mt5-base 3: https://huggingface.co/xlm-roberta-base https://huggingface.co/xlm-roberta-base