5 ms·
> to express the same sentiment, some languages require up to 10 times more tokens this does not make sense, does it? it may be a true metric as per the setup
by nologic01 3y ago
> to express the same sentiment, some languages require up to 10 times more tokens
this does not make sense, does it? it may be a true metric as per the setup of the comparison, the existing models, the existing corpus etc. but logically it seems an artifact rather than something deep about language information density.
nevertheless it seems worth investigating. I would suspect that once various irrelevant biases are removed (a sort of ur-LLM) there will be an interesting comparative landscape.
- wongarsu 3y agoOne of the quoted blogposts [1] has a comparison of various tokenizers. It seems like the GPT-2/3 tokenizer is performing especially bad, while tokenizers of models with the explicit goal of modeling many languages (like google's mt5 [2] or xlm-roberta [3]) have a much smaller disparity. Maybe OpenAI trains their tokenizers on predominately English data, and treats other languages more like a nice bonus? 1: https://denyslinkov.medium.com/why-is-gpt-3-15-77x-more-expensive-for-certain-languages-2b19a4adc4bc https://denyslinkov.medium.com/why-is-gpt-3-15-77x-more-expe... 2: https://huggingface.co/google/mt5-base https://huggingface.co/google/mt5-base 3: https://huggingface.co/xlm-roberta-base https://huggingface.co/xlm-roberta-base
- ramraj07 3y agoIt is absolutely true (I use these models on other languages regularly) and is also clearly explained elsewhere here as an artifact of the tokenizer optimizing for the English heavy training corpus.