4 ms·
Most LLMs are trained using subword-tokenization such as BPE (which is investigated in this post) and Sentencepiece. These algorithms minimize the number of to
by Lelleander 3y ago
Most LLMs are trained using subword-tokenization such as BPE (which is investigated in this post) and Sentencepiece.
These algorithms minimize the number of tokens required for representing the training corpus. I.e. for a training set mostly consisting of English this is a natural consequence.
In my opinion a more interesting question would be to ask if chatGPT performs better or worse on languages with "unique" sets of characters (like Burmese & Amharic?) compared to other European languages (like French & German) which might tokenize to shorter lengths but share subwords with English while having different meanings.
Also, OpenAI being an American company and training a model which is optimised for English seems very natural... Just query it in English for better and cheaper results. If it would be equally good at 200 different languages it would probably be bad at all of them instead.
- ramraj07 3y agoI’ve tried using chatgpt on Tamil and it’s firstly hella slow and secondly can’t do much with it beyond a few hundred words. I figured it’s considering each letter in the sentence as a token, but originality and inventiveness wise it wasn’t necessarily worse.
- fnordpiglet 3y agoI’ve talked to folks whose native language is represented on the right side of the distribution and GPT4 performs poorly both in terms of speed as well as language facility for these languages. Interestingly the right side tends to be largely south east and south Asian languages. Malay is an outlier in that it tokenizes fairly small. But Burmese, Khmer, and Thai perform poorly.