4 ms·
This is very interesting. A question, how do less widespread languages fare against English? Is the quality output the same in i.e. Thai and English/Spanish?
by mmasu 3y ago
This is very interesting. A question, how do less widespread languages fare against English? Is the quality output the same in i.e. Thai and English/Spanish?
- fnordpiglet 3y agoGiven the performance of tokenizers by language the answer must be “worse,” there are a variety of sources for this .. they typically focus on cost, but it’s also borne out in quality. https://denyslinkov.medium.com/why-is-gpt-3-15-77x-more-expensive-for-certain-languages-2b19a4adc4bc https://denyslinkov.medium.com/why-is-gpt-3-15-77x-more-expe... https://arxiv.org/pdf/2305.13707 https://arxiv.org/pdf/2305.13707 https://arxiv.org/pdf/2305.15425 https://arxiv.org/pdf/2305.15425 A quick Kagi can find a lot more.
- cwbuilds 3y agoMixed reviews so far. I've had one person say the Hebrew was brilliant, and one say that it was terrible. It seems to be slightly less accurate than English, but good enough for most purposes