14 ms·

Language models cost much more in some languages than others

260 points by yenniejun111 3y ago

asplake 3y ago
Curious to know if it’s more than just a representational problem. Are some languages harder in some deep way than others? And with what human consequences, let alone the LLM costs?
- qeternity 3y ago
  It’s not. That plays a role, but language efficiency is a real thing.
- deleted 3y ago
  [deleted]
- ajuc 3y ago
  I don't think languages are significantly easier or harder, there's a limit on how hard it can get for average person to be able to speak it comfortably. I think there are different tradeoffs taken and if you come from a language with similar tradeoffs it's easier to learn. My native language is Polish, I know English and a little German and Spanish. Slavic languages often have the reputation of being difficult, but IMHO that's because they are easy in places English-speakers expect to be difficult, and difficult in places English-speakers expect to be easy. There's 15 tenses in English and 3 in Polish. There's no articles in Polish, and the pronunciation is almost perfectly regular. And there's probably 20 times fewer word roots, because of the pre/post-fix system. What in English is 20 unrelated words in Polish is one word root + 20 different combinations of pre/post fixes :) But to take advantage of this when you're learning you have to think in the language you are learning - to realize these words are related and how the postfixes modify the meaning. Otherwise you'll still need to memorize 20 separate words - and on top of that all that crap that is harder in Polish, like cases. I wonder if this influences LLMs (for example if they "think in Polish" when producing Polish text, or "think in English" and translate on the fly). I noticed GPT-3 was much better at rhyming in English than in Polish, despite the fact that rhyming in Polish is very easy (if the final letters match - it rhymes). When I explained this rule to it - it started rhyming better :)
  - tempotemporary 3y ago
    I'd argue it's a bit easier in English if it comes to learning a grammatically correct expression of a thought. Polish or other slavic languages have quite a complexity hidden in "odmiany", "przypadki", and "koniugacje", etc.
  - TRiG_Ireland 3y ago
    One could argue that English has only two true tenses: past and non-past. Everything else is a mood or aspectual distinction.
- ajsnigrutin 3y ago
  Some languages have some specifics that are not common in other languages, and if the language is small, it's hard to account for that too. In slovene, you have singular, plural but also dual forms, so even the basic "strings.xml" types of localizations don't work: eg: "I eat" would be "Jaz jem", if it was two of us the "We eat" would be "Midva jeva", and if 3+ of us would be eating, it would be "Mi jemo". Also when counting, we have a different form for one thing, two things, three-or-four things and five+ things, So, "(1-5) beer/s" would be "1 pivo, 2 pivi, 3 piva, 4 piva, 5 piv" So yeah, good luck :)
  - orthoxerox 3y ago
    Is it "piv" all the way to 20 and then it's "21 pivo" and the pattern repeats every 10 items?
    - ajsnigrutin 3y ago
      All the way luckily, "19 piv, 20 piv, 21 piv,...", so at least this is consistent :)
  - HexDecOctBin 3y ago
    > In slovene, you have singular, plural but also dual forms This is not unusual amongst Indo-European Languages, Sanskrit is similar. In fact, Baltic languages like Lithuanian seem to be a close sister to Indic languages like Sanskrit. [1] [1]: https://www.news9live.com/art-culture/why-lithuanian-sanskrit-similarities-continue-to-intrigue-linguists-two-centuries-on-158492 https://www.news9live.com/art-culture/why-lithuanian-sanskri...
  - numpad0 3y ago
    Do "strings.xml" types of localization work for anything anyway? To me it just seems to let garbages match originals easily and giving false sense of accuracy to English speakers.
- v4dok 3y ago
  No its not. Greek are tokenized on a per character level, while the language structure while not as simple as english or latin languages in General, is not that different.
- rsiqueira 3y ago
  No. It's because OpenAI's token model is highly biased towards English words. In this way, other languages are greatly disadvantaged, reducing competitiveness in non-English-speaking countries. In my language (Portuguese and Spanish), for example, my cost increases by 60% to 75% due to the content I need to process not being in English.
- numpad0 3y ago
  I have a suspicion that LLMs has primary language that it must be always consistent with. If so it makes sense that it performs worse in non-native languages.
deleted 3y ago
[deleted]
hunglee2 3y ago
Fascinating analysis. English language efficiency, on top of English language dominance in training data, should mean advantage for English AI. Along with Nvidia chip dominance, this should see the permanent supremacy of Anglo Saxon culture
- flanked-evergl 3y ago
  > Anglo Saxon culture A language is not a culture. Many people with English as a native language are not culturally Anglo-Saxon. And even if it was, prevalence is not supremacy.
  - montecarl 3y ago
    I really enjoy speaking with native English speakers from India. Fully mastery of the language, but with different idioms! For example "do the needful"[1] https://ell.stackexchange.com/questions/17621/do-the-needful-why-is-it-used-instead-of-asking-a-question https://ell.stackexchange.com/questions/17621/do-the-needful...
  - hunglee2 3y ago
    language is clearly a major part of culture, maybe the essential element of it. Those who control language and are able to impose it - explicitly or implicitly - onto others, have cultural supremacy. The entire world is being Americanised via the English transported via hegemonic control over social and broadcast media. I see this is neither laudable nor worthy of condemnation but simply as a description of observable phenomena
    - nsplayer 3y ago
      In a weird and perhaps slightly twisted way I am hoping that LLMs become so good at spam and creating garbage to flood the net in english that they forfeit the ability to do so in other languages resulting in a balkanization (in a good way) of discussion and thought back to the native languages of peoples and thus more integrated with their culture. in practice translation engines (e.g. deepl or LLMs themselves - though i still expect deepl to be better) probably will throw a wrench into this, but perhaps some localization approach in CAPTCHAs or simply just outright banning geoips not belonging to the countries with natives of that language (or significant minority populations) is a quick enough fix. i know some imageboards (e.g. british ones) use this because otherwise they would be flooded with americans.
compsciphd 3y ago
top picture has no the normal right to left / left to right text copy paste issue with hebrew :) has the trasliteration of "hey" backwards.
- yoavm 3y ago
  Same for Arabic, and the characters also do not use their connected form...
  - yenniejun111 3y ago
    I've tried to address this in the image, could you let me know if it still looks a bit unconnected?
    - yoavm 3y ago
      Sorry for the late response! Your Hebrew is correct now, but the Arabic is still wrong - letters are never written separately as they they appear in the picture. The word is سلام but you're displaying س ل ا م, which are exactly the same letters but as if there were spaces between them.
- Narishma 3y ago
  The Arabic in that pic is even more broken. It's written in the wrong direction and the letters aren't connected correctly.
- oz_zyn 3y ago
  Yep the right to left texts are still not rendered correctly. Same goes for Persian. Disconnected letters appearing from left to right, it should look like: سلام
- yenniejun111 3y ago
  Thank you for bringing this up! I've tried to update the right to left languages... Please let me know if it still looks wonky
pkaye 3y ago
How does tokenization works> Could a different algorithm make say Burmese need fewer tokens? Or will it inherently have more tokens?
- snordgren 3y ago
  I believe it comes down to the statistical frequency in the dataset used to build the tokenizer. If the dataset is 90% Burmese the resulting tokenizer will have single tokens representing common complex constructs in Burmese instead of English. Today's tokenizer might tokenize Burmese letter-by-letter or even byte-by-byte as it is not a very common language on the internet.
  - jiggawatts 3y ago
    The choice of tokenizer is arbitrary, and is configured by the people building the LLM. It's not dictated by the data. A simple model would be to simply tokenize the input character-by-character using Unicode as-is. Heck, you could use 256 (8-bit) tokens and just feed the model UTF-8 as a raw byte stream. The AI would "figure it out". However, it is much more efficient to use a tokenization tuned to the statistics of the input data set. Since most of the Internet is in English, it's more efficient to assign single tokens to entire English words, but not to other languages.
    - astrange 3y ago
      > However, it is much more efficient to use a tokenization tuned to the statistics of the input data set. A program "tuned to the statistics of the input data set" is what an LLM is. So choosing to use a fixed tokenizer rather than letting the LLM learn one is a performance optimization, but not one we'll have to accept if models were designed better for performance themselves.
    - bonzini 3y ago
      > it's more efficient to assign single tokens to entire English words Not entirely true, it's just that English has relatively limited inflection so words are not modified that much. In the weather example Spanish uses one more token than in English just because of the difference in sentence structure. The eight tokens are essentially "what weather will-be the week that come/s", i.e. the same sentence would use nine tokens in English. For a simpler example, Italian will use two tokens "legg/o" where English will also use two tokens but they will be entire words "I read". But "he read/s" may be three tokens where Italian uses two for "legg/e", because English in this case has some redundancy from its remaining vestiges of inflection. The fact that the tokenizers explored in the article aren't as efficient for non Latin alphabets is a different story.
Animats 3y ago
It's interesting that the token length for Chinese is only slightly longer than for English. What does tokenization of an ideographic language look like, anyway? One token per ideograph? Something else?
- derefr 3y ago
  Either one token per radical (as some minimalist proposals for CJK Unicode normalization suggested way back when) — or just map everything to sense-annotated pinyin, i.e. what you type into a Chinese IME to get ideographs out (and which is also, I think, what Chinese text-to-speech engines do internally as an intermediate step.)
pcthrowaway 3y ago
Does the encoding have anything to do with it? When talking about token length, I couldn't help but wonder if they were judging length in UTF-8 bit size, in which case languages using non-Latin alphabets (and even those that do, but with accents) would pay a penalty
HPsquared 3y ago
How about parsing difficulty? Some languages are more context dependent than others. In programming, C++ takes probably 10x more cycles to compile than simpler languages. There are so many possible interpretations of each statement, the correct one of which depends on context.
arp242 3y ago
> This led me down a rabbit hole of trying to find a font that could render all of the language scripts. I went on Google Fonts to find this perfect font and found that one did not exist. This seems like a rather odd example of "technological disparity"; it's just modular and "one huge font file" isn't in wide use because it's unwieldy. Install/add an additional Chinese font, or Tamil font, etc. as needed. Most use cases don't need "all the scripts" and a modular approach is much better as fonts are large: NotoSans-Regular is 590K; NotoSansCJK-Regular is 26M, etc. In total Noto Fonts is 373M on my system. And it contains hundreds of thousands of different glyphs, and requires expertise on dozens of writing systems. Creating such a font is a significant effort, which is why few "universal" fonts exist, and why there are many "for this script" fonts. --- I wonder how one could design a better Morse code for Chinese; Japanese Morse code used Kana, rather than Kanji, and Hangul can be composed from smaller blocks. As near as I can find, Chinese is kind of an outlier here. Any system I can think of would probably be equally difficult to use or error-prone (due to either operator error or line noise mangling things).
- masklinn 3y ago
  > Hangul can be composed from smaller blocks Not “can be”, “is”. Hangul was designed, it’s alphabetic and morpho-syllabic features were intentional.
  - arp242 3y ago
    What is even the point of pedantic "well, duh" comments like this? "English {can be,is} written using the Latin character set" "The symbol 'e' {can be,is} used as a vowel" "Italian {can be,is} spoken in Rome" "Python {can be,is} used to program a computer" "A car {can be,is} used to drive on the road"
    - smallnix 3y ago
      Because not everyone knows Hangul and could learn something wrong.
      - 3y ago
TruthWillHurt 3y ago
I think LLMs need a language of their own to cross communicate and to compile to code. This strengthens that reasoning I believe.
- transfire 3y ago
  Someone asked ChatGPT make its own language that could be used as prompts and it spit some crazy concise emoji. But it somehow worked. Wish I could find the link.
gizajob 3y ago
Good luck with your Finnish language model!
- nsplayer 3y ago
  The irony here is that the Finnish internet will probably be less impacted by AI spam as a result. Perhaps in the end internet times it will be nothing but bots and finns. A fitting outcome.
nsplayer 3y ago
There's a lot of stuff in various news articles about various forms of imperialism and such, but i would eat my hat if the internal workings of LLMs makes their high jouranlistic cutoff for publishing. Tokenization strategies will almost certainly play a significant role in language extinction, and as someone with tremendous respect for linguistics and the role that language plays in thought and culture, it pisses me the hell off. Must we all bow to the anglophone?
ruuda 3y ago
This is no surprise, the tokenizer is constructed to minimize the encoded length of the training corpus, and most of that is in English or at least using the Latin alphabet.
wkat4242 3y ago
Interesting article. I had no idea of this phenomenon. It's very well explained too. Tangential remark: substack is becoming as annoying as medium now. Especially on mobile. One big popup asking to register. One constant toolbar asking to register. One constant toolbar asking to install the app. Many interruptions in the main text for subscribe and share reminders. That didn't take long for it to go bad :( I only heard about substack about a year ago when Snowden's blog was in the news and they were still saying they'd keep it clean (like medium promised initially as well). And it was pretty clean then. I was even thinking of putting my own blog there (which is free and unmonetized) but no. On medium it's become so bad now that I don't even open their links anymore unless it's really something I am so curious about I'm willing to put up with the experience. I really hope substack doesn't go the same way. Sure they have to make money but alienating your userbase doesn't seem a great way to do so in the long term.
- Karellen 3y ago
  > Tangential remark: substack is becoming as annoying as medium now. [...] On medium it's become so bad now that I don't even open their links anymore. FWIW I find that Reader Mode works fine for making posts on substack and medium interruption-free, both on desktop (Firefox) and mobile (Safari). I realise it's possible that either of them might try some Reader Mode defeating hijinks in the future, so you still might want to avoid putting your blog there if you don't want that possibility looming beyond the horizon. But when it comes to reading existing stuff that other people have written, an accessible version should be just one or two clicks away.
  - ninkendo 3y ago
    I've definitely been on Medium pages where Reader Mode only shows one paragraph of the article... it doesn't happen all the time though. They may be intentionally trying to defeat Reader Mode, or it may have just been an accident, I'm not sure.
    - radlad 3y ago
      I have found that for many websites, putting it into Reader Mode and then refreshing the page gives a better result.
nologic01 3y ago
> to express the same sentiment, some languages require up to 10 times more tokens this does not make sense, does it? it may be a true metric as per the setup of the comparison, the existing models, the existing corpus etc. but logically it seems an artifact rather than something deep about language information density. nevertheless it seems worth investigating. I would suspect that once various irrelevant biases are removed (a sort of ur-LLM) there will be an interesting comparative landscape.
- wongarsu 3y ago
  One of the quoted blogposts [1] has a comparison of various tokenizers. It seems like the GPT-2/3 tokenizer is performing especially bad, while tokenizers of models with the explicit goal of modeling many languages (like google's mt5 [2] or xlm-roberta [3]) have a much smaller disparity. Maybe OpenAI trains their tokenizers on predominately English data, and treats other languages more like a nice bonus? 1: https://denyslinkov.medium.com/why-is-gpt-3-15-77x-more-expensive-for-certain-languages-2b19a4adc4bc https://denyslinkov.medium.com/why-is-gpt-3-15-77x-more-expe... 2: https://huggingface.co/google/mt5-base https://huggingface.co/google/mt5-base 3: https://huggingface.co/xlm-roberta-base https://huggingface.co/xlm-roberta-base
- ramraj07 3y ago
  It is absolutely true (I use these models on other languages regularly) and is also clearly explained elsewhere here as an artifact of the tokenizer optimizing for the English heavy training corpus.
justinclift 3y ago
So, to maximise cost savings... pass the input through Google Translate first? ;)
agluszak 3y ago
Isn't that the case with UTF-8 as well? A single Latin character is 1 byte, characters from other alphabets are 2 or 4 bytes
lysecret 3y ago
English has such a moat behind it haha
29athrowaway 3y ago
Languages are technology and as such some are more efficient than others. To me, my native language imposes a tax on learning compared to English.
Lelleander 3y ago
Most LLMs are trained using subword-tokenization such as BPE (which is investigated in this post) and Sentencepiece. These algorithms minimize the number of tokens required for representing the training corpus. I.e. for a training set mostly consisting of English this is a natural consequence. In my opinion a more interesting question would be to ask if chatGPT performs better or worse on languages with "unique" sets of characters (like Burmese & Amharic?) compared to other European languages (like French & German) which might tokenize to shorter lengths but share subwords with English while having different meanings. Also, OpenAI being an American company and training a model which is optimised for English seems very natural... Just query it in English for better and cheaper results. If it would be equally good at 200 different languages it would probably be bad at all of them instead.
- ramraj07 3y ago
  I’ve tried using chatgpt on Tamil and it’s firstly hella slow and secondly can’t do much with it beyond a few hundred words. I figured it’s considering each letter in the sentence as a token, but originality and inventiveness wise it wasn’t necessarily worse.
- fnordpiglet 3y ago
  I’ve talked to folks whose native language is represented on the right side of the distribution and GPT4 performs poorly both in terms of speed as well as language facility for these languages. Interestingly the right side tends to be largely south east and south Asian languages. Malay is an outlier in that it tokenizes fairly small. But Burmese, Khmer, and Thai perform poorly.
fvdessen 3y ago
This is probably not entirely invalidating the result, but the language samples in the dataset seem to be extremely badly translated from english, with unnatural, verbose and grammatically wrong sentences. That would not help with good tokenisation. For example the english text > please add milk to the grocery list Is compared to the french text > s'il vous plaît ajouter du lait à la liste d' épicerie But a native would say > veuillez ajouter du lait à la liste de courses
- yenniejun111 3y ago
  This is a really good point! I also noticed that some of the translations were not good or very stilted for the languages I do speak. However, this is a limitation of the dataset of this size and breadth
nico 3y ago
Fascinating concept I wonder how humans decode these symbols, because it doesn’t seem to be 10x more “costly” for a person to natively learn one of those languages vs another Also, supposedly, independent of the language, humans communicate at an aprox constant rate (39 bits/second according to this: https://www.science.org/content/article/human-speech-may-have-universal-transmission-rate-39-bits-second https://www.science.org/content/article/human-speech-may-hav...) — Here’s the Twitter thread about it from the same author: https://twitter.com/yenniejun/status/1653791622197579776?s=46 https://twitter.com/yenniejun/status/1653791622197579776?s=4...
- K0balt 3y ago
  Some languages require significantly more effort to think in. I believe this has a non-negligible effect on human productivity and advancement. I suspect that the tokenizer is biased towards English, but I also suspect that a self optimizing or universally optimized tokenizer would see similar results. I am multi-lingual and find that some concepts are easier to explore in different languages. I am native English, so I lack certain insights into the relative usefulness of English, but I know many non- native speakers of English that tell me that they prefer to think in English for many cognitive tasks because it is less tiresome or more efficient or more accurate, to use their subjective experiences. I also wonder how the way that a language is learned (instruction vs immersion) might impact this? I suspect that the end result of these two learning styles is much more divergent than external appearance might suggest, as I find that I don’t or only very rarely solve problems in languages I have learned through instruction, while I have a completely separate personaje available in language/cultures I have learned through immersion. Interpersonal relationships in one language are non-translatable to other languages, for example. Certainly language choice impacts the density of textual knowledge, and English seems to have a density advantage here. Books translated from English tend to be simply bigger in order to convey the same information. I wonder if there are languages more dense textually than English and how they tokenize? It is interesting to consider the possibility that language choice could offer a cognitive advantage, and what that might imply for the potential for intentional human language optimization.
  - mcswell 3y ago
rfittich 3y ago
This is a fascinating read. However, the disparity in cost is not that surprising when you consider the factors contributing to it. Very well explained. Kudos!
nadermx 3y ago
Can't this be explained by the special charecters and the extra step to translate them, such as Diccionario Español changes to xn--diccionarioespaol-txb in a domain. So the extra step adds a level of complexity/compute?
13of40 3y ago
Just on the face of it, as someone who has implemented a handful of tokenizers over time and can read Hangul, the Korean example there it four tokens, not twelve. You can break that down more with semantic parsing, but tokenization is not semantic parsing.
stickfigure 3y ago
Languages are subject Metcalfe's Law. In the long run - let's say a thousand years - it's likely that the majority of humans will speak a single language. And that language will most likely be derived from English. The only thing that kept languages separate, historically, was isolation. The internet has fixed that. If you want to publish content to the widest audience, you publish in English. If you want to consume that knowledge, you'd better understand English. The network effect is powerful and English has a substantial lead. Maybe it's the seed planted by British empire. Maybe it's the fact that English fits into 7-bit ascii. Maybe it's the fact that English is already a hodge-podge of germanic and romance languages. Maybe it has something to do with English readily adopting neologisms from other languages. Whatever the historical reasons, if right this second you put a random group of people from different non-English-speaking countries together, they're probably going to talk to each other in English. So yeah, this problem - if you think it's a problem - is going to get worse over time. But there's nothing you can do about it in the long run.
- Ambix 3y ago
  From what I know nobody in China cares about English and that's a nation like 2-3x bigger than all population of English-speaking countries :) Can't see a demise of Chinese even after 1,000 years. As well other languages with big enough user base (I'm not speaking about smallish European countries)
  - Levitz 3y ago
    China is so large that it can afford to ignore other languages, it's also important enough in the global economy that it can get other people to learn Chinese in order to deal with them rather than doing things the other way around. I think the point stands that the more time passes, the more interconnected the world is, the more English becomes beneficial. If the great chinese firewall fell today, millions of Chinese citizens would start leaning more and more English and using it more, odds are that the same wouldn't happen the other way around.
  - stickfigure 3y ago
    The important metric is not "how many people live in English-speaking countries". The important metric is "how many people speak English". There are several times more non-native English speakers than native English speakers. Mandarin doesn't seem to be very popular beyond native speakers, and the Chinese population is shrinking. I wouldn't bet on it in a race for the next Lingua Franca. Though I can imagine a Firefly-like future with Mandarin words mixed in with English.
fennecfoxy 3y ago
Interesting. This sort of thing could be solved by having a transformer work with intermediate and language-less concepts/ideas and then translating to/from a language with an additional model/encode/decoder separately. I think the impact of ensuring "fairness" with language models at this point in time of their development would be quite negative. Does every model need to support Burmese? How large does a company offering a model have to be before it's considered a requirement? OpenAI's homepage (https://openai.com/ https://openai.com/) only seems to be available in English, these transformers are quite new, why haven't we applied the same logic of fairness & inclusivity to every site on the internet? Because it's infeasible unless usage calls for it, I don't believe it's political.

260 points by yenniejun111 3y ago

Curious to know if it’s more than just a representational problem. Are some languages harder in some deep way than others? And with what human consequences, let alone the LLM costs?

qeternity 3y ago
It’s not. That plays a role, but language efficiency is a real thing.
deleted 3y ago
[deleted]
ajuc 3y ago
I don't think languages are significantly easier or harder, there's a limit on how hard it can get for average person to be able to speak it comfortably. I think there are different tradeoffs taken and if you come from a language with similar tradeoffs it's easier to learn. My native language is Polish, I know English and a little German and Spanish. Slavic languages often have the reputation of being difficult, but IMHO that's because they are easy in places English-speakers expect to be difficult, and difficult in places English-speakers expect to be easy. There's 15 tenses in English and 3 in Polish. There's no articles in Polish, and the pronunciation is almost perfectly regular. And there's probably 20 times fewer word roots, because of the pre/post-fix system. What in English is 20 unrelated words in Polish is one word root + 20 different combinations of pre/post fixes :) But to take advantage of this when you're learning you have to think in the language you are learning - to realize these words are related and how the postfixes modify the meaning. Otherwise you'll still need to memorize 20 separate words - and on top of that all that crap that is harder in Polish, like cases. I wonder if this influences LLMs (for example if they "think in Polish" when producing Polish text, or "think in English" and translate on the fly). I noticed GPT-3 was much better at rhyming in English than in Polish, despite the fact that rhyming in Polish is very easy (if the final letters match - it rhymes). When I explained this rule to it - it started rhyming better :)
- tempotemporary 3y ago
  I'd argue it's a bit easier in English if it comes to learning a grammatically correct expression of a thought. Polish or other slavic languages have quite a complexity hidden in "odmiany", "przypadki", and "koniugacje", etc.
- TRiG_Ireland 3y ago
  One could argue that English has only two true tenses: past and non-past. Everything else is a mood or aspectual distinction.
ajsnigrutin 3y ago
Some languages have some specifics that are not common in other languages, and if the language is small, it's hard to account for that too. In slovene, you have singular, plural but also dual forms, so even the basic "strings.xml" types of localizations don't work: eg: "I eat" would be "Jaz jem", if it was two of us the "We eat" would be "Midva jeva", and if 3+ of us would be eating, it would be "Mi jemo". Also when counting, we have a different form for one thing, two things, three-or-four things and five+ things, So, "(1-5) beer/s" would be "1 pivo, 2 pivi, 3 piva, 4 piva, 5 piv" So yeah, good luck :)
- orthoxerox 3y ago
  Is it "piv" all the way to 20 and then it's "21 pivo" and the pattern repeats every 10 items?
  - ajsnigrutin 3y ago
    All the way luckily, "19 piv, 20 piv, 21 piv,...", so at least this is consistent :)
- HexDecOctBin 3y ago
  > In slovene, you have singular, plural but also dual forms This is not unusual amongst Indo-European Languages, Sanskrit is similar. In fact, Baltic languages like Lithuanian seem to be a close sister to Indic languages like Sanskrit. [1] [1]: https://www.news9live.com/art-culture/why-lithuanian-sanskrit-similarities-continue-to-intrigue-linguists-two-centuries-on-158492 https://www.news9live.com/art-culture/why-lithuanian-sanskri...
- numpad0 3y ago
  Do "strings.xml" types of localization work for anything anyway? To me it just seems to let garbages match originals easily and giving false sense of accuracy to English speakers.
v4dok 3y ago
No its not. Greek are tokenized on a per character level, while the language structure while not as simple as english or latin languages in General, is not that different.
rsiqueira 3y ago
No. It's because OpenAI's token model is highly biased towards English words. In this way, other languages are greatly disadvantaged, reducing competitiveness in non-English-speaking countries. In my language (Portuguese and Spanish), for example, my cost increases by 60% to 75% due to the content I need to process not being in English.
numpad0 3y ago
I have a suspicion that LLMs has primary language that it must be always consistent with. If so it makes sense that it performs worse in non-native languages.

deleted 3y ago

[deleted]

hunglee2 3y ago

Fascinating analysis. English language efficiency, on top of English language dominance in training data, should mean advantage for English AI. Along with Nvidia chip dominance, this should see the permanent supremacy of Anglo Saxon culture

flanked-evergl 3y ago
> Anglo Saxon culture A language is not a culture. Many people with English as a native language are not culturally Anglo-Saxon. And even if it was, prevalence is not supremacy.
- montecarl 3y ago
  I really enjoy speaking with native English speakers from India. Fully mastery of the language, but with different idioms! For example "do the needful"[1] https://ell.stackexchange.com/questions/17621/do-the-needful-why-is-it-used-instead-of-asking-a-question https://ell.stackexchange.com/questions/17621/do-the-needful...
- hunglee2 3y ago
  language is clearly a major part of culture, maybe the essential element of it. Those who control language and are able to impose it - explicitly or implicitly - onto others, have cultural supremacy. The entire world is being Americanised via the English transported via hegemonic control over social and broadcast media. I see this is neither laudable nor worthy of condemnation but simply as a description of observable phenomena
  - nsplayer 3y ago
    In a weird and perhaps slightly twisted way I am hoping that LLMs become so good at spam and creating garbage to flood the net in english that they forfeit the ability to do so in other languages resulting in a balkanization (in a good way) of discussion and thought back to the native languages of peoples and thus more integrated with their culture. in practice translation engines (e.g. deepl or LLMs themselves - though i still expect deepl to be better) probably will throw a wrench into this, but perhaps some localization approach in CAPTCHAs or simply just outright banning geoips not belonging to the countries with natives of that language (or significant minority populations) is a quick enough fix. i know some imageboards (e.g. british ones) use this because otherwise they would be flooded with americans.

compsciphd 3y ago

top picture has no the normal right to left / left to right text copy paste issue with hebrew :) has the trasliteration of "hey" backwards.

yoavm 3y ago
Same for Arabic, and the characters also do not use their connected form...
- yenniejun111 3y ago
  I've tried to address this in the image, could you let me know if it still looks a bit unconnected?
  - yoavm 3y ago
    Sorry for the late response! Your Hebrew is correct now, but the Arabic is still wrong - letters are never written separately as they they appear in the picture. The word is سلام but you're displaying س ل ا م, which are exactly the same letters but as if there were spaces between them.
Narishma 3y ago
The Arabic in that pic is even more broken. It's written in the wrong direction and the letters aren't connected correctly.
oz_zyn 3y ago
Yep the right to left texts are still not rendered correctly. Same goes for Persian. Disconnected letters appearing from left to right, it should look like: سلام
yenniejun111 3y ago
Thank you for bringing this up! I've tried to update the right to left languages... Please let me know if it still looks wonky

pkaye 3y ago

How does tokenization works> Could a different algorithm make say Burmese need fewer tokens? Or will it inherently have more tokens?

snordgren 3y ago
I believe it comes down to the statistical frequency in the dataset used to build the tokenizer. If the dataset is 90% Burmese the resulting tokenizer will have single tokens representing common complex constructs in Burmese instead of English. Today's tokenizer might tokenize Burmese letter-by-letter or even byte-by-byte as it is not a very common language on the internet.
- jiggawatts 3y ago
  The choice of tokenizer is arbitrary, and is configured by the people building the LLM. It's not dictated by the data. A simple model would be to simply tokenize the input character-by-character using Unicode as-is. Heck, you could use 256 (8-bit) tokens and just feed the model UTF-8 as a raw byte stream. The AI would "figure it out". However, it is much more efficient to use a tokenization tuned to the statistics of the input data set. Since most of the Internet is in English, it's more efficient to assign single tokens to entire English words, but not to other languages.
  - astrange 3y ago
    > However, it is much more efficient to use a tokenization tuned to the statistics of the input data set. A program "tuned to the statistics of the input data set" is what an LLM is. So choosing to use a fixed tokenizer rather than letting the LLM learn one is a performance optimization, but not one we'll have to accept if models were designed better for performance themselves.
  - bonzini 3y ago
    > it's more efficient to assign single tokens to entire English words Not entirely true, it's just that English has relatively limited inflection so words are not modified that much. In the weather example Spanish uses one more token than in English just because of the difference in sentence structure. The eight tokens are essentially "what weather will-be the week that come/s", i.e. the same sentence would use nine tokens in English. For a simpler example, Italian will use two tokens "legg/o" where English will also use two tokens but they will be entire words "I read". But "he read/s" may be three tokens where Italian uses two for "legg/e", because English in this case has some redundancy from its remaining vestiges of inflection. The fact that the tokenizers explored in the article aren't as efficient for non Latin alphabets is a different story.

Animats 3y ago

It's interesting that the token length for Chinese is only slightly longer than for English. What does tokenization of an ideographic language look like, anyway? One token per ideograph? Something else?

derefr 3y ago
Either one token per radical (as some minimalist proposals for CJK Unicode normalization suggested way back when) — or just map everything to sense-annotated pinyin, i.e. what you type into a Chinese IME to get ideographs out (and which is also, I think, what Chinese text-to-speech engines do internally as an intermediate step.)

pcthrowaway 3y ago

Does the encoding have anything to do with it? When talking about token length, I couldn't help but wonder if they were judging length in UTF-8 bit size, in which case languages using non-Latin alphabets (and even those that do, but with accents) would pay a penalty

HPsquared 3y ago

How about parsing difficulty? Some languages are more context dependent than others. In programming, C++ takes probably 10x more cycles to compile than simpler languages. There are so many possible interpretations of each statement, the correct one of which depends on context.

arp242 3y ago

> This led me down a rabbit hole of trying to find a font that could render all of the language scripts. I went on Google Fonts to find this perfect font and found that one did not exist. This seems like a rather odd example of "technological disparity"; it's just modular and "one huge font file" isn't in wide use because it's unwieldy. Install/add an additional Chinese font, or Tamil font, etc. as needed. Most use cases don't need "all the scripts" and a modular approach is much better as fonts are large: NotoSans-Regular is 590K; NotoSansCJK-Regular is 26M, etc. In total Noto Fonts is 373M on my system. And it contains hundreds of thousands of different glyphs, and requires expertise on dozens of writing systems. Creating such a font is a significant effort, which is why few "universal" fonts exist, and why there are many "for this script" fonts. --- I wonder how one could design a better Morse code for Chinese; Japanese Morse code used Kana, rather than Kanji, and Hangul can be composed from smaller blocks. As near as I can find, Chinese is kind of an outlier here. Any system I can think of would probably be equally difficult to use or error-prone (due to either operator error or line noise mangling things).

masklinn 3y ago
> Hangul can be composed from smaller blocks Not “can be”, “is”. Hangul was designed, it’s alphabetic and morpho-syllabic features were intentional.
- arp242 3y ago
  What is even the point of pedantic "well, duh" comments like this? "English {can be,is} written using the Latin character set" "The symbol 'e' {can be,is} used as a vowel" "Italian {can be,is} spoken in Rome" "Python {can be,is} used to program a computer" "A car {can be,is} used to drive on the road"
  - smallnix 3y ago
    Because not everyone knows Hangul and could learn something wrong.
    - 3y ago

TruthWillHurt 3y ago

I think LLMs need a language of their own to cross communicate and to compile to code. This strengthens that reasoning I believe.

transfire 3y ago
Someone asked ChatGPT make its own language that could be used as prompts and it spit some crazy concise emoji. But it somehow worked. Wish I could find the link.

gizajob 3y ago

Good luck with your Finnish language model!

nsplayer 3y ago
The irony here is that the Finnish internet will probably be less impacted by AI spam as a result. Perhaps in the end internet times it will be nothing but bots and finns. A fitting outcome.

nsplayer 3y ago

There's a lot of stuff in various news articles about various forms of imperialism and such, but i would eat my hat if the internal workings of LLMs makes their high jouranlistic cutoff for publishing. Tokenization strategies will almost certainly play a significant role in language extinction, and as someone with tremendous respect for linguistics and the role that language plays in thought and culture, it pisses me the hell off. Must we all bow to the anglophone?

ruuda 3y ago

This is no surprise, the tokenizer is constructed to minimize the encoded length of the training corpus, and most of that is in English or at least using the Latin alphabet.

wkat4242 3y ago

Interesting article. I had no idea of this phenomenon. It's very well explained too. Tangential remark: substack is becoming as annoying as medium now. Especially on mobile. One big popup asking to register. One constant toolbar asking to register. One constant toolbar asking to install the app. Many interruptions in the main text for subscribe and share reminders. That didn't take long for it to go bad :( I only heard about substack about a year ago when Snowden's blog was in the news and they were still saying they'd keep it clean (like medium promised initially as well). And it was pretty clean then. I was even thinking of putting my own blog there (which is free and unmonetized) but no. On medium it's become so bad now that I don't even open their links anymore unless it's really something I am so curious about I'm willing to put up with the experience. I really hope substack doesn't go the same way. Sure they have to make money but alienating your userbase doesn't seem a great way to do so in the long term.

Karellen 3y ago
> Tangential remark: substack is becoming as annoying as medium now. [...] On medium it's become so bad now that I don't even open their links anymore. FWIW I find that Reader Mode works fine for making posts on substack and medium interruption-free, both on desktop (Firefox) and mobile (Safari). I realise it's possible that either of them might try some Reader Mode defeating hijinks in the future, so you still might want to avoid putting your blog there if you don't want that possibility looming beyond the horizon. But when it comes to reading existing stuff that other people have written, an accessible version should be just one or two clicks away.
- ninkendo 3y ago
  I've definitely been on Medium pages where Reader Mode only shows one paragraph of the article... it doesn't happen all the time though. They may be intentionally trying to defeat Reader Mode, or it may have just been an accident, I'm not sure.
  - radlad 3y ago
    I have found that for many websites, putting it into Reader Mode and then refreshing the page gives a better result.

nologic01 3y ago

> to express the same sentiment, some languages require up to 10 times more tokens this does not make sense, does it? it may be a true metric as per the setup of the comparison, the existing models, the existing corpus etc. but logically it seems an artifact rather than something deep about language information density. nevertheless it seems worth investigating. I would suspect that once various irrelevant biases are removed (a sort of ur-LLM) there will be an interesting comparative landscape.

wongarsu 3y ago
One of the quoted blogposts [1] has a comparison of various tokenizers. It seems like the GPT-2/3 tokenizer is performing especially bad, while tokenizers of models with the explicit goal of modeling many languages (like google's mt5 [2] or xlm-roberta [3]) have a much smaller disparity. Maybe OpenAI trains their tokenizers on predominately English data, and treats other languages more like a nice bonus? 1: https://denyslinkov.medium.com/why-is-gpt-3-15-77x-more-expensive-for-certain-languages-2b19a4adc4bc https://denyslinkov.medium.com/why-is-gpt-3-15-77x-more-expe... 2: https://huggingface.co/google/mt5-base https://huggingface.co/google/mt5-base 3: https://huggingface.co/xlm-roberta-base https://huggingface.co/xlm-roberta-base
ramraj07 3y ago
It is absolutely true (I use these models on other languages regularly) and is also clearly explained elsewhere here as an artifact of the tokenizer optimizing for the English heavy training corpus.

justinclift 3y ago

So, to maximise cost savings... pass the input through Google Translate first? ;)

agluszak 3y ago

Isn't that the case with UTF-8 as well? A single Latin character is 1 byte, characters from other alphabets are 2 or 4 bytes

lysecret 3y ago

English has such a moat behind it haha

29athrowaway 3y ago

Languages are technology and as such some are more efficient than others. To me, my native language imposes a tax on learning compared to English.

Lelleander 3y ago

Most LLMs are trained using subword-tokenization such as BPE (which is investigated in this post) and Sentencepiece. These algorithms minimize the number of tokens required for representing the training corpus. I.e. for a training set mostly consisting of English this is a natural consequence. In my opinion a more interesting question would be to ask if chatGPT performs better or worse on languages with "unique" sets of characters (like Burmese & Amharic?) compared to other European languages (like French & German) which might tokenize to shorter lengths but share subwords with English while having different meanings. Also, OpenAI being an American company and training a model which is optimised for English seems very natural... Just query it in English for better and cheaper results. If it would be equally good at 200 different languages it would probably be bad at all of them instead.

ramraj07 3y ago
I’ve tried using chatgpt on Tamil and it’s firstly hella slow and secondly can’t do much with it beyond a few hundred words. I figured it’s considering each letter in the sentence as a token, but originality and inventiveness wise it wasn’t necessarily worse.
fnordpiglet 3y ago
I’ve talked to folks whose native language is represented on the right side of the distribution and GPT4 performs poorly both in terms of speed as well as language facility for these languages. Interestingly the right side tends to be largely south east and south Asian languages. Malay is an outlier in that it tokenizes fairly small. But Burmese, Khmer, and Thai perform poorly.

fvdessen 3y ago

This is probably not entirely invalidating the result, but the language samples in the dataset seem to be extremely badly translated from english, with unnatural, verbose and grammatically wrong sentences. That would not help with good tokenisation. For example the english text > please add milk to the grocery list Is compared to the french text > s'il vous plaît ajouter du lait à la liste d' épicerie But a native would say > veuillez ajouter du lait à la liste de courses

yenniejun111 3y ago
This is a really good point! I also noticed that some of the translations were not good or very stilted for the languages I do speak. However, this is a limitation of the dataset of this size and breadth

nico 3y ago

Fascinating concept I wonder how humans decode these symbols, because it doesn’t seem to be 10x more “costly” for a person to natively learn one of those languages vs another Also, supposedly, independent of the language, humans communicate at an aprox constant rate (39 bits/second according to this: https://www.science.org/content/article/human-speech-may-have-universal-transmission-rate-39-bits-second https://www.science.org/content/article/human-speech-may-hav...) — Here’s the Twitter thread about it from the same author: https://twitter.com/yenniejun/status/1653791622197579776?s=46 https://twitter.com/yenniejun/status/1653791622197579776?s=4...

K0balt 3y ago
Some languages require significantly more effort to think in. I believe this has a non-negligible effect on human productivity and advancement. I suspect that the tokenizer is biased towards English, but I also suspect that a self optimizing or universally optimized tokenizer would see similar results. I am multi-lingual and find that some concepts are easier to explore in different languages. I am native English, so I lack certain insights into the relative usefulness of English, but I know many non- native speakers of English that tell me that they prefer to think in English for many cognitive tasks because it is less tiresome or more efficient or more accurate, to use their subjective experiences. I also wonder how the way that a language is learned (instruction vs immersion) might impact this? I suspect that the end result of these two learning styles is much more divergent than external appearance might suggest, as I find that I don’t or only very rarely solve problems in languages I have learned through instruction, while I have a completely separate personaje available in language/cultures I have learned through immersion. Interpersonal relationships in one language are non-translatable to other languages, for example. Certainly language choice impacts the density of textual knowledge, and English seems to have a density advantage here. Books translated from English tend to be simply bigger in order to convey the same information. I wonder if there are languages more dense textually than English and how they tokenize? It is interesting to consider the possibility that language choice could offer a cognitive advantage, and what that might imply for the potential for intentional human language optimization.
- mcswell 3y ago

rfittich 3y ago

This is a fascinating read. However, the disparity in cost is not that surprising when you consider the factors contributing to it. Very well explained. Kudos!

nadermx 3y ago

Can't this be explained by the special charecters and the extra step to translate them, such as Diccionario Español changes to xn--diccionarioespaol-txb in a domain. So the extra step adds a level of complexity/compute?

13of40 3y ago

Just on the face of it, as someone who has implemented a handful of tokenizers over time and can read Hangul, the Korean example there it four tokens, not twelve. You can break that down more with semantic parsing, but tokenization is not semantic parsing.

stickfigure 3y ago

Languages are subject Metcalfe's Law. In the long run - let's say a thousand years - it's likely that the majority of humans will speak a single language. And that language will most likely be derived from English. The only thing that kept languages separate, historically, was isolation. The internet has fixed that. If you want to publish content to the widest audience, you publish in English. If you want to consume that knowledge, you'd better understand English. The network effect is powerful and English has a substantial lead. Maybe it's the seed planted by British empire. Maybe it's the fact that English fits into 7-bit ascii. Maybe it's the fact that English is already a hodge-podge of germanic and romance languages. Maybe it has something to do with English readily adopting neologisms from other languages. Whatever the historical reasons, if right this second you put a random group of people from different non-English-speaking countries together, they're probably going to talk to each other in English. So yeah, this problem - if you think it's a problem - is going to get worse over time. But there's nothing you can do about it in the long run.

Ambix 3y ago
From what I know nobody in China cares about English and that's a nation like 2-3x bigger than all population of English-speaking countries :) Can't see a demise of Chinese even after 1,000 years. As well other languages with big enough user base (I'm not speaking about smallish European countries)
- Levitz 3y ago
  China is so large that it can afford to ignore other languages, it's also important enough in the global economy that it can get other people to learn Chinese in order to deal with them rather than doing things the other way around. I think the point stands that the more time passes, the more interconnected the world is, the more English becomes beneficial. If the great chinese firewall fell today, millions of Chinese citizens would start leaning more and more English and using it more, odds are that the same wouldn't happen the other way around.
- stickfigure 3y ago
  The important metric is not "how many people live in English-speaking countries". The important metric is "how many people speak English". There are several times more non-native English speakers than native English speakers. Mandarin doesn't seem to be very popular beyond native speakers, and the Chinese population is shrinking. I wouldn't bet on it in a race for the next Lingua Franca. Though I can imagine a Firefly-like future with Mandarin words mixed in with English.

fennecfoxy 3y ago

Interesting. This sort of thing could be solved by having a transformer work with intermediate and language-less concepts/ideas and then translating to/from a language with an additional model/encode/decoder separately. I think the impact of ensuring "fairness" with language models at this point in time of their development would be quite negative. Does every model need to support Burmese? How large does a company offering a model have to be before it's considered a requirement? OpenAI's homepage (https://openai.com/ https://openai.com/) only seems to be available in English, these transformers are quite new, why haven't we applied the same logic of fairness & inclusivity to every site on the internet? Because it's infeasible unless usage calls for it, I don't believe it's political.