4 ms·

This is interesting because my impression is that LLMs perform poorly on non-english languages[1]. This is likely 'just' a corpus size problem - but it's not li

This is interesting because my impression is that LLMs perform poorly on non-english languages[1]. This is likely 'just' a corpus size problem - but it's not like that is going to be fixed any time soon. That said, at least in language instruction there's a lot of meta material that will be helpful. I wonder how Duolingo is detecting when GPT-4 is hallucinating. [1] https://twitter.com/asmelashteka/status/1630966233217605632 https://twitter.com/asmelashteka/status/1630966233217605632

ssnistfajen 4y ago
LLM outputs in Standard Chinese are fluent and coherent but the grammar style and choice of words are clearly non-native. I think the source training material just isn't as comprehensive as English, and contemporary Chinese Internet is packed full of latin abbreviations and obscure homophones/wordplay due to its evolution under severe censorship.
- jazzyjackson 4y ago
  Awesome observation re: censorship. Last I looked into training sets, Chinese and English were the frontrunners by far in available corpus size, and benefit from a certain amount of government-enforced-homogeneity (a standard mode is taught in compulsory education). Arabic is safe from automation due to its limited exposure to the open web and extreme variability of dialects, and Icelandic will remain obscure due to sample size and the fact that they mostly speak English online.
  - bayesian_horse 4y ago
    The islandic government is apparently partnering with OpenAI to "preserve their language". I suppose they have a GPT-4 customized for that language.
    - jazzyjackson 4y ago
      oh neat, thanks for the heads up
  - ssnistfajen 4y ago
    Formal text wise Chinese is still a very comprehensive language. However in terms of user-generated content it has degraded severely especially in the past ~5 years. Internet slangs in English can be confusing too, but users have a choice in whether to use them or not. On Chinese social media using abbreviations/homophones has become practically baseline for certain common terms because these platforms do so much preemptive self-censorship due to the government's near-invisible red line that fluctuates constantly.
  - rafram 4y ago
    Arabic is also often written in the so-called Arabic chat alphabet / Arabizi online, so a lot of Arabic text available on the internet looks less like حروف عربية and more like “7ruf 3rabiye”.
famouswaffles 4y ago
They don't perform poorly (depends on how much corpus training the language received). https://arxiv.org/abs/2108.13349 https://arxiv.org/abs/2108.13349. The important thing is you don't need an equivalent number of tokens for an equivalent performance. So even though GPT-3 only had 1.8 % french by word count, it can speak fluent french. There's positive transfer between languages. Also GLM-130B trained on equal English/Chinese (200b tokens each. 400b total) speaks both just fine.
Toutouxc 4y ago
For what it's worth, ChatGPT is able to hold a meaningful conversation in Czech (pretty difficult, latin script, ~11M speakers). I can almost tell it's a bot, but it could also be a PR person from a big corporation who can't even afford to talk like a normal person anymore.
KeplerBoy 4y ago
ChatGPT's and Bing's German is flawless.
slkdjfalzkdfj 4y ago
The Twitter thread you linked is just somebody pointing out poor performance in Tigrinya. That doesn't mean it performs poorly on all non-English languages. It's performing poorly on the languages with little representation in the corpus, but those are also the languages that are least likely to be learned on Duolingo. Duolingo users mainly learn languages like English, Spanish, French, German, etc. and the models do just fine with these languages.
gwern 4y ago
The GPT-4 post/paper covers this: GPT-4 does other languages excellently, and performs on par in many other languages with GPT-3.5 in English. As people found that useful...
kzrdude 4y ago
Bing chatbot's swedish is pretty good, but if you ask it to tell a joke it's often an english language pun that got translated and the point was lost. At least that's impressive in another way, that the knowledge model behind the pun seems to be a bit independent of language.
pallas_athena 4y ago
>the only courses that can utilize these new features are Spanish and French for English speakers >We also constantly review AI-generated explanations in Roleplay and Explain My Answer to ensure that the answers are factually correct

ssnistfajen 4y ago

LLM outputs in Standard Chinese are fluent and coherent but the grammar style and choice of words are clearly non-native. I think the source training material just isn't as comprehensive as English, and contemporary Chinese Internet is packed full of latin abbreviations and obscure homophones/wordplay due to its evolution under severe censorship.

jazzyjackson 4y ago
Awesome observation re: censorship. Last I looked into training sets, Chinese and English were the frontrunners by far in available corpus size, and benefit from a certain amount of government-enforced-homogeneity (a standard mode is taught in compulsory education). Arabic is safe from automation due to its limited exposure to the open web and extreme variability of dialects, and Icelandic will remain obscure due to sample size and the fact that they mostly speak English online.
- bayesian_horse 4y ago
  The islandic government is apparently partnering with OpenAI to "preserve their language". I suppose they have a GPT-4 customized for that language.
  - jazzyjackson 4y ago
    oh neat, thanks for the heads up
- ssnistfajen 4y ago
  Formal text wise Chinese is still a very comprehensive language. However in terms of user-generated content it has degraded severely especially in the past ~5 years. Internet slangs in English can be confusing too, but users have a choice in whether to use them or not. On Chinese social media using abbreviations/homophones has become practically baseline for certain common terms because these platforms do so much preemptive self-censorship due to the government's near-invisible red line that fluctuates constantly.
- rafram 4y ago
  Arabic is also often written in the so-called Arabic chat alphabet / Arabizi online, so a lot of Arabic text available on the internet looks less like حروف عربية and more like “7ruf 3rabiye”.

famouswaffles 4y ago

They don't perform poorly (depends on how much corpus training the language received). https://arxiv.org/abs/2108.13349 https://arxiv.org/abs/2108.13349. The important thing is you don't need an equivalent number of tokens for an equivalent performance. So even though GPT-3 only had 1.8 % french by word count, it can speak fluent french. There's positive transfer between languages. Also GLM-130B trained on equal English/Chinese (200b tokens each. 400b total) speaks both just fine.

Toutouxc 4y ago

For what it's worth, ChatGPT is able to hold a meaningful conversation in Czech (pretty difficult, latin script, ~11M speakers). I can almost tell it's a bot, but it could also be a PR person from a big corporation who can't even afford to talk like a normal person anymore.

KeplerBoy 4y ago

ChatGPT's and Bing's German is flawless.

slkdjfalzkdfj 4y ago

The Twitter thread you linked is just somebody pointing out poor performance in Tigrinya. That doesn't mean it performs poorly on all non-English languages. It's performing poorly on the languages with little representation in the corpus, but those are also the languages that are least likely to be learned on Duolingo. Duolingo users mainly learn languages like English, Spanish, French, German, etc. and the models do just fine with these languages.

gwern 4y ago

The GPT-4 post/paper covers this: GPT-4 does other languages excellently, and performs on par in many other languages with GPT-3.5 in English. As people found that useful...

kzrdude 4y ago

Bing chatbot's swedish is pretty good, but if you ask it to tell a joke it's often an english language pun that got translated and the point was lost. At least that's impressive in another way, that the knowledge model behind the pun seems to be a bit independent of language.

pallas_athena 4y ago

>the only courses that can utilize these new features are Spanish and French for English speakers >We also constantly review AI-generated explanations in Roleplay and Explain My Answer to ensure that the answers are factually correct