6 ms·

Aren't all frontier models already able to use all these languages? Support for specific languages doesn't need to be built in, LLMs support all languages becau

Aren't all frontier models already able to use all these languages? Support for specific languages doesn't need to be built in, LLMs support all languages because they are trained on multilingual data.

charlieyu1 1y ago
Training is a very different thing. Can’t speak for European, but LLMs are often much worse in Japanese because tokenisation used Unicode and a single Japanese character often has to be represented by more than one token
- unscaled 1y ago
  I think you meant to say that tokenization is usually done with UTF-8 and a single Japanese character generally takes 3 or more code units (i.e. bytes). Unicode itself is not the culprit (in fact, even with UTF-16 tokenization, most Japanese characters would fit in a single code unit, and the ones that won't are exceedingly rare). I have to admit I have not encountered significant mistokenization issues in Japanese, but I'm not using it on a daily basis LLMs. I'm somewhat dobutful this can be a major issue, since frontier LLMs are absolutely in love with Emoji, and Emoji requires at least 4 UTF-8 bytes, while most Japanese characters are happy with just 3 bytes.
ideasarecool 1y ago
Term support is vague. Can you do basic interaction in most other languages? Sure. Is it anywhere close to competence it has in english? No. Most models seem to just translate english responses at beginners simplistic monotone level.
melvinmelih 1y ago
> because they are trained on multilingual data But they were not trained on government-sanctioned homegrown EU data.
- saretup 1y ago
  The entirety of the internet vs government-sanctioned homegrown EU data.
- raverbashing 1y ago
  > But they were not trained on government-sanctioned homegrown EU data. If none of the LLM makers used the very big corpus of EU multilingual data I have an EU regulation bridge to sell it to you
- tonyhart7 1y ago
  "But they were not trained on government-sanctioned homegrown EU data." ok what are you implying on this
  - mock-possum 1y ago
    Sidesteps potential legal issues probably
- sunaookami 1y ago
  Who in their right mind would use this?
  - tensor 1y ago
    I'd use a model trained on a targeted and curated data set over one trained on all the crap on the internet any day.
    - sunaookami 1y ago
      We are talking about government-curated data here, the bias should be obvious. Popular LLMs still have huge bias problems but it would be way worse with only government-curated data.
    - loandbehold 1y ago
      I keep hearing that LLMs are trained on "Internet crap" but is it true? For instance we know from Anthropic copyright case that they scanned millions of books to make a training set. They certainly use Internet content for training but I'm sure it's curated to a large degree. They don't just scrap random pages and feed into LLM.
lm28469 1y ago
Meh, it depends a lot on the dataset, which are heavily skewed towards the main languages. For example they almost always confuse Czech and Slovak and often swap one for the other in middle of chats
- mirekrusin 1y ago
  But the only way to unskew it is to remove main language data because there isn't really any to add, no?
  - tensor 1y ago
    You can also correctly bias your sampling so that when selecting new training instances each language is chosen equally. Generally the diversity of data is good, unless that data is "wrong" which, ironically, is probably most of the internet, but I digress.
- RobotToaster 1y ago
  Aren't they about as different as American English and British English?
  - svobodovic 1y ago
    The difference ia larger than let's say just a "dialect". They really are different languages, even though we generally understand each other quite well (younger generations less so). I've heard it's about as different as e. g. Danish and Swedish - not sure if that comparison is helpful.
intended 1y ago
Nope. Capability begins to degrade once you move away from english. Plus all your T&S/AI Safety is not solved with translation, you need lexicons and data sets of examples. Like, people use someone in Malaysia, to label the Arabic spoken by someone playing a video game in Doha - the cultural context is missing. The best proxy to show the degree of lopsidedness was from this : https://cdt.org/insights/lost-in-translation-large-language-models-in-non-english-content-analysis/ https://cdt.org/insights/lost-in-translation-large-language-... Which in turn had to base it on this: https://stats.aclrollingreview.org/submissions/linguistic-diversity/ https://stats.aclrollingreview.org/submissions/linguistic-di... From what I am aware of, LLM capability degrades once you move out of English, and many nation states are either building, or considering the option of building their own LLMs.
tensor 1y ago
No, that's not how training works. It's not just about having an example in a given language, but also how many examples and the ratio of examples compared to other languages. English hugely eclipses any other language on most US models and that's why performance on other languages is subpar compared to performance on english.
- andy12_ 1y ago
  I have never noticed any major difference in performance of ChatGPT between English and Spanish. The truth is that as long as the amount of training data of a given language is above some threshold, knowledge transfers between languages.
  - FinnKuhn 1y ago
    The issue starts, when an LLM's transfers knowledge between languages, even though that knowledge is not correct in that language. I have seen this with e.g. ChatGPT answers regarding laws for example where it refers to US laws when asked in German, which are obviously not relevant.
    - dragonwriter 1y ago
      > The issue starts, when an LLM's transfers knowledge between languages, even though that knowledge is not correct in that language. I have seen this with e.g. ChatGPT answers regarding laws for example where it refers to US laws when asked in German, which are obviously not relevant. There is no necessary correlation between language and the correct set of laws to reference. The language of the question (or the answer, if for some reason they are not the same) is an orthogonal issue to the intended scope. There is no reason US laws couldn't be the relevant to a question asked in German (and, conversely, no reason US laws couldn't be wrong for a question asked in English, even if it was specifically and distinguishably US English.)
      - FinnKuhn 1y ago
        When you ask an LLM (in German) without further clarifying your location I expect it to refer to German (or Austrian/Swiss) laws. For most questions it does this pretty well (e.g. asking for the legal age to drink). However once the answer becomes more complex it starts to halucinate very quickly. The fact that some of the hallucinations are just translated US laws makes me think that the knowledge transfer between languages is probably not helping in instances like this.
numpad0 1y ago
Not natively, they all sound translated in languages other than English. I occasionally come across French people complaining about LLMs' use of non-idiomatic French, but it's probably not a French problem at all, considering that this effort includes so many Indo-European languages.
- FinnKuhn 1y ago
  I can at least also confirm this for German. Here is one example that is quite annyoing: Chat GPT for example tends to start emails with "ich hoffe, es geht dir gut!", which means "I hope you are well!". In English (especially American) corporate emails this is a really common way to start an email. In German it is not as "how are you" isn't a common phrase used here.
deleted 1y ago
[deleted]
whazor 1y ago
European governments have huge collections of digitalised books, research, public data. But also European culture could maybe make a difference? You can already see big differences between Grok and ChatGPT in terms of values.
- pembrook 1y ago
  If it's publicly available data, books and research, I can assure you the big models have already all been trained on it. European culture is already embedded in all the models, unless the people involved in this project have some hidden trove of private data that they're training on which diverges drastically from things Europeans have published publicly (I'm 99.9% positive they don't...especially given Europe's alarmist attitude around anything related to data). I think people don't understand a huge percentage of the employees at OpenAI, Anthropic, etc. are non-US born.

charlieyu1 1y ago

Training is a very different thing. Can’t speak for European, but LLMs are often much worse in Japanese because tokenisation used Unicode and a single Japanese character often has to be represented by more than one token

unscaled 1y ago
I think you meant to say that tokenization is usually done with UTF-8 and a single Japanese character generally takes 3 or more code units (i.e. bytes). Unicode itself is not the culprit (in fact, even with UTF-16 tokenization, most Japanese characters would fit in a single code unit, and the ones that won't are exceedingly rare). I have to admit I have not encountered significant mistokenization issues in Japanese, but I'm not using it on a daily basis LLMs. I'm somewhat dobutful this can be a major issue, since frontier LLMs are absolutely in love with Emoji, and Emoji requires at least 4 UTF-8 bytes, while most Japanese characters are happy with just 3 bytes.

ideasarecool 1y ago

Term support is vague. Can you do basic interaction in most other languages? Sure. Is it anywhere close to competence it has in english? No. Most models seem to just translate english responses at beginners simplistic monotone level.

melvinmelih 1y ago

> because they are trained on multilingual data But they were not trained on government-sanctioned homegrown EU data.

saretup 1y ago
The entirety of the internet vs government-sanctioned homegrown EU data.
raverbashing 1y ago
> But they were not trained on government-sanctioned homegrown EU data. If none of the LLM makers used the very big corpus of EU multilingual data I have an EU regulation bridge to sell it to you
tonyhart7 1y ago
"But they were not trained on government-sanctioned homegrown EU data." ok what are you implying on this
- mock-possum 1y ago
  Sidesteps potential legal issues probably
sunaookami 1y ago
Who in their right mind would use this?
- tensor 1y ago
  I'd use a model trained on a targeted and curated data set over one trained on all the crap on the internet any day.
  - sunaookami 1y ago
    We are talking about government-curated data here, the bias should be obvious. Popular LLMs still have huge bias problems but it would be way worse with only government-curated data.
  - loandbehold 1y ago
    I keep hearing that LLMs are trained on "Internet crap" but is it true? For instance we know from Anthropic copyright case that they scanned millions of books to make a training set. They certainly use Internet content for training but I'm sure it's curated to a large degree. They don't just scrap random pages and feed into LLM.

lm28469 1y ago

Meh, it depends a lot on the dataset, which are heavily skewed towards the main languages. For example they almost always confuse Czech and Slovak and often swap one for the other in middle of chats

mirekrusin 1y ago
But the only way to unskew it is to remove main language data because there isn't really any to add, no?
- tensor 1y ago
  You can also correctly bias your sampling so that when selecting new training instances each language is chosen equally. Generally the diversity of data is good, unless that data is "wrong" which, ironically, is probably most of the internet, but I digress.
RobotToaster 1y ago
Aren't they about as different as American English and British English?
- svobodovic 1y ago
  The difference ia larger than let's say just a "dialect". They really are different languages, even though we generally understand each other quite well (younger generations less so). I've heard it's about as different as e. g. Danish and Swedish - not sure if that comparison is helpful.

intended 1y ago

Nope. Capability begins to degrade once you move away from english. Plus all your T&S/AI Safety is not solved with translation, you need lexicons and data sets of examples. Like, people use someone in Malaysia, to label the Arabic spoken by someone playing a video game in Doha - the cultural context is missing. The best proxy to show the degree of lopsidedness was from this : https://cdt.org/insights/lost-in-translation-large-language-models-in-non-english-content-analysis/ https://cdt.org/insights/lost-in-translation-large-language-... Which in turn had to base it on this: https://stats.aclrollingreview.org/submissions/linguistic-diversity/ https://stats.aclrollingreview.org/submissions/linguistic-di... From what I am aware of, LLM capability degrades once you move out of English, and many nation states are either building, or considering the option of building their own LLMs.

tensor 1y ago

No, that's not how training works. It's not just about having an example in a given language, but also how many examples and the ratio of examples compared to other languages. English hugely eclipses any other language on most US models and that's why performance on other languages is subpar compared to performance on english.

andy12_ 1y ago
I have never noticed any major difference in performance of ChatGPT between English and Spanish. The truth is that as long as the amount of training data of a given language is above some threshold, knowledge transfers between languages.
- FinnKuhn 1y ago
  The issue starts, when an LLM's transfers knowledge between languages, even though that knowledge is not correct in that language. I have seen this with e.g. ChatGPT answers regarding laws for example where it refers to US laws when asked in German, which are obviously not relevant.
  - dragonwriter 1y ago
    > The issue starts, when an LLM's transfers knowledge between languages, even though that knowledge is not correct in that language. I have seen this with e.g. ChatGPT answers regarding laws for example where it refers to US laws when asked in German, which are obviously not relevant. There is no necessary correlation between language and the correct set of laws to reference. The language of the question (or the answer, if for some reason they are not the same) is an orthogonal issue to the intended scope. There is no reason US laws couldn't be the relevant to a question asked in German (and, conversely, no reason US laws couldn't be wrong for a question asked in English, even if it was specifically and distinguishably US English.)
    - FinnKuhn 1y ago
      When you ask an LLM (in German) without further clarifying your location I expect it to refer to German (or Austrian/Swiss) laws. For most questions it does this pretty well (e.g. asking for the legal age to drink). However once the answer becomes more complex it starts to halucinate very quickly. The fact that some of the hallucinations are just translated US laws makes me think that the knowledge transfer between languages is probably not helping in instances like this.

numpad0 1y ago

Not natively, they all sound translated in languages other than English. I occasionally come across French people complaining about LLMs' use of non-idiomatic French, but it's probably not a French problem at all, considering that this effort includes so many Indo-European languages.

FinnKuhn 1y ago
I can at least also confirm this for German. Here is one example that is quite annyoing: Chat GPT for example tends to start emails with "ich hoffe, es geht dir gut!", which means "I hope you are well!". In English (especially American) corporate emails this is a really common way to start an email. In German it is not as "how are you" isn't a common phrase used here.

deleted 1y ago

[deleted]

whazor 1y ago

European governments have huge collections of digitalised books, research, public data. But also European culture could maybe make a difference? You can already see big differences between Grok and ChatGPT in terms of values.

pembrook 1y ago
If it's publicly available data, books and research, I can assure you the big models have already all been trained on it. European culture is already embedded in all the models, unless the people involved in this project have some hidden trove of private data that they're training on which diverges drastically from things Europeans have published publicly (I'm 99.9% positive they don't...especially given Europe's alarmist attitude around anything related to data). I think people don't understand a huge percentage of the employees at OpenAI, Anthropic, etc. are non-US born.