3 ms·

That's clearly a different question. It'd be possible for these models to be Mandarin-first while still supporting other languages, like American models are Eng

That's clearly a different question. It'd be possible for these models to be Mandarin-first while still supporting other languages, like American models are English-first while doing the same, but that's not what's happening.

overfeed 1y ago
> That's clearly a different question. It'd be possible for these models to be Mandarin-first while still supporting other languages What would a hypothetical "Mandarin-first" model look like to you? I challenge the notion that the current models are "English-first" - that is an unsubstantiated opinion not supported by fact. I bet, dollars to donuts, these models are SoTA in Mandarin as well. When framed that way, asking "Why are they marketed as English-speaking models outside of China" or "Why are they really good at English" are simply not interesting questions - they have obvious answers.
- yorwba 1y ago
  > What would a hypothetical "Mandarin-first" model look like to you? Given a language-agnostic prompt like "12 + 89", any explanatory text it outputs could be expected to be in Mandarin most of the time. According to this test, Xiaomi's MiMo-7B-RL is an English-first model.
  - overfeed 1y ago
    "12 + 89" uses the latin alphabet and is in no way language-agnostic in this context. I expect borrowed constructs to appear relatively more frequently in the language they were borrowed from. Now I'm curious how Mistral models would respond to a "language-agnostic" phrases like "Rendezvous" or "coup d'etat"
    - yorwba 1y ago
      You may think of these symbols as "Latin" because they're how people writing in Latin script happen to write mathematical expressions, but the exact same symbols are also used by Mandarin speakers, as well as in numerous other scripts. Writing math in Chinese characters is literally as uncommon as someone writing "twelve plus eighty-nine" in English. In contrast, your examples would be spelled « rendez-vous » and « coup d’État » in French, i.e. easily distinguishable from their English descendants.
      - overfeed 1y ago
        > You may think of these symbols as "Latin" because they're how people writing in Latin script happen to write mathematical expressions No need for scare-quotes, Latin script is a proper noun and a technical term with precise meaning wrt text encoding - not "what I think." > the exact same symbols are also used by Mandarin speakers, as well as in numerous other scripts. Writing math in Chinese Which unicode code points do the Mandarin speakers and "numerous other scripts" use to write "12 + 89"? Could it be the very same code points as Latin script, which then are tokenized to the same vectors that the LLMs learn to associate more with English text rather than CJK in the latent space? > i.e. easily distinguishable from their English descendants. You're making broad assumptions about the tokenization design here that do not apply universally.
        yorwba 1y ago
        Precisely because the exact same codepoints are used for digits and mathematical symbols, there's nothing script-specific about them and their linguistic association is determined by the training data mixture. A model trained predominantly on text scraped from Chinese websites would learn to associate them more with Mandarin than English in the latent space, since that would be the context where they most often appear.

overfeed 1y ago

> That's clearly a different question. It'd be possible for these models to be Mandarin-first while still supporting other languages What would a hypothetical "Mandarin-first" model look like to you? I challenge the notion that the current models are "English-first" - that is an unsubstantiated opinion not supported by fact. I bet, dollars to donuts, these models are SoTA in Mandarin as well. When framed that way, asking "Why are they marketed as English-speaking models outside of China" or "Why are they really good at English" are simply not interesting questions - they have obvious answers.

yorwba 1y ago
> What would a hypothetical "Mandarin-first" model look like to you? Given a language-agnostic prompt like "12 + 89", any explanatory text it outputs could be expected to be in Mandarin most of the time. According to this test, Xiaomi's MiMo-7B-RL is an English-first model.
- overfeed 1y ago
  "12 + 89" uses the latin alphabet and is in no way language-agnostic in this context. I expect borrowed constructs to appear relatively more frequently in the language they were borrowed from. Now I'm curious how Mistral models would respond to a "language-agnostic" phrases like "Rendezvous" or "coup d'etat"
  - yorwba 1y ago
    You may think of these symbols as "Latin" because they're how people writing in Latin script happen to write mathematical expressions, but the exact same symbols are also used by Mandarin speakers, as well as in numerous other scripts. Writing math in Chinese characters is literally as uncommon as someone writing "twelve plus eighty-nine" in English. In contrast, your examples would be spelled « rendez-vous » and « coup d’État » in French, i.e. easily distinguishable from their English descendants.
    - overfeed 1y ago
      > You may think of these symbols as "Latin" because they're how people writing in Latin script happen to write mathematical expressions No need for scare-quotes, Latin script is a proper noun and a technical term with precise meaning wrt text encoding - not "what I think." > the exact same symbols are also used by Mandarin speakers, as well as in numerous other scripts. Writing math in Chinese Which unicode code points do the Mandarin speakers and "numerous other scripts" use to write "12 + 89"? Could it be the very same code points as Latin script, which then are tokenized to the same vectors that the LLMs learn to associate more with English text rather than CJK in the latent space? > i.e. easily distinguishable from their English descendants. You're making broad assumptions about the tokenization design here that do not apply universally.
      - yorwba 1y ago
        Precisely because the exact same codepoints are used for digits and mathematical symbols, there's nothing script-specific about them and their linguistic association is determined by the training data mixture. A model trained predominantly on text scraped from Chinese websites would learn to associate them more with Mandarin than English in the latent space, since that would be the context where they most often appear.