4 ms·

I have a similar experience using Gemini for quick Catalan translations for iOS apps given enough context. I once asked it to summarize The Hobbit in Catalan t

I have a similar experience using Gemini for quick Catalan translations for iOS apps given enough context. I once asked it to summarize The Hobbit in Catalan to explain it to my daughter before sleep. I was expecting a lot of mistakes as I see regularly if I ask anything in my native language when using GPT or Claude, but it was surprisingly good. I was going just to kind of skim ahead and retell it my own way, but ended up almost saying it verbatim because it was good already. She loves Zelda so I asked it to explain the story of Breath of The Wild keeping the original names, and to make it fun, etc.. I was surprised again. I did retell some bits in my own style and taste but it is very convincing. I haven't tried Catalan on newer models like GTP-6 Astra or Fable tho. We have all these benchmarks based on software development, and AGI, etc.. but it would be cool to have some language benchmarks for different communities. As I work in english and use them in english, I wonder if using LLMs in a different language to code renders a different result as well. Like, if some of these benchmarks were made in other languages, would the result be similar.

jaggederest 17d ago
> As I work in english and use them in english, I wonder if using LLMs in a different language to code renders a different result as well. Like, if some of these benchmarks were made in other languages, would the result be similar. I am not a Chinese speaker but my understanding is that all of the models have substantially different behavior in Chinese, to the degree that it's kind of like a second model. Would be interested in hearing more if anyone has direct experience.
- mncharity 17d ago
  Fwiw, it's been very easy to nudge Qwen 27B/35B to think in Chinese. No need for <think> prefills like "思考：" ("Think:") or such. Relatedly, when chatting with frontier models about science education content design, I've found it very helpful to mix in Chinese education terms. In English, for example, NGSS is such a massive attractor, discussing nearby topics often yields NGSS "slop". And "estimation" (educational) in the US means one (dysfunctional:) thing, which similarly distracts. Perhaps if AIs become increasingly multilingual, but remain weak at deep conceptual reasoning, it may be fruitful to have multilingual thesauruses, to use language-associated cultural conceptual differences as a way to convey conceptual nuances with which LLMs otherwise struggle?
  - eru 17d ago
    > In English, for example, NGSS is such a massive attractor, discussing nearby topics often yields NGSS "slop". For that particular attractor, even German or Spanish or so might help avoid it? Or perhaps even just using British English?
    - mncharity 16d ago
      Oh, good point! Something to check. I've been thinking of Chinese as a strength of Chinese models, but other languages might be sufficient here.
  - normie3000 17d ago
    This is an interesting post, but as a non-American it's hard to understand what's it's alluding to in some parts. The one criticism of NGSS I found from a quick skim of https://en.wikipedia.org/wiki/Next_Generation_Science_Standards https://en.wikipedia.org/wiki/Next_Generation_Science_Standa... is that it teaches evolution. What does "estimation" mean in the US?!
    - mncharity 16d ago
      > What does "estimation" mean in the US?! "Estimation" as a skill in US education, primary school on, is pervasively about point estimates - single numbers. Bounding, a range of safe/reasonable/likely/possible numbers, is very rare. In Fermi questions/problems also. Even in such goodness as Sanjoy Mahajan's Art of Insight and Street-Fighting[1]... the only only "bound" mentioned is "Printed and bound in [USA]", and there is no "round".[2] In contrast, IIUC, China is pervasively about "Da Gu" and "Xiao Gu", "round up" and "round down". Point estimates are secondary. Lower and upper bounds are apparently even a cultural concept: an individual's degree vs capability/fate; policy equity vs excellence; economic targets. In education, my understanding is point estimates have much poorer conversational/collaborative dynamics than range estimates. Bounding lends itself to incremental collaboration. "Can anyone suggest another low or high bound?". Discussions around point estimates seem usually merely about the collection of individual estimates. > NGSS Ah, NGSS isn't bad particularly. I meant that discussion/mention of NGSS appears so very much in English training data, that AI chat on topics related to NGSS seem to draw regurgitations of NGSS. AI "slop" in the sense that "ignore NGSS, just reason from a superset of its underlying concepts"... isn't an LLM strength. > hard to understand Sorry - Thanks for asking! [1] open access: https://direct.mit.edu/books/oa-monograph/5345/The-Art-of-Insight-in-Science-and https://direct.mit.edu/books/oa-monograph/5345/The-Art-of-In... https://direct.mit.edu/books/oa-monograph/5339/Street-FightingMathematicsThe-Art-of-Educated https://direct.mit.edu/books/oa-monograph/5339/Street-Fighti... [2] search "bound": https://www.google.com/books/edition/The_Art_of_Insight_in_Science_and_Engine/xRgeBQAAQBAJ?hl=en&gbpv=1&bsq=%20bound https://www.google.com/books/edition/The_Art_of_Insight_in_S... https://www.google.com/books/edition/Street_Fighting_Mathematics/VrkZN0T0GaUC?hl=en&gbpv=1&bsq=bound https://www.google.com/books/edition/Street_Fighting_Mathema...

jaggederest 17d ago

> As I work in english and use them in english, I wonder if using LLMs in a different language to code renders a different result as well. Like, if some of these benchmarks were made in other languages, would the result be similar. I am not a Chinese speaker but my understanding is that all of the models have substantially different behavior in Chinese, to the degree that it's kind of like a second model. Would be interested in hearing more if anyone has direct experience.

mncharity 17d ago
Fwiw, it's been very easy to nudge Qwen 27B/35B to think in Chinese. No need for <think> prefills like "思考：" ("Think:") or such. Relatedly, when chatting with frontier models about science education content design, I've found it very helpful to mix in Chinese education terms. In English, for example, NGSS is such a massive attractor, discussing nearby topics often yields NGSS "slop". And "estimation" (educational) in the US means one (dysfunctional:) thing, which similarly distracts. Perhaps if AIs become increasingly multilingual, but remain weak at deep conceptual reasoning, it may be fruitful to have multilingual thesauruses, to use language-associated cultural conceptual differences as a way to convey conceptual nuances with which LLMs otherwise struggle?
- eru 17d ago
  > In English, for example, NGSS is such a massive attractor, discussing nearby topics often yields NGSS "slop". For that particular attractor, even German or Spanish or so might help avoid it? Or perhaps even just using British English?
  - mncharity 16d ago
    Oh, good point! Something to check. I've been thinking of Chinese as a strength of Chinese models, but other languages might be sufficient here.
- normie3000 17d ago
  This is an interesting post, but as a non-American it's hard to understand what's it's alluding to in some parts. The one criticism of NGSS I found from a quick skim of https://en.wikipedia.org/wiki/Next_Generation_Science_Standards https://en.wikipedia.org/wiki/Next_Generation_Science_Standa... is that it teaches evolution. What does "estimation" mean in the US?!
  - mncharity 16d ago
    > What does "estimation" mean in the US?! "Estimation" as a skill in US education, primary school on, is pervasively about point estimates - single numbers. Bounding, a range of safe/reasonable/likely/possible numbers, is very rare. In Fermi questions/problems also. Even in such goodness as Sanjoy Mahajan's Art of Insight and Street-Fighting[1]... the only only "bound" mentioned is "Printed and bound in [USA]", and there is no "round".[2] In contrast, IIUC, China is pervasively about "Da Gu" and "Xiao Gu", "round up" and "round down". Point estimates are secondary. Lower and upper bounds are apparently even a cultural concept: an individual's degree vs capability/fate; policy equity vs excellence; economic targets. In education, my understanding is point estimates have much poorer conversational/collaborative dynamics than range estimates. Bounding lends itself to incremental collaboration. "Can anyone suggest another low or high bound?". Discussions around point estimates seem usually merely about the collection of individual estimates. > NGSS Ah, NGSS isn't bad particularly. I meant that discussion/mention of NGSS appears so very much in English training data, that AI chat on topics related to NGSS seem to draw regurgitations of NGSS. AI "slop" in the sense that "ignore NGSS, just reason from a superset of its underlying concepts"... isn't an LLM strength. > hard to understand Sorry - Thanks for asking! [1] open access: https://direct.mit.edu/books/oa-monograph/5345/The-Art-of-Insight-in-Science-and https://direct.mit.edu/books/oa-monograph/5345/The-Art-of-In... https://direct.mit.edu/books/oa-monograph/5339/Street-FightingMathematicsThe-Art-of-Educated https://direct.mit.edu/books/oa-monograph/5339/Street-Fighti... [2] search "bound": https://www.google.com/books/edition/The_Art_of_Insight_in_Science_and_Engine/xRgeBQAAQBAJ?hl=en&gbpv=1&bsq=%20bound https://www.google.com/books/edition/The_Art_of_Insight_in_S... https://www.google.com/books/edition/Street_Fighting_Mathematics/VrkZN0T0GaUC?hl=en&gbpv=1&bsq=bound https://www.google.com/books/edition/Street_Fighting_Mathema...