4 ms·

Someone apparently did observe ChatGPT (I think it was ChatGPT) switch to Chinese for some parts of it's reasoning/calculations and then back to English for the

Someone apparently did observe ChatGPT (I think it was ChatGPT) switch to Chinese for some parts of it's reasoning/calculations and then back to English for the final answer. That's somehow even weirder than the LLM giving different answers depending on the input.

ApolloFortyNine 2y ago
I've seen this happen as well with o3-mini, but I'm honestly not sure what triggered it. I use it all the time but have only had it switch to Chinese during reasoning maybe twice.
- Telemakhos 2y ago
  I've seen Grok sprinkle random Chinese characters into responses I asked for in ancient Greek and Latin.
  - andai 2y ago
    I get strange languages sprinkled through my Gemini responses, including some very obscure ones. It just randomly changes language for one or two words.
    - genewitch 2y ago
      Is it possible the "vector" is more accurate in another language? Like espirit d'esclair or schadenfreude, or any number of other things that are a single word in a language but paragraphs or more in others?
      - sanxiyn 2y ago
        Possibly. I have seen Claude switching to Russian for a word or two when it is about revolution!
- numpad0 2y ago
  Isn't it just it getting increasingly incoherent as non-English data fraction increases? Last I checked, none of open weight LLMs has languages other than English as its sole dominant language represented in the dataset.
maxloh 2y ago
> the LLM giving different answers depending on the input. LLMs are actually designed to have some randomness in their responses. To make the answer reproducible, set the temperature to O (eliminating randomness) and provide a static seed (ensuring consistent results) in the LLM's configuration.
- lolinder 2y ago
  It's not necessary in most inference engines I've seen to set the temperature to 0—the randomness in the temperature is drawn from the seed, so a static seed will work for any temperature.
- jll29 2y ago
  The influence of the (pseudo-)random number generator is called "temperature" in most models. Setting it to 0 in theory eliminates all randomness, and instead of choosing one from a list of next words that may be predicted, always only the MOST PROBABLY word would be chosen. However, in practice, setting the temperature to 0 in most GUIs does not actually set the temperature to 0, but to a "very small" value ("epsilon"), the reason being to avoid a division by zero exception/crawsh in a mathematical formula. So don't be surprised if you cannot get rid of random behavior entirely.
  - Asraelite 2y ago
    > the reason being to avoid a division by zero exception/crawsh in a mathematical formula Why don't they just special-case it?
laurent_du 2y ago
Reminds me of this funny video: https://www.youtube.com/watch?v=NY3yWXWjYjA https://www.youtube.com/watch?v=NY3yWXWjYjA ("You know something has gone wrong when he switches to Chinese")
ricochet11 2y ago
i’ve seen that with deepseek
rzz3 2y ago
I saw Claude 3.7 write a comment in my code in Russian followed by, likely from a previous modification, the English text “Russian coding” for no reason.
jananas 2y ago
In had it doing the reasoning in Turkish and English despite the question being in German.

ApolloFortyNine 2y ago

I've seen this happen as well with o3-mini, but I'm honestly not sure what triggered it. I use it all the time but have only had it switch to Chinese during reasoning maybe twice.

Telemakhos 2y ago
I've seen Grok sprinkle random Chinese characters into responses I asked for in ancient Greek and Latin.
- andai 2y ago
  I get strange languages sprinkled through my Gemini responses, including some very obscure ones. It just randomly changes language for one or two words.
  - genewitch 2y ago
    Is it possible the "vector" is more accurate in another language? Like espirit d'esclair or schadenfreude, or any number of other things that are a single word in a language but paragraphs or more in others?
    - sanxiyn 2y ago
      Possibly. I have seen Claude switching to Russian for a word or two when it is about revolution!
numpad0 2y ago
Isn't it just it getting increasingly incoherent as non-English data fraction increases? Last I checked, none of open weight LLMs has languages other than English as its sole dominant language represented in the dataset.

maxloh 2y ago

> the LLM giving different answers depending on the input. LLMs are actually designed to have some randomness in their responses. To make the answer reproducible, set the temperature to O (eliminating randomness) and provide a static seed (ensuring consistent results) in the LLM's configuration.

lolinder 2y ago
It's not necessary in most inference engines I've seen to set the temperature to 0—the randomness in the temperature is drawn from the seed, so a static seed will work for any temperature.
jll29 2y ago
The influence of the (pseudo-)random number generator is called "temperature" in most models. Setting it to 0 in theory eliminates all randomness, and instead of choosing one from a list of next words that may be predicted, always only the MOST PROBABLY word would be chosen. However, in practice, setting the temperature to 0 in most GUIs does not actually set the temperature to 0, but to a "very small" value ("epsilon"), the reason being to avoid a division by zero exception/crawsh in a mathematical formula. So don't be surprised if you cannot get rid of random behavior entirely.
- Asraelite 2y ago
  > the reason being to avoid a division by zero exception/crawsh in a mathematical formula Why don't they just special-case it?

laurent_du 2y ago

Reminds me of this funny video: https://www.youtube.com/watch?v=NY3yWXWjYjA https://www.youtube.com/watch?v=NY3yWXWjYjA ("You know something has gone wrong when he switches to Chinese")

ricochet11 2y ago

i’ve seen that with deepseek

rzz3 2y ago

I saw Claude 3.7 write a comment in my code in Russian followed by, likely from a previous modification, the English text “Russian coding” for no reason.

jananas 2y ago

In had it doing the reasoning in Turkish and English despite the question being in German.