3 ms·

Citation needed

• inigyou 2mo ago
  You know you can just try it and see on any inference system thst has this knob, right? Related: if you don't have a limit on sampling (top-K or top-P), eventually you'll hit one of the really unlikely tokens by chance and then the model will switch to Japanese because the most likely completion after a random Japanese character in the middle of an English sentence is more Japanese writing, not a reversal back to English.
  • cadamsdotcom 2mo ago
    That could be how it works, but in practice it takes into account all previous tokens when producing the next-token distribution to sample from. So a switch back is more likely than your explanation supposes.
    • inigyou 2mo ago
      No, if you switched to Japanese the LLM wouldn't ignore it, it would "assume" there's a reason for that. The same if the previous iteration of the LLM switched to Japanese. Else you're expecting an LLM to ignore its own previous outputs and restart "thinking" from scratch with every token?
      • cadamsdotcom 2mo ago
        It's situational and I suspect there are situations where it would and others where it wouldn't. Would depend on the almost infinite variables of how training was done. You'd be right that it'd be likely to switch but while it's possible it's due to temperature, there are just so many things going on. But it would be one sensible explanation among many.