2 ms·
You know you can just try it and see on any inference system thst has this knob, right? Related: if you don't have a limit on sampling (top-K or top-P), eventu
by inigyou 3mo ago
You know you can just try it and see on any inference system thst has this knob, right?
Related: if you don't have a limit on sampling (top-K or top-P), eventually you'll hit one of the really unlikely tokens by chance and then the model will switch to Japanese because the most likely completion after a random Japanese character in the middle of an English sentence is more Japanese writing, not a reversal back to English.
- cadamsdotcom 3mo agoThat could be how it works, but in practice it takes into account all previous tokens when producing the next-token distribution to sample from. So a switch back is more likely than your explanation supposes.
- inigyou 3mo agoNo, if you switched to Japanese the LLM wouldn't ignore it, it would "assume" there's a reason for that. The same if the previous iteration of the LLM switched to Japanese. Else you're expecting an LLM to ignore its own previous outputs and restart "thinking" from scratch with every token?
- cadamsdotcom 3mo agoIt's situational and I suspect there are situations where it would and others where it wouldn't. Would depend on the almost infinite variables of how training was done. You'd be right that it'd be likely to switch but while it's possible it's due to temperature, there are just so many things going on. But it would be one sensible explanation among many.