4 ms·
That could be how it works, but in practice it takes into account all previous tokens when producing the next-token distribution to sample from. So a switch bac
by cadamsdotcom 2mo ago
That could be how it works, but in practice it takes into account all previous tokens when producing the next-token distribution to sample from. So a switch back is more likely than your explanation supposes.
- inigyou 2mo agoNo, if you switched to Japanese the LLM wouldn't ignore it, it would "assume" there's a reason for that. The same if the previous iteration of the LLM switched to Japanese. Else you're expecting an LLM to ignore its own previous outputs and restart "thinking" from scratch with every token?
- cadamsdotcom 2mo agoIt's situational and I suspect there are situations where it would and others where it wouldn't. Would depend on the almost infinite variables of how training was done. You'd be right that it'd be likely to switch but while it's possible it's due to temperature, there are just so many things going on. But it would be one sensible explanation among many.