> If the tokens are bit-for-bit-identical, where does the non-determinism come in? By design, most LLM’s have a randomization factor to their model. Some use t

> If the tokens are bit-for-bit-identical, where does the non-determinism come in? By design, most LLM’s have a randomization factor to their model. Some use the concept of “temperature” which makes them randomly choose the 2nd or 3rd highest ranked next token, the higher the temperature the more often/lower they pick a non-best next token. OpenAI described this in their papers around the GPT-2 timeframe IIRC.