3 ms·
My understanding is that temperature applies to the output side and allows for some randomness in the next predicted token. Here Justine has constrained the mac
by none_to_remain 3y ago
My understanding is that temperature applies to the output side and allows for some randomness in the next predicted token. Here Justine has constrained the machine to start with either "yes" or "no" and to predict only one token. This makes the issue stark: leaving a non-zero temperature here would just add a chance of flipping a boolean.
- refulgentis 3y agoIt's more nuanced than that, in practice: this is true for the shims you see from API providers (ex. OpenAI, Anthropic, Mistral). With llama.cpp, it's actually not a great idea to have temperature purely at 0: in practice, especially with smaller models, this leads to pure repeating or nonsense. I can't remember where I picked this up, but, a few years back, without _some_ randomness, the next likely token was always the last token.