3 ms·
It's more nuanced than that, in practice: this is true for the shims you see from API providers (ex. OpenAI, Anthropic, Mistral). With llama.cpp, it's actually
by refulgentis 3y ago
It's more nuanced than that, in practice: this is true for the shims you see from API providers (ex. OpenAI, Anthropic, Mistral).
With llama.cpp, it's actually not a great idea to have temperature purely at 0: in practice, especially with smaller models, this leads to pure repeating or nonsense.
I can't remember where I picked this up, but, a few years back, without _some_ randomness, the next likely token was always the last token.