4 ms·
Last time I checked, modern LLMs could give you a different answer for the same sequence of prompts each time. Did that change?
by xigoi 1y ago
Last time I checked, modern LLMs could give you a different answer for the same sequence of prompts each time. Did that change?
- perching_aix 1y agoThey can, but won't necessarily. If you use a managed service, they likely will, due to batched inference. Otherwise, it's simply a matter of configuring the seed to a fixed value and the temperature to 0. At least that's what I did, and then as long as the prompts were exactly the same, the responses remained exactly the same too. Tested with a quantized gemma3 using ollama, I'd say that's modern enough (barely a month or so old). Maybe lowering the temp is not even necessary as long as you keep the seed stickied, didn't test that.
- Workaccount2 1y agoJust a note for everyone, people do these fixed prompts with fixed seed and 0-temp tests to track model versions. A cool application of the technique
- perching_aix 1y agoSmall update: now that I brought it up, I figured I should just test it. And indeed, as long as the seed is pinned, the replies are bit-for-bit identical run to run (note that the conversation context was cleared before each attempt, of course). So even setting the temp to 0 is not actually needed. This is handy in case somebody makes the claim that the randomness (nonzero temp parameter) makes a model perform better. The devil really is in the "insignificant for humans but significant for the model" details, basically. Not in the computational determinism.
- xigoi 1y agoA random number generator also gives deterministic results when you fix the seed. Does that make it not a random number generator?
- perching_aix 1y agoYes, it does. They're called pseudo-random number generators (PRNGs) in literature for that very reason instead.