4 ms·
> I want to see some assurance we get the same results every time Genuine question, but why not set the temperature to 0? I do this for non-code related infere
by properbrew 8mo ago
> I want to see some assurance we get the same results every time
Genuine question, but why not set the temperature to 0? I do this for non-code related inference when I want the same response to a prompt each time.
- willj 8mo agoA temperature of 0 doesn’t result in the same responses every time, in part due to floating-point precision and in part to to lack of batch invariance [1] [1] https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
- properbrew 8mo agoThank you for this, this was a really interesting read about batch invariance, something I didn't even know about.
- assbuttbuttass 8mo agoThis still doesn't help when you update your compiler to use a newer model