LLM's have a "temperature" value which defines how much randomness to allow in their answer. Like with everything LLM, this is basically a kludge that makes the

LLM's have a "temperature" value which defines how much randomness to allow in their answer. Like with everything LLM, this is basically a kludge that makes the performance better, even if it has little theoretical basis. So their results are non-deterministic and actually quite difficult to evaluate correctly at scale: feed responses to outputs into another model which tries to score it? Human score lots of responses? Raise the temperature enough to make it deterministic, score that, and presume that gives good enough results with a lower temperature?