4 ms·
I found this other paper that tests Temperature: https://arxiv.org/abs/2402.05201 https://arxiv.org/abs/2402.05201 It appears that temperature has no impact on
by phire 3y ago
I found this other paper that tests Temperature: https://arxiv.org/abs/2402.05201 https://arxiv.org/abs/2402.05201
It appears that temperature has no impact on problem solving performance. So this paper isn't getting improved performance because the token for the correct answer is more probable.
My theory is that the multiple queries are allowing the whole probability space of possible answers to be sampled. Not just the probabilities of the most likely output token, but the probabilities of all possible internal model states.
And sampling that probability space of the whole model state and finding the average is a very different mathematical operation to just picking a single model state at random and then picking the most probable output tokens.
- mlsu 3y agoI wonder if there is a clever/more efficient shortcut that could come from before the sample is taken on each token. We have the logits after all.
- bt1a 3y agoIf I'm reading this correctly, they had to discard Llama 2 answers and only use GPT-3.5 given answers to test the hypothesis. GPT-3.5 answering questions through the OAI API alone is not an acceptable method of testing problem solving ability across a range of temperatures. OpenAI does some blackbox wizardry on their end. There are many complex and clever sampling techniques for which temperature is just one (possibly dynamic) component One example from the llama.cpp codebase is dynamic temperature sampling https://github.com/ggerganov/llama.cpp/pull/4972/files https://github.com/ggerganov/llama.cpp/pull/4972/files Not sure what you mean by whole model state given that there are tens of thousands of possible tokens and the models have billions of parameters in XX,XXX-dimensional space. How many queries across how many sampling methods might you need? Err..how much time? :)