If I'm reading this correctly, they had to discard Llama 2 answers and only use GPT-3.5 given answers to test the hypothesis. GPT-3.5 answering questions throu

If I'm reading this correctly, they had to discard Llama 2 answers and only use GPT-3.5 given answers to test the hypothesis. GPT-3.5 answering questions through the OAI API alone is not an acceptable method of testing problem solving ability across a range of temperatures. OpenAI does some blackbox wizardry on their end. There are many complex and clever sampling techniques for which temperature is just one (possibly dynamic) component One example from the llama.cpp codebase is dynamic temperature sampling https://github.com/ggerganov/llama.cpp/pull/4972/files https://github.com/ggerganov/llama.cpp/pull/4972/files Not sure what you mean by whole model state given that there are tens of thousands of possible tokens and the models have billions of parameters in XX,XXX-dimensional space. How many queries across how many sampling methods might you need? Err..how much time? :)