3 ms·
By virtue of increasing randomness, we got the correct answer once ... a monkey at a typewriter will also spit out the correct answer occasionally. Temperature
by throwaway63820 3y ago
By virtue of increasing randomness, we got the correct answer once ... a monkey at a typewriter will also spit out the correct answer occasionally. Temperature 0 is the correct evaluation.
- scrollop 3y agoHere's a quick analysis of the model vs it's peers: https://www.youtube.com/watch?v=ReO2CWBpUYk https://www.youtube.com/watch?v=ReO2CWBpUYk
- scrollop 3y agoSo your theory would have it that if you repeated the question at temp 1 it would give the wrong answer more often than the correct answer?
- throwaway63820 3y agoThere's no theory. Just in real life usage, it is extremely uncommon to stochastically query the model and use the most common answer. Using it with temperature 0 is the "best" answer as it uses the most likely tokens in each completion.
- moffkalast 3y ago> Temperature 0 is the correct evaluation. In theory maybe, but I don't think it is in practice. It feels like each model has its own quasi-optimal temperature and other settings at which it performs vastly better. Sort of like a particle filter that must do random sampling to find the optimal solution.