4 ms·
Microsoft research’s team took advantage of this fact by querying multiple times per question (with randomized option ordering) and choosing the most common ans
by eightysixfour 3y ago
Microsoft research’s team took advantage of this fact by querying multiple times per question (with randomized option ordering) and choosing the most common answer.
Here is the paper:
https://www.microsoft.com/en-us/research/publication/can-generalist-foundation-models-outcompete-special-purpose-tuning-case-study-in-medicine/ https://www.microsoft.com/en-us/research/publication/can-gen...
- hlfshell 3y agoThis is driven by the idea that since LLMs are probabilistic engines during inference that, if the LLM "knows" or can "surmise" the correct answer, it will have a higher probability of being generated and thus self consistency methods can more reliably tease it out.