10 ms·
So Deepseek gives out the correct answer the highest percentage of all SOTA models, yet is the least confident of all models?
by fakedang 2y ago
So Deepseek gives out the correct answer the highest percentage of all SOTA models, yet is the least confident of all models?
- myrmidon 2y agoThere is no text-only evaluation of the other models, though. The comparison might be completely invalid.
- famouswaffles 2y agoThere is actually. It's a bit buried. Section C.2 of the paper(page 24). R1 is still the best. o1 drops a little (8.9)
- sottol 2y agoI think it might mean the opposite of what one would expect. Afaict, calibration error means something along the lines of "how often was the model wrong but confident that the answer was correct". That means a low calibration error would be a good thing, ie the model correctly recognizes when it is unsure about answers instead of confidently stating the wrong answer.