3 ms·
Two of the models used have retrieval capabilities and have access to newer information through search. The other three are parametric.
by kostaj 5mo ago
Two of the models used have retrieval capabilities and have access to newer information through search. The other three are parametric.
- furyofantares 5mo agoYes, so in that case you set them up to disagree and then measured disagreement.
- simonw 5mo agoComparing models with search tools to models without - when there's no option for "I am unable to answer this question without access to search" - doesn't make sense to me.
- kostaj 5mo agoAgree about comparing models with and without search capabilities. Even the two models with search capabilities (Sonar Pro and Gemini) agree only on 58% of the claims.
- throw310822 5mo agoThe title mention "fact-checks", but "fact checking" is a process in which facts are checked against sources, not one where you are given a random fact and have to tell if it's true or false from your own memory. That's what is normally called a quiz game. So a more honest title for this research would be "Models answer differently to quiz questions".
- jiggawatts 4mo ago> "Models answer differently to quiz questions". from their future It's as if you asked a human in 2026 to "fact check" something from 2027. You're going to get an educated guess at best, a coin flip at worst.