3 ms·
I also tried two hallucinations that I've seen in the latest Gemini and Claude. The first one is the famous "how many r in strawberry" and another one is asking
by adriantoine 11mo ago
I also tried two hallucinations that I've seen in the latest Gemini and Claude. The first one is the famous "how many r in strawberry" and another one is asking about king sacrifice in chess where most LLMs tell you it's a very powerful move and tell you how to do it (obviously a nonsense since the point of the game is to protect the king).
Kagi actually got those two correct, explaining how losing your king means losing your chess game, which is quite impressive.
- sometimes_all 11mo agoI tried both these prompts (along with the SINAC one as per GP) in Sonnet 4.5 and Gemini 3, and they both answered correctly for all three. Both also provided context on the chess question as well.
- VHRanger 11mo agoAll of this will depend on the settings on the model (reasoning effort, temperature, top_k,etc) as well. Which is why you should have benchmarks that are a bit broader generally (>10 questions for a personal setup) otherwise you overfit to noise