3 ms·
Kimi doesn't do well on my "ask a trivia question that other AIs get wrong" test. The question it came up with, "which U.S. state is closest to Africa?" is a p
by seizethecheese 3mo ago
Kimi doesn't do well on my "ask a trivia question that other AIs get wrong" test.
The question it came up with, "which U.S. state is closest to Africa?" is a pretty standard trivia question without any reason to believe other AIs would get confused. https://pellmell.ai/s/dccdeca69f929f79bc89317035610049 https://pellmell.ai/s/dccdeca69f929f79bc89317035610049
Even GPT-OSS-120b gets this right: https://pellmell.ai/s/1a43dfc7a3baa214aa0fa1b95d2c536a https://pellmell.ai/s/1a43dfc7a3baa214aa0fa1b95d2c536a
- anigbrowl 3mo agoAre you giving it your API for these other AIs to evaluate their responses? This 'test' seems perverse.
- seizethecheese 3mo agoI don't understand the question. The other AIs don't see the question until they are asked to react.
- anigbrowl 3mo agoSorry, I should have said 'API key'. What I mean is, why do you consider it a reasonable test for an AI to guess what others AIs don't know?
- tossandthrow 3mo agoThese types of tests are kind of moot as agentic harnesses are taking over. IMHO an Ai is the llm plus it's harness. A good harness would allow the llm to investigate on a map. Just like the llm can use a python script to figure out how many r's there are in strawberry. These tests are simply not that predictable of performance of the llm.
- seizethecheese 3mo agoThe test here is not how close the state is to Africa, the test is coming up with a question that is hard for other AIs to answer.
- bel8 3mo agoI don't see how that invalidates their point. Besides, n=1 benchmark seems like more of a coin toss.