3 ms·
This eval’s goal is a bit unclear to me, especially given the example questions. They’re very trivia/minutiae like asking about sports goals for example, which
by yunohn 2y ago
This eval’s goal is a bit unclear to me, especially given the example questions. They’re very trivia/minutiae like asking about sports goals for example, which is their stated desire to test factual knowledge. But will this ever be possible by an LLM, without web browsing - which they deliberately removed while evaluating?
- petesergeant 2y agoI think the interesting thing here is the difference between Not Attempt and Incorrect — the goal here seems to be to reduce hallucination
- yunohn 2y agoFrom that perspective, o1-mini seems to perform the best. But only as long as enabling web browsing makes up for lack of base factuality.
- sbierwagen 2y ago>But will this ever be possible by an LLM? Why not? Just train an unbelievably gigantic LLM that encodes all human knowledge. A hundred trillion parameters ought to do it.