3 ms·
We need results from these harder/different benchmarks which give pretty bad scores to current top LLMs. https://www.wolfram.com/llm-benchmarking-project/ http
by smusamashah 2y ago
We need results from these harder/different benchmarks which give pretty bad scores to current top LLMs.
https://www.wolfram.com/llm-benchmarking-project/ https://www.wolfram.com/llm-benchmarking-project/
https://help.kagi.com/kagi/ai/llm-benchmark.html https://help.kagi.com/kagi/ai/llm-benchmark.html
Edit : There are few other benchmarks that give pretty low scores (<20%) to top LLMs. Can't find them atm. There was a benchmark with common sense easy looking questions.
Edit: found two more papers
https://arxiv.org/html/2405.19616 https://arxiv.org/html/2405.19616
https://arxiv.org/html/2406.02061v1 https://arxiv.org/html/2406.02061v1
Edit: How about Wordle?
https://www.strangeloopcanon.com/p/what-can-llms-never-do https://www.strangeloopcanon.com/p/what-can-llms-never-do
https://news.ycombinator.com/item?id=40179232 https://news.ycombinator.com/item?id=40179232
- freediver 2y agoI am happy to run the tests on Kagi LLM benchmark. Is there an API endpoint for this model anywhere?
- botro 2y ago"The task consists of going from English-language specifications to Wolfram Language code. The test cases are exercises from Stephen Wolfram's An Elementary Introduction to the Wolfram Language." I think this benchmark would really only tell me whether Wolframs book was in the training data.
- smusamashah 2y agoYeah, may be should skip that benchmark.
- bufferoverflow 2y agoIt's available online in HTML form, for free: https://www.wolfram.com/language/elementary-introduction/3rd-ed/01-starting-out-elementary-arithmetic.html https://www.wolfram.com/language/elementary-introduction/3rd...
- viraptor 2y agoThere's always the new sets from Leaderboard v2 https://huggingface.co/spaces/open-llm-leaderboard/blog https://huggingface.co/spaces/open-llm-leaderboard/blog
- thethirdone 2y agoThe sample answers for the horse race question are crazy. [0] Pretty much all the LLM really want to split 6 horses into two groups of three. Only LLAMA 3 makes the justification that only 2 horses can be raced at a time, but then gets its modified question wrong by racing three horses. I personally would consider an answer that presumes some restriction to how the horses can be raced to be valid if it answers the restricted version correctly. [0]: https://arxiv.org/html/2405.19616v2#S9.SS2.SSS1 https://arxiv.org/html/2405.19616v2#S9.SS2.SSS1