5 ms·
Evaluating LLM Reasoning Through Live Computer Games
- deleted 2y ago[deleted]
- snyhlxde 2y agoChallenge yourself with latest reasoning LLMs and checkout our latest leaderboard!
- deleted 2y ago[deleted]
- zhisbug 2y agoThis is pretty clever and seems to have high potential, but it still relies on humans. What if some day all humans cannot outsmart AI?
- snyhlxde 2y agoWhen super intelligence comes, it would be very interesting to see multi-party game play among AI too. What role humans play in this story is unclear. Maybe humans can't directly engage in the games neither as they are too naive and will be immediately identified and exploited by AI :)
- elpocko 2y agoSo transparent. https://news.ycombinator.com/item?id=43017857 https://news.ycombinator.com/item?id=43017857 zhisbug 1 day ago | next [–] We hope to redefine ai evaluation via our gamified AI evaluation platform: game arena!
- meru_2025 2y agoA dynamic human-in-the-loop evaluation benchmark is great for preventing data contamination and test saturation. Worth my time to read.
- mino1234uiui27 2y agoThat is an interesting perspective that I hadn't previously considered, like using games to evaluate LLM.
- ginda307 2y agoJust played the game and it seems pretty fun - especially when you see the LLM speaking nonsense haha
- snyhlxde 2y agoIt's funny that reasoning models sometime speaking nonsense and perform worse than well-aligned models like claude-3.5-sonnet in multi-turn games like Akinator. I think it's one current weak point of applying longCoT RL vs. instruction-following alignment. Maybe we need to find a way to address both? Would be interesting to see some results
- wlsaidhi 2y agoIs it possible to setup a MLLM pipeline to play other roblox games and use that as another evaluation?
- snyhlxde 2y agoI think it's totally possible. Multimodal reasoning eval would be fun to consider too
- Yuxuan_Zhang13 2y agoI played the game and found hard mode to be an exciting challenge—it's incredibly fun, and the AI is so clever it even guessed my intentions in the taboo game!
- PY007 2y agoStatic evaluation --> chatbot arena --> game arena. Seems to be promising.
- jasonkongie 2y ago[dead]
- flaciplam 2y agogreat project
- leemack 2y agoCool, actually not boring and hard to play, good gamification