11 ms·
Advancing AI Benchmarking with Game Arena
- eamag 9mo agoCurious why they decided to curate poker hands instead of a normal poker
- qsort 9mo agoPoker has very high variance, you'd need several hundred thousand hands to confidently say who's better. Also, you probably want to precompute the GTO-optimal play for benchmarking purposes.
- tiahura 9mo agoHow about nethack?
- tux3 9mo agoFor reference for anyone who missed it, the 2021 NetHack challenge results: https://nethackchallenge.com/report.html https://nethackchallenge.com/report.html That was a whole half a decade ago, but back then deep learning AIs were defeated very badly by handcrafted scripts. Even the best bot in the neural net category was actual a symbolic script/neural net hybrid.
- tiahura 9mo agoIt’s my understanding that ai has had some advances in the last 5 years.
- chaostheory 9mo agoAnecdotal data point, but recently I’ve found Gemini to perform better than ChatGPT when it came to intent analysis.
- ofirpress 9mo agoThis is a good way to benchmark models. We [the SWE-bench team] took the meta-version of this and implemented it as a new benchmark called CodeClash - We have agents implement agents that play games against each other- so Claude isn't playing against GPT, but an agent written by Claude plays poker against an agent written by GPT, and this really tough task leads to very interesting findings on AI for coding. https://codeclash.ai/ https://codeclash.ai/
- riku_iki 9mo agoLeaderboard looks very outdated..
- Instantnoodl 9mo agoCool to see core war! I feel it's mostly forgotten by now. My dad is still playing it to this day though and even attends tournaments
- 63stack 9mo ago>this really tough task leads to very interesting findings on AI for coding Are you going to share those with the class or?
- RobRivera 9mo agohttps://ai.meta.com/research/publications/gaia-a-benchmark-for-general-ai-assistants/ https://ai.meta.com/research/publications/gaia-a-benchmark-f... ?
- cv5005 9mo agoMy personal threshold for AGI is when an AI can 'sit down' - it doesn't need to have robotic hands, but it needs to only use visual and audio inputs to make its moves - and complete a modern RPG or FPS single player game that it hasn't pre-trained on (it can train on older games).
- bob1029 9mo agohttps://arxiv.org/abs/2507.03793 https://arxiv.org/abs/2507.03793
- anematode 9mo agoIsn't this a bit too visual-centric? By this criterion Helen Keller, author of 14 books, would not be generally intelligent. Ultimately I think it's impossible to define AGI. Maybe "I know it when I see it"—except everyone sees it at a different point (evidently).
- jamilton 9mo agoIt could have hands that feel but no vision, I think they were getting at that they thought embodiment and playing games in the modality of humans, without thousands of hours of play to reach competency, would be an important milestone.
- 10xDev 9mo agoIf AI can program, why does it matter if it can play Chess using CoT when it can program a Chess Engine instead? This applies to other domains as well.
- Davidzheng 9mo agoThey should be allowed to! In fact i think better benchmark would be to invent new games and test the models ability to allocate compute to minmax/alphazero new games in compute constraints
- simianwords 9mo agoIts the same reason we are asked to write exams without using calculators but the real world does have them. How you work without calculators is a proxy for real world competency.
- 10xDev 9mo agoFunny, you used probably the most useless form of benchmarking used on people as an example of measuring "competency" in the real world.
- simianwords 9mo agoare you in favour of children using calculators in exams?
- 10xDev 9mo agoIt is a program. I need it to get task X done and I don't care how, whether it is strictly through CoT or with tools. There is no such thing as cheating in real work and no reason to handicap it. Just test the limits of what it can do with whatever means possible. Trying to solve everything with CoT alone without utilising tools seems futile.
- simianwords 9mo ago
- simianwords 9mo agoGemini tops all benchmarks but when it comes to real world usage it is genuinely unusable
- goniszewski 9mo agoIt’s not that bad. I’ve been using 3 Pro for some time now and I’m quite happy with how it works. Best paired with Opus and Codex, like most models, but it’s solid as a full-stack buddy.
- CuriouslyC 9mo agoIt's legit good at visual stuff. It's not just a great agent and does some weird stuff sometimes.
- bennyfreshness 9mo agoWow. I'm generally in the AI maximalist camp. But adding Werewolf feels dangerous to me. Anyone who's played knows lying, deceipt, and manipulation is often key to winning. We really want models climbing this benchmark?
- bilekas 9mo agoGood question, but who's going to stop them? AI already has a very creative imagination for role play so this just adds extra to their arsenal.
- PunchyHamster 9mo agoconfidently and charismatically lying to clueless users has been one of fundaments of AI adoption
- rustyhancock 9mo agoOddly in the highlighted game I watched the werewolf simply gives up in the last round and says I'm the werewolf well-done... Vote me. Bizarre.
- minihat 9mo agoThis is a legitimate strategy for the werewolf, no?
- rustyhancock 9mo agoProbably not in this case. There were two villagers and one werewolf. The werewolf started the round by saying I'm the werewolf vote for me and then the game ended with a villager win. Over night he had successfully taken out the doctor. It made no sense in my opinion. There were some funny bits like on of the Anthropics models forgetting a rule and leading to everyone accusing him of being a werewolf in a pile on. He wasn't a werewolf he genuinely forgot the rule. Happens nearly every human game of werewolf.
- Rastonbury 9mo ago
- ZeroCool2u 9mo agoI'd really like to see them add a complex open world fully physicalized game like Star Citizen (assuming the game itself is stable) with a single primary goal like accumulating currency as a measure of general autonomy and a proxy for how the model might behave in the real world given access to a bipedal robot.
- PunchyHamster 9mo agomaking models target benchmark about being good at lying and getting away with it (werewolf) is certainly an interesting choice
- iNic 9mo agoI feel uneasy about werewolf being included here. I don't want AI labs to actively try and make their LLMs deceptive!
- mclau153 9mo agoClaude plays Pokemon Red
- kenforthewin 9mo agoLet's add NetHack to the mix! https://kenforthewin.github.io/blog/posts/nethack-agent/ https://kenforthewin.github.io/blog/posts/nethack-agent/
- mohsen1 9mo agoOh hey, I've been running Werewolves/Mafia games as benchmarks for a while now https://mafia-arena.com https://mafia-arena.com Gemini is consistently winning against top models
- deyiao 9mo agoI believe that if a model can outperform humans in all board/card games, and can autonomously complete all video games, then AGI — or even ASI — has essentially been achieved. We’re still a long way from that.
- jjcm 9mo agoThis was effectively what OpenAI did in the very early days with Dota 2: https://en.wikipedia.org/wiki/OpenAI_Five https://en.wikipedia.org/wiki/OpenAI_Five As someone who's been playing dota for nearly 20 years now, it was fascinating to watch it play. Some of it's decision making process didn't seem logical in the short term, but would often be set ups for future plays, even though their observation window was fairly small. Even more impressively was the ai bot changed the meta of professional players, since tactics that arose out of its training ended up being more optimal. I wish we got to the point where other ai bots were out there, but it's entirely understandable that you couldn't drive a complex game like Dota with LLMs, whereas you can with the ones the Game Arena has selected.
- Ntrails 9mo agoI'd still love to see them play more games, and indeed it'd be fun to play against them. Sad they died off