5 ms·
Impressed by the model so far. As far as independent testing goes, it is topping our leaderboard for chess puzzle solving by a wide margin now: https://github.
by freediver 2y ago
Impressed by the model so far. As far as independent testing goes, it is topping our leaderboard for chess puzzle solving by a wide margin now:
https://github.com/kagisearch/llm-chess-puzzles?tab=readme-ov-file#results https://github.com/kagisearch/llm-chess-puzzles?tab=readme-o...
- parhamn 2y agoIs the test set public?
- freediver 2y agoYes, in the repo.
- gengelbro 2y agoPossible it's in the training set then?
- mewpmewp2 2y agoGood point, would be interesting to have one public dataset and one hidden as well, just to see how scores compare, to understand if any of it might actually have got to a dataset somewhere.
- freediver 2y agoI'd be quite surprised if OpenAI took such a niche and small dataset into consideration. Then again...
- mewpmewp2 2y agoI would assume it goes over all the public github codebases, but no clue if there's some sort of filtering for filetypes, sizes or amount of stars on a repo etc.
- unbrice 2y agoAuthors note that this is probably the case: > we wanted to verify whether the model is actually capable of reasoning by building a simulation for a much simpler game - Connect 4 (see 'llmc4.py'). > When asked to play Connect 4, all LLMs fail to do so, even at most basic level. This should not be the case, as the rules of the game are simpler and widely available.
- bongodongobob 2y agoWouldn't there have to be historical matches to train on? Tons of chess games out there but doubt there are any connect 4 games. Is there even official notation for that? My assumption is that chatgpt can play chess because it has studied the games rather than just reading the rules.
- whimsicalism 2y agowould love if you could do multiple samples or even just resampling and get a boostrapped CI estimate
- Powdering7082 2y agoWow from adjusted ELO of 1144 to 1790, that's a huge leap. I wonder if they are giving it access to a 'scratch pad'
- muzani 2y agoMy guess is that handling visual stuff directly accidentally gives it some powers similar to Beth Harmon.
- mritchie712 2y agowoah, that's a huge leap, any idea why it's that large of a margin? using it in chat, it doesnt feel that different
- thrance 2y agoNice project! Are you aware of the following investigations: https://blog.mathieuacher.com/GPTsChessEloRatingLegalMoves/ https://blog.mathieuacher.com/GPTsChessEloRatingLegalMoves/ Some have been able to achieve greater elo with a different prompt based on the pgn format. gpt-3.5-turbo-instruct was able to reach an elo of ~1750.
- mewpmewp2 2y agoI see you have Connect 4 test there. I tried playing against the model, it didn't do well in terms of blocking my win. However it feels like it might be possible to make it try to think ahead in terms of making sure that all the threats are blocked by prompting well. Maybe that could lead to somewhere, where it will explain its reasoning first? This prompt worked for me to get it to block after I put 3 in the 4th column. It otherwise didn't Let's play connect 4. Before your move, explain your strategy concisely. Explain what you must do to make sure that I don't win in the next step, as well as explain what your best strategy would be. Then finally output the column you wish to drop. There are 7 columns. Always respond with JSON of the following format: type Response ={ am_i_forced_to_block: boolean; other_considerations: string[]; explanation_for_the_move: string; column_number: number; } I start with 4. Edit: So it went Me: 4 It: 3 Me: 4 It: 3 Me: 4 It: 4 - Successful block Me: 5 It: 3 Me: 6 - Intentionally, to see if it will win by putting another 3. It: 2 -- So here it failed, I will try to tweak the prompt to add more instructions. me: 4
- freediver 2y agoCare to add a PR?
- mewpmewp2 2y agoI just did it in the playground to test out actually, but it still seems to fail/lose state after some time. Right now where I got a win was after: [{ "who": "you", "column": 4 }, { "who": "me", "column": 3 }, { "who": "you", "column": 4 }, { "who": "me", "column": 2 }, { "who": "you", "column": 4 }, { "who": "me", "column": 4 }, { "who": "you", "column": 5 }, { "who": "me", "column": 6 }, { "who": "you", "column": 5 }, { "who": "me", "column": 1 }, { "who": "you", "column": 5 }, { "who": "me", "column": 5 }, { "who": "you", "column": 3 }] Where "me" was AI and "you" was I. It did block twice though. My final prompt I tested with right now was: Let's play connect 4. Before your move, explain your strategy concisely. Explain what you must do to make sure that I don't win in the next step, as well as explain what your best strategy would be. Then finally output the column you wish to drop. There are 7 columns. Always respond with JSON of the following format: type Response ={ move_history: { who: string; column: number; }[] am_i_forced_to_block: boolean; do_i_have_winning_move: boolean; other_considerations: string[]; explanation_for_the_move: string; column_number: number; } I start with 4. ONLY OUTPUT JSON
- elicksaur 2y ago> and Kagi is well positioned to serve this need. >CEO & founder of Kagi Important context for anyone like me who was wondering where the boldness of the first statement was coming from. Edit: looks like the parent has been edited to remove the claim I was responding to.
- freediver 2y agoYeah, it was an observation that was better suited for a tweet than HN. Here it is: https://twitter.com/vladquant/status/1790130917849137612 https://twitter.com/vladquant/status/1790130917849137612
- elicksaur 2y agoThanks for the transparency!
- muzani 2y agoMy favorite part of HN is just casually bumping into domain experts and celebrities without realizing it. No profile pic is such a feature.
- itissid 2y agoHave you tried replacing the input string with a random but fixed mapping and obfuscate that its chess(like replace the word 'chess' with say, 'an alien ritual practice') and see how it does?
- borgdefense 2y agoI wasn't impressed in the first 5 minutes of using it but it is quite impressive after 2 solid hours of random topics. Much faster for sure but I have also not had anything give an error in python with jupyter. Usually you could only stray so far with more obscure python libraries before it starts producing errors. That much better than 4 in chess is pretty shocking in a great way.
- chermanowicz 2y agoOn one hand, some of these results are impressive; on the other, the illegal moves count is alarming - it suggests no reasoning ability as there should never be an illegal move? I mean, how could a violation of a fairly basic game (from a rules perspective) be acceptable in assigning any 'outcome' to a model other than failure?
- mistrial9 2y agomy guess is -- the probabilistic engine does sequence variation and it just will not do anything else.. so a simple A->B sort of logic is elusive at a deep level; secondly the adaptive and very broad kinds of questions and behaviors it handles, also make it difficult to write logic that could correct defective answers to simple logic.
- vbezhenar 2y agoThat's because LLM does not reason. For me, as a layman, that seems strange that they don't wire some kind of Prolog engine to fill the gap, (like they wired Python to fill the gap in arithmetic) but probably it's not that easy.
- jodrellblank 2y agoProlog doesn’t reason either, it does a simple brute force search over all possible states of your code and if that’s not fast enough it can table (cache, memoize) previous states. People build reasoning engines from it, in the same way they do with Python and LISPs.
- eggdaft 2y agoWhat do you mean by “an LLM doesn’t reason”?
- vbezhenar 2y agoI mean that it does not follow basic logic rules when constructing its thoughts. For many tasks they'll get it right, however it's not that hard to find a task for which LLM will yield obviously logically wrong answer. That would be impossible for human with basic reasoning.
- DatoClement 2y agoI think specifying chess rules at the beginning of the prompt might help mitigate the problem of illegal moves