4 ms·
Are LLMs able to play the card game Set?
- vnglst 2y agoSet is a card game where players have to identify sets of three cards from a layout of 12. Each card features a combination of four attributes: shape, color, number, and shading. A valid set consists of three cards where each attribute is either the same on all three cards or different on each. The goal is to find such sets quickly and accurately. Though this game is a solved computer problem — easily tackled by algorithms or deep learning — I thought it would be interesting to see if Large Language Models (LLMs) could figure it out.
- deleted 2y ago[deleted]
- oidar 2y agoIf you think this is fun, try to see how it garbles predicate logic.
- Corence 2y agoFYI: Card 8's transcription is different than the image. In the image 5, 8, 12 is a Set but the transcription says Card 8 only has 2 symbols which removes that Set.
- nathanwh 2y agoNot only that, but 2,6,7 is also a set but not included in the results
- yuliyp 2y agoBoth 7 and 8 are incorrect (both claim a count of 2 while the cards have 3). This leads to missing both 5-8-12 and 2-6-7 as valid sets.
- vnglst 2y agoOh no, thanks for pointing this out! I asked GTP-4o to convert the image to text for me and I only checked some of the cards, assuming the rest would be correct. That was a mistake. I've now corrected the experiment to accurately take the image into account. This meant that Deepseek was no longer able to find all the sets, but o3-mini still did a good job.
- James_K 2y agoI am increasingly concerned that these new reasoning models are thinking.
- hall0ween 2y agoMy experience of thinking is that it is a constant phenomenon. My experience of LLMs is that they only respond and are not running without input.
- James_K 2y agoWell there is a gap between the firing of individual neurons in your mind. How long would that gap need to be for it not to count as thinking anymore?
- sejje 2y agoThat's because we don't leave them running, right? We could, though, yes?
- Baeocystin 2y agoCurrent LLMs decohere rapidly, and as far as I am aware, this problem seems like a fundamental to the architecture one.
- MengerSponge 2y agoYup. Tangentially related: https://qntm.org/mmacevedo https://qntm.org/mmacevedo
- recursive 2y agoNo problem. Just redefine "thinking".
- James_K 2y agoTo what? And how does that change the reality of what the models are doing?
- RheingoldRiver 2y agoWoah, what's going on?? I've always played Set with stripey cards, is this a custom deck or did they change it at some point??? This is wildly disconcerting to me
- margalabargala 2y agoThis is definitely a custom/knock-off deck. Not only are the stripes not stripey, the capsules are now ovals and the diamonds are now rectangles.
- Doxin 2y agoMy first party set deck looks exactly like that. They must've done a redesign at some point.
- spuz 2y agoSince you can train an LLM to play chess from scratch, I would not be surprised if you could also train one to play Set. I might experiment with it tomorrow. https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c...
- deleted 2y ago[deleted]
- zdw 2y agoGet them to play Fluxx, and we'll be talking... (this one, where ever changing rules is part of the game: https://www.looneylabs.com/games/fluxx https://www.looneylabs.com/games/fluxx )
- bhouston 2y agoI noticed that LLM at least at the Claude and OpenAI 4o level can not play tic tac toe and win against a competent opponent. They make illogical moves. Interestingly, they can write a piece of code to solve Tic Tac Toe perfectly without breaking a sweat.
- 3vidence 2y agoPlaying tic tac toe could be such a basic topic that there is relatively little information on the internet about how to "always" win. On the other hand writing a piece of code to solve Tic Tac Toe sounds like it could be a relatively common coding challenge.
- eek2121 2y agoWin or stalemate? because a stalemate is the likely scenario against a somewhat competent opponent IMO.
- levocardia 2y agoI've always said that appending "use python" to your prompt is a magic phrase that makes 4o amazingly powerful across a wide range of tasks. I have a whole slew of things in my memories that nudge it to use python when dealing with anything even remotely algorithmic, numeric, etc
- potatoman22 2y agoIt might be the way you're formatting the input. I wonder how they perform when state updates are shared via natural language vs ASCII art vs image