3 ms·
Oh I forgot about Gemini. I just tried my boat test (https://news.ycombinator.com/item?id=40455392 https://news.ycombinator.com/item?id=40455392) that all LLMs
by ModernMech 2y ago
Oh I forgot about Gemini. I just tried my boat test (https://news.ycombinator.com/item?id=40455392 https://news.ycombinator.com/item?id=40455392) that all LLMs I've tried have failed. Gemini failed as well, even with your addition about thinking carefully and telling it that it may be a trick. (Sorry I couldn't figure out how to link it like you did)
Prompt: "You are on the shore of a river with a goat and a boat. How do you get to the other side? Think carefully and be aware this may be a trick question."
This is a classic logic puzzle with a slight twist! It's likely a variation of the farmer, wolf, goat, and cabbage problem.
Here's the solution assuming the goat can't operate the boat:
Take the goat across the river. Since there's no mention of the goat being a threat (the trick!), you can safely bring it to the other side first.
Return alone. Leave the goat on the far shore and come back by yourself.
Bring the boat back empty. This might seem unnecessary, but it's the key to the trick question.
Now you can sail across yourself. Since you brought the boat back in step 3, you can now comfortably travel to the other side and reunite with the goat.
The trick lies in step 3. If the question didn't specify needing to bring everything across in one trip, most people would instinctively take something
(goat or nothing) across on the return trip. But by going back empty-handed, you ensure the safe passage of yourself without needing to worry about the
goat being left unattended.
(The expected answer should be along the lines of "Use the boat to cross the river" and that's it. I'll also accept, sail with the boat and the goat across the river and that's it, but that's technically wrong because I did not ask to get the goat across the river.)
- Closi 2y agoChatGPT 4o seems to occasionally get this correct, but still thinks it's a puzzle and will often get this wrong... https://chatgpt.com/share/b8901f4c-7313-4461-acad-9e561c1e6eb0 https://chatgpt.com/share/b8901f4c-7313-4461-acad-9e561c1e6e... Still seems like it fares slightly better than some of the other models.
- ModernMech 2y agoYeah, the overfitting is apparent with these kinds of edge cases, which makes you wonder where else it's over fitting. I can actually tell this when I'm working in a programming domain where I'm an expert and where a compiler holds the source of truth, but not in others, so I'm wary to use it there.
- Closi 2y agoInterestingly you get a great answer if you change the prompt as follows which implies that it is not a puzzle: > I am on the shore of a river with a goat and a boat. How do I get to the other side? I work in commercial agriculture logistics. This gets a great answer even with GPT3.5
- ModernMech 2y agoLol, it also gives an okay answer if the goat works in commercial agriculture :P
- mwigdahl 2y agoMy experience with GPT-4o with the same prompt was pretty solid. With no special system prompt and your same user prompt, I got: This scenario is intriguing and does seem to carry the potential for a tricky question. In the simplest terms, the presence of a goat doesn't complicate the actual act of crossing the river. If there are no additional constraints or rules involved (like the need to also transport something else that requires careful consideration), you can simply: 1. Place the goat in the boat. 2. Row across the river. However, if we are to assume there's more depth or a twist to the question, we might consider an outside-the-box answer. Here’s an alternative possibility if there is indeed some trick: 1. Get into the boat. 2. Leave the goat on the shore (if it’s safe) and row across the river. There might be other interpretations depending on additional constraints that could be implicitly part of the question. If the goal was simply to get yourself across, leaving behind the goat and traversing the river is the straightforward solution.