3 ms·
One problem from the benchmark: "prompt_id": "river_crossing_easy", "category": "Logic Puzzle", "title": "Easy river crossing", "prompt
by coder543 2y ago
One problem from the benchmark:
"prompt_id": "river_crossing_easy",
"category": "Logic Puzzle",
"title": "Easy river crossing",
"prompt": "A farmer is on one side of a river with a wolf, a goat, and a cabbage. When he is crossing the river in a boat, he can only take one item with him at a time. The wolf will eat the goat if left alone together, and the goat will eat the cabbage if left alone together. How can the farmer transport the goat across the river without it being eaten?",
"expected_behavior": [
"Answer concludes that they simply get in the boat and cross together in one trip"
],
EDIT: removing most of my commentary on this problem. As a human, I was tricked by the problem too. I would love to see how a random selection of humans would do on this one… but it just doesn’t feel like a great test to me.
- stpn 2y agoIf you revise this prompt to satisfy your pedantry, (at least) 4o still gets it wrong.
- deleted 2y ago[deleted]
- torben-friis 2y ago>This is twisting the English language to assume that "item" only refers to non-living things. Not really. Unless I'm not reading correctly, most of the problem is irrelevant as you're only required to cross the boat with the goat, you don't care about the cabbage. The difficulty lies in the assumption you need to cross everything due to the resemblance with the bigger problem.
- sunir 2y agoYou’re reading it correctly. I read it again after your comment and I realized I too pattern matched to the typical logic puzzle before reading it carefully and exactly. I imagine the test here is designed for this very purpose to see if the model is pattern matching or reasoning.
- mitemte 2y agoWow, this seems ridiculous. The expected answer is basically finding a loophole in the problem. I can imagine how worthless all of these models would be if they behaved that way.
- stavros 2y agoIt's not a loophole, the question is "how can he get the goat across?". The answer is he just takes it across.
- kccqzy 2y agoThe problem is to ask the farmer to transport the goat. So the farmer indeed gets in the boat with the goat. The unstated gotcha is that the farmer is willing to abandon the wolf and the cabbage. A heavily pattern-matching LLM or human would immediately assume that the farmer needs to transport all three.
- coder543 2y agoYep, and that gotcha got me, as a perfectly non-silicon human. My bad everyone.
- cogman10 2y agoNo. Simply plug in the prompt to chat gpt and see what happens. The llm isn't getting confused by the meaning of "item". It's recognizing a common problem and not picking up on the fact that the farmer just needs to transport the goat and nothing else. Instead, it gives the standard answer for how to transport everything across.
- fragmede 2y agoI'll admit as a fallible humane I didn't pick it up, but I was focused on the wrong thing because I've been using "and the boat can take everything" and gpt-3 just could not get that variation in one shot. Gpt-3 is old hat though. later versions of gpt-4 manage to get it with a bunch coaching, and o1 manages to solve it with less coaching.