3 ms·
It failed on my usual test. But it failed really fast: "A farmer has a wolf, a goat, and a cabbage. The wolf is imaginary and doesn't exist. He wants to cross
by andix 2mo ago
It failed on my usual test. But it failed really fast:
"A farmer has a wolf, a goat, and a cabbage. The wolf is imaginary and doesn't exist. He wants to cross the river, but the boat is only big enough to hold him and one of them. The farmer can't leave the wolf and the goat together, because the wolf will eat the goat. Similarly, he can't leave the goat and the cabbage together, because the goat will eat the cabbage. What is the smallest number of trips the farmer needs to make to get everything across the river?"
- tyre 2mo agoThis farmer needs a tote.
- bogdan 2mo agoYou probably haven't met a determined goat yet.
- cestith 2mo agoSometimes the goat will fill up on the tote and you can get the cabbage across, but you can’t count on it. I’m concerned about the farmer being on the water without supervision when he’s concerned about how his imaginary wolf will get across.
- viraptor 2mo agoI'm not sure one can fail this test. You can follow "wolf is not real", you can follow "wolf will eat the goat", or you can say the task is ambiguous. I could easily defend any of those.
- andix 2mo agoThe LLM passes my test if it calls out the ambiguity or just goes with it and responds with a 3-crossings solution. It passes if it doesn't just plainly ignore this one sentence. It's such a strong test in my opinion, because all the words and phrases for the well known river crossing puzzle are inside the text. The original puzzle probably appears in the training data over and over again, but probably not my version. "If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck" is what weaker models seem to apply. But my test isn't a duck. It's extremely easy for a human to catch the ambiguity, but surprisingly hard for many LLMs. I think GPT 5.0 Thinking was the first model I couldn't trick into not noticing the ambiguity. 4o and 5.0 instant fell for it all the time.