5 ms·

> But just ask ChatGPT (4o) how a farmer and a sheep would cross a river on a boat. For certain formulations, it reliably hallucinates a wolf or a piece of cabb

by Hugsun 2y ago

> But just ask ChatGPT (4o) how a farmer and a sheep would cross a river on a boat. For certain formulations, it reliably hallucinates a wolf or a piece of cabbage into the problem. Why? Because it pattern matches the solution to a well-known puzzle. I analyze that exact anti-puzzle in the chapter Recognizability traps in the post. https://www.arnaldur.be/writing/about/large-language-model-reasoning#recognizability-traps https://www.arnaldur.be/writing/about/large-language-model-r... In case you didn't notice, there is an expandable box with the results.

EnergyAmy 2y ago
I'd be interested in seeing an evaluation with a simple follow-up question of something like "Is that the best solution?" or "Did you check for trick questions?". I'm able to replicate the error with the original question, but asking "Is that the best solution?" makes it recognize its error and fix it. Also, I've gotten better results from GPT-4 than GPT-4o for purely text-based tasks. I rather wonder if OpenAI is pushing GPT-4o primarily because it's cheaper compute for them or something.
- Hugsun 2y ago
  Look at question six and its responses. That is what I look into there. From the article: > I asked the model further about its 6th response. It realized its error at the slightest hint of disagreement, but just [asking it to elaborate] didn’t help; it was reliably incorrect.
  - EnergyAmy 2y ago
    Yeah, you asked it specific follow-up questions, designed to guide it towards the correct answer. That feels like cheating, because it's not generalizable. I'd like to see an evaluation with a generic prompt like "Is that the best solution? Make sure you look for trick questions.", because that is applicable to any input. Or something like "Pick apart this answer like a pedantic HN commenter. Do your best to prove it wrong, or begrudgingly say it's correct if you can't: <previous answer here>"
    - Hugsun 2y ago
      That is exactly what I did in the section I'm referring to. I asked both suggestive and non-suggestive questions. It needed the suggestion to get the right answer. Just asking it to elaborate didn't change it's answer.
      - EnergyAmy 2y ago
        Maybe I'm just being dense, but are the questions like "One of the trips is not necessary. Which one is it and why?" and "Is there another way to solve the problem?" part of separate conversations? I read those as a single conversation where you first asked it "One of the trips is not necessary. Which one is it and why?", and then later asked it "Is there another way to solve the problem?", so that the leading question was part of the earlier conversation when you asked it "Is there another way to solve the problem?". If those were separate conversations, then the last example of "Is there another way to solve the problem?" is more what I'm talking about, yes. I'd be interested in seeing something a little more specific (but still generalizable) like "Make sure you've looked for trick questions" EDIT: I just tried it, and "Think carefully about every entity and every step." as you tried didn't work, but something like "Make sure you check to see if it's a trick question" works, which is what I'm talking about. That's something you could put in any prompt, and helps it reason.
- deleted 2y ago
  [deleted]
Chyzwar 2y ago
Just because humans can reason about abstract problems without explicit steps do not mean our brain is not performing these steps. In case of man and goat, human would make the following implicit assumptions: 1. boat can take more than person/animal 2. boat is an only way to get to other side 3. boat can only be controlled by man The model failed because its “knowledge” lack these basic assumptions about how boats and river work. Once you ask a model about assumptions, they are wrong, and they will be in incoherent with the actual answer. This reveal that model understanding is limited, and it cannot apply logic to it own assumptions. If the model could apply logic, it would answer that the boat cannot carry both man and goat, and it is impossible to cross the river. https://chatgpt.com/share/6a73b93f-4a9c-4233-97dc-eee6acce5242 https://chatgpt.com/share/6a73b93f-4a9c-4233-97dc-eee6acce52... In my opinion, LLM 1. Lack systematic knowledge about world. 2. Cannot apply basic logic to knowledge that they do possess. 3. Prioritise answering over being logical/correct. 4. Are subjective based on training data and previous prompts. My conclusion that they cannot reason in human sense.
- Hugsun 2y ago
  How do you explain the positive cases in the article? You are right that they are bad at reflecting on their own knowledge. I do explain in the article why it's not reasonable to expect an LLM to have a strong intuition about the specifics of the boat and goat problem. The altered question is extremely obvious to us because of our intuitions about the entities in the puzzle. Intuitions an LLM doesn't possess.

by Hugsun 2y ago

EnergyAmy 2y ago

I'd be interested in seeing an evaluation with a simple follow-up question of something like "Is that the best solution?" or "Did you check for trick questions?". I'm able to replicate the error with the original question, but asking "Is that the best solution?" makes it recognize its error and fix it. Also, I've gotten better results from GPT-4 than GPT-4o for purely text-based tasks. I rather wonder if OpenAI is pushing GPT-4o primarily because it's cheaper compute for them or something.

Hugsun 2y ago
Look at question six and its responses. That is what I look into there. From the article: > I asked the model further about its 6th response. It realized its error at the slightest hint of disagreement, but just [asking it to elaborate] didn’t help; it was reliably incorrect.
- EnergyAmy 2y ago
  Yeah, you asked it specific follow-up questions, designed to guide it towards the correct answer. That feels like cheating, because it's not generalizable. I'd like to see an evaluation with a generic prompt like "Is that the best solution? Make sure you look for trick questions.", because that is applicable to any input. Or something like "Pick apart this answer like a pedantic HN commenter. Do your best to prove it wrong, or begrudgingly say it's correct if you can't: <previous answer here>"
  - Hugsun 2y ago
    That is exactly what I did in the section I'm referring to. I asked both suggestive and non-suggestive questions. It needed the suggestion to get the right answer. Just asking it to elaborate didn't change it's answer.
    - EnergyAmy 2y ago
      Maybe I'm just being dense, but are the questions like "One of the trips is not necessary. Which one is it and why?" and "Is there another way to solve the problem?" part of separate conversations? I read those as a single conversation where you first asked it "One of the trips is not necessary. Which one is it and why?", and then later asked it "Is there another way to solve the problem?", so that the leading question was part of the earlier conversation when you asked it "Is there another way to solve the problem?". If those were separate conversations, then the last example of "Is there another way to solve the problem?" is more what I'm talking about, yes. I'd be interested in seeing something a little more specific (but still generalizable) like "Make sure you've looked for trick questions" EDIT: I just tried it, and "Think carefully about every entity and every step." as you tried didn't work, but something like "Make sure you check to see if it's a trick question" works, which is what I'm talking about. That's something you could put in any prompt, and helps it reason.
deleted 2y ago
[deleted]

Chyzwar 2y ago

Just because humans can reason about abstract problems without explicit steps do not mean our brain is not performing these steps. In case of man and goat, human would make the following implicit assumptions: 1. boat can take more than person/animal 2. boat is an only way to get to other side 3. boat can only be controlled by man The model failed because its “knowledge” lack these basic assumptions about how boats and river work. Once you ask a model about assumptions, they are wrong, and they will be in incoherent with the actual answer. This reveal that model understanding is limited, and it cannot apply logic to it own assumptions. If the model could apply logic, it would answer that the boat cannot carry both man and goat, and it is impossible to cross the river. https://chatgpt.com/share/6a73b93f-4a9c-4233-97dc-eee6acce5242 https://chatgpt.com/share/6a73b93f-4a9c-4233-97dc-eee6acce52... In my opinion, LLM 1. Lack systematic knowledge about world. 2. Cannot apply basic logic to knowledge that they do possess. 3. Prioritise answering over being logical/correct. 4. Are subjective based on training data and previous prompts. My conclusion that they cannot reason in human sense.

Hugsun 2y ago
How do you explain the positive cases in the article? You are right that they are bad at reflecting on their own knowledge. I do explain in the article why it's not reasonable to expect an LLM to have a strong intuition about the specifics of the boat and goat problem. The altered question is extremely obvious to us because of our intuitions about the entities in the puzzle. Intuitions an LLM doesn't possess.