6 ms·

I also tried this out and the balloon example as well. User: Barbara, Mary, and Harry had red balloons. Ted had a blue balloon. Sam also had a red balloon. How

I also tried this out and the balloon example as well. User: Barbara, Mary, and Harry had red balloons. Ted had a blue balloon. Sam also had a red balloon. How many children have red balloons? ChatGPT (GPT-4): Barbara, Mary, Harry, and Sam each had a red balloon. That means there are 4 children with red balloons. User: What about Ted? ChatGPT (GPT-4): Ted had a blue balloon, not a red one. So, he is not counted among the children with red balloons. Therefore, the total number of children with red balloons remains 4. User: Why do you think I asked my original question? ChatGPT (GPT-4): It seems that you asked your original question to test my ability to understand and process information provided in a text. You wanted to see if I could correctly identify and count the number of children with red balloons based on the details given. User: Can you explain what mistakes a simpler model may have made on this question? ChatGPT (GPT-4): A simpler language model may have made the following mistakes when answering your original question: ... I like trying these sort of puzzles and "gotcha" prompts and I've found GPT-3.5 and lower versions are indeed not hard to confuse, but it's much harder to find simple examples that confuse GPT-4. Often times GPT-4 will even be able to explain the puzzle and why it's interesting. I think this is part of why GPT-4 feels qualitatively different. I understand (roughly) how transformers work, but GPT-4 does seem to have some emergent ability to reason, at least in the sense of making short chains of logical inferences, and this does seem to inform text completions not just on these sort of puzzles. That said I do think there's real danger of overconfidence and over reliance.

imjonse 4y ago
I thought the gotcha was that nowhere was it stated which of them are actually children.
- kzrdude 4y ago
  How it just takes these implicit assumptions in stride makes me worry that in the future these chatbots will be fine with assumptions and me - human - has a too square brain to go on without definitions. :)
usaar333 4y ago
Agreed GPT4 is better, but there's still simple world modeling issues: > I have a chair in front of the TV and one in front of the computer. I put a watch under the chair in front of the TV. I swap the chairs. Where is the watch? > After swapping the chairs, the watch is now under the chair that is in front of the computer. Interestingly, GPT3 got it correct.
- kenjackson 4y ago
  I just tried it and it says it’s under the chair in front of the computer. I used the Bing model.
- letmevoteplease 4y ago
  I asked GPT 4 (Bing) your question, but added "(Please give a plausible explanation for why this question could be answered either way.)" > This question could be answered either way depending on how you interpret the phrase “under the chair”. If you mean under the chair as in attached to the bottom of the chair, then the watch would move with the chair when you swap them. In that case, the watch would be in front of the computer. However, if you mean under the chair as in on the floor below the chair, then the watch would stay on the same spot when you swap the chairs. In that case, the watch would be in front of the TV. So, it depends on what you mean by “under”.
  - deleted 4y ago
    [deleted]
  - usaar333 4y ago
    Yes, but that's a huge hint. GPT4 naively doesn't understand this. Question: > I have a chair in front of the TV and one in front of the computer. I put a watch under the chair in front of the TV. I swap the chairs. Where is the watch? > Work out step by step. Is this question ambiguous? Answer: This question is not ambiguous, but let's break it down step by step to understand the situation and find the location of the watch. You have a chair in front of the TV (Chair A) and one in front of the computer (Chair B). You put a watch under Chair A (the one in front of the TV). You swap the chairs, meaning Chair A is now in front of the computer and Chair B is in front of the TV. After swapping the chairs, the watch remains under Chair A. Since Chair A is now in front of the computer, the watch is under the chair in front of the computer.

imjonse 4y ago

I thought the gotcha was that nowhere was it stated which of them are actually children.

kzrdude 4y ago
How it just takes these implicit assumptions in stride makes me worry that in the future these chatbots will be fine with assumptions and me - human - has a too square brain to go on without definitions. :)

usaar333 4y ago

Agreed GPT4 is better, but there's still simple world modeling issues: > I have a chair in front of the TV and one in front of the computer. I put a watch under the chair in front of the TV. I swap the chairs. Where is the watch? > After swapping the chairs, the watch is now under the chair that is in front of the computer. Interestingly, GPT3 got it correct.

kenjackson 4y ago
I just tried it and it says it’s under the chair in front of the computer. I used the Bing model.
letmevoteplease 4y ago
I asked GPT 4 (Bing) your question, but added "(Please give a plausible explanation for why this question could be answered either way.)" > This question could be answered either way depending on how you interpret the phrase “under the chair”. If you mean under the chair as in attached to the bottom of the chair, then the watch would move with the chair when you swap them. In that case, the watch would be in front of the computer. However, if you mean under the chair as in on the floor below the chair, then the watch would stay on the same spot when you swap the chairs. In that case, the watch would be in front of the TV. So, it depends on what you mean by “under”.
- deleted 4y ago
  [deleted]
- usaar333 4y ago
  Yes, but that's a huge hint. GPT4 naively doesn't understand this. Question: > I have a chair in front of the TV and one in front of the computer. I put a watch under the chair in front of the TV. I swap the chairs. Where is the watch? > Work out step by step. Is this question ambiguous? Answer: This question is not ambiguous, but let's break it down step by step to understand the situation and find the location of the watch. You have a chair in front of the TV (Chair A) and one in front of the computer (Chair B). You put a watch under Chair A (the one in front of the TV). You swap the chairs, meaning Chair A is now in front of the computer and Chair B is in front of the TV. After swapping the chairs, the watch remains under Chair A. Since Chair A is now in front of the computer, the watch is under the chair in front of the computer.