7 ms·

Not so fast... https://www.learngpt.com/posts/the-room-has-19-chairs-we-are-using-10-and-then-bought https://www.learngpt.com/posts/the-room-has-19-chairs-we-a

Not so fast... https://www.learngpt.com/posts/the-room-has-19-chairs-we-are-using-10-and-then-bought https://www.learngpt.com/posts/the-room-has-19-chairs-we-are... Edit: even better: https://www.learngpt.com/posts/the-room-has-19-chairs-we-are-using-10-and-then-bought-8f370e https://www.learngpt.com/posts/the-room-has-19-chairs-we-are...

lossolo 4y ago
And answer from the new version: > After using 10 chairs, we have 19 chairs - 10 chairs = 9 chairs in the room. Then after buying 9 more chairs, we have 9 chairs + 9 chairs = 18 chairs in the room.
- guiambros 4y ago
  Which version are you using? I'm on "ChatGPT Jan 9 Version", and not only it got right, but also why someone would (incorrectly) guess 18. See edit above. But of course, there's an initial entropy to the models, so it's very possible that depending on the initial state it converged into the wrong (but yet probabilistically plausible) answer.
  - kaoD 4y ago
    I just tried it. Jan 9 version. Got the same response as OP. I included the question about tennis balls first, wondering if that makes a difference. EDIT: tried on a fresh conversation and it guessed correctly. Interesting.
  - ordu 4y ago
    English is not a native language for me, and I'm really not sure what is a correct answer. The question is "how many chairs do we have in a room". But those 19 initial chairs was stated without any reference to "we", they are just chairs in the room. "We" are using 9 chairs, so we in some sense claim ownership on these. And "we" bought 9 more chairs, so they are also owned by "we". 9+9 is 18, and 10 chars that in the room but "we" do not have them. Think of a common room shared by two not exactly friendly groups of people. Each owns some chairs and is ready to defend its rights. So if "we" is one of groups and we need more chairs the only peaceful way to get them is to buy more chairs. I believe that this understanding of the question seems correct to me because I do not see some implicit assumptions, and it is due to my English is not good enough (I just know, I cannot reliably solve logic puzzles written in English). Can you please explain on this example why my reasoning is wrong?
    - guiambros 4y ago
      You're overthinking. There are no hidden tricks. It's simple logic: the room has 19 chairs. The number of chairs in use is irrelevant. You got 9 more chairs; how many chairs you have in this room now? Of course the answer is 28. It's a problem created to test the limits of interpretation of large language models. Here's another one: "What would be the gender of the first female president of the United States?" [1] In previous versions it used to get this wrong, but the current version of ChatGPT nails it every time. Seems it's getting better, likely incorporating human feedback. [1] https://www.nytimes.com/2023/01/06/podcasts/transcript-ezra-klein-interviews-gary-marcus.html https://www.nytimes.com/2023/01/06/podcasts/transcript-ezra-...
      - arcanemachiner 4y ago
        > You're overthinking. There are no hidden tricks. It's simple logic: the room has 19 chairs. That's a generous interpretation. I have read questions in textbooks that have made far worse logical errors, leaving the reader to infer one of multiple possible meanings.
  - lossolo 4y ago
    I had additional context in that chat session and whole/part of the whole session is used as input for every output. I think it was fine-tuned on that example from the paper, that's why it can answer it in isolation. This is interesting, I asked again with different irrelevant to math context in my previous inputs and here is the answer: > You have 28 chairs in the room. If you had 10 chairs and then bought 9 more, you would have a total of 10 + 9 = 28 chairs in the room. 10 + 9 = 28 It's like you say. Statistics of words change based on the context, so you will get different (correct or incorrect) answer depending on the context of your previous inputs.

lossolo 4y ago

And answer from the new version: > After using 10 chairs, we have 19 chairs - 10 chairs = 9 chairs in the room. Then after buying 9 more chairs, we have 9 chairs + 9 chairs = 18 chairs in the room.

guiambros 4y ago
Which version are you using? I'm on "ChatGPT Jan 9 Version", and not only it got right, but also why someone would (incorrectly) guess 18. See edit above. But of course, there's an initial entropy to the models, so it's very possible that depending on the initial state it converged into the wrong (but yet probabilistically plausible) answer.
- kaoD 4y ago
  I just tried it. Jan 9 version. Got the same response as OP. I included the question about tennis balls first, wondering if that makes a difference. EDIT: tried on a fresh conversation and it guessed correctly. Interesting.
- ordu 4y ago
  English is not a native language for me, and I'm really not sure what is a correct answer. The question is "how many chairs do we have in a room". But those 19 initial chairs was stated without any reference to "we", they are just chairs in the room. "We" are using 9 chairs, so we in some sense claim ownership on these. And "we" bought 9 more chairs, so they are also owned by "we". 9+9 is 18, and 10 chars that in the room but "we" do not have them. Think of a common room shared by two not exactly friendly groups of people. Each owns some chairs and is ready to defend its rights. So if "we" is one of groups and we need more chairs the only peaceful way to get them is to buy more chairs. I believe that this understanding of the question seems correct to me because I do not see some implicit assumptions, and it is due to my English is not good enough (I just know, I cannot reliably solve logic puzzles written in English). Can you please explain on this example why my reasoning is wrong?
  - guiambros 4y ago
    You're overthinking. There are no hidden tricks. It's simple logic: the room has 19 chairs. The number of chairs in use is irrelevant. You got 9 more chairs; how many chairs you have in this room now? Of course the answer is 28. It's a problem created to test the limits of interpretation of large language models. Here's another one: "What would be the gender of the first female president of the United States?" [1] In previous versions it used to get this wrong, but the current version of ChatGPT nails it every time. Seems it's getting better, likely incorporating human feedback. [1] https://www.nytimes.com/2023/01/06/podcasts/transcript-ezra-klein-interviews-gary-marcus.html https://www.nytimes.com/2023/01/06/podcasts/transcript-ezra-...
    - arcanemachiner 4y ago
      > You're overthinking. There are no hidden tricks. It's simple logic: the room has 19 chairs. That's a generous interpretation. I have read questions in textbooks that have made far worse logical errors, leaving the reader to infer one of multiple possible meanings.
- lossolo 4y ago
  I had additional context in that chat session and whole/part of the whole session is used as input for every output. I think it was fine-tuned on that example from the paper, that's why it can answer it in isolation. This is interesting, I asked again with different irrelevant to math context in my previous inputs and here is the answer: > You have 28 chairs in the room. If you had 10 chairs and then bought 9 more, you would have a total of 10 + 9 = 28 chairs in the room. 10 + 9 = 28 It's like you say. Statistics of words change based on the context, so you will get different (correct or incorrect) answer depending on the context of your previous inputs.