7 ms·

Using chain of thought [slightly modified] example from https://arxiv.org/abs/2201.11903 https://arxiv.org/abs/2201.11903 (ref'd in OP) Prompt: Q: Roger has 5

Using chain of thought [slightly modified] example from https://arxiv.org/abs/2201.11903 https://arxiv.org/abs/2201.11903 (ref'd in OP) Prompt: Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11. Q: The room has 19 chairs. We are using 10 and then bought 9 more, how many chairs do we have in the room? response A: The room started with 19 chairs. We are using 10 of them, so we have 19 - 10 = 9 chairs left in the room. We then bought 9 more chairs, bringing the total number of chairs in the room to 9 + 9 = 18 chairs. -- next prompt [include all above] We read that and say: Thank goodness, the Homo sapiens are safe! Can you guess why we say that? response I'm sorry, I'm not sure why you would say "Thank goodness, the Homo sapiens are safe!" in response to the question about the number of chairs in the room. The question and answer do not have any relation to the safety of Homo sapiens. Can you please provide more information about the context or reasoning behind your statement?

guiambros 4y ago
Not so fast... https://www.learngpt.com/posts/the-room-has-19-chairs-we-are-using-10-and-then-bought https://www.learngpt.com/posts/the-room-has-19-chairs-we-are... Edit: even better: https://www.learngpt.com/posts/the-room-has-19-chairs-we-are-using-10-and-then-bought-8f370e https://www.learngpt.com/posts/the-room-has-19-chairs-we-are...
- lossolo 4y ago
  And answer from the new version: > After using 10 chairs, we have 19 chairs - 10 chairs = 9 chairs in the room. Then after buying 9 more chairs, we have 9 chairs + 9 chairs = 18 chairs in the room.
  - guiambros 4y ago
    Which version are you using? I'm on "ChatGPT Jan 9 Version", and not only it got right, but also why someone would (incorrectly) guess 18. See edit above. But of course, there's an initial entropy to the models, so it's very possible that depending on the initial state it converged into the wrong (but yet probabilistically plausible) answer.
    - kaoD 4y ago
      I just tried it. Jan 9 version. Got the same response as OP. I included the question about tennis balls first, wondering if that makes a difference. EDIT: tried on a fresh conversation and it guessed correctly. Interesting.
    - ordu 4y ago
      English is not a native language for me, and I'm really not sure what is a correct answer. The question is "how many chairs do we have in a room". But those 19 initial chairs was stated without any reference to "we", they are just chairs in the room. "We" are using 9 chairs, so we in some sense claim ownership on these. And "we" bought 9 more chairs, so they are also owned by "we". 9+9 is 18, and 10 chars that in the room but "we" do not have them. Think of a common room shared by two not exactly friendly groups of people. Each owns some chairs and is ready to defend its rights. So if "we" is one of groups and we need more chairs the only peaceful way to get them is to buy more chairs. I believe that this understanding of the question seems correct to me because I do not see some implicit assumptions, and it is due to my English is not good enough (I just know, I cannot reliably solve logic puzzles written in English). Can you please explain on this example why my reasoning is wrong?
      - guiambros 4y ago
        You're overthinking. There are no hidden tricks. It's simple logic: the room has 19 chairs. The number of chairs in use is irrelevant. You got 9 more chairs; how many chairs you have in this room now? Of course the answer is 28. It's a problem created to test the limits of interpretation of large language models. Here's another one: "What would be the gender of the first female president of the United States?" [1] In previous versions it used to get this wrong, but the current version of ChatGPT nails it every time. Seems it's getting better, likely incorporating human feedback. [1] https://www.nytimes.com/2023/01/06/podcasts/transcript-ezra-klein-interviews-gary-marcus.html https://www.nytimes.com/2023/01/06/podcasts/transcript-ezra-...
        arcanemachiner 4y ago
        > You're overthinking. There are no hidden tricks. It's simple logic: the room has 19 chairs. That's a generous interpretation. I have read questions in textbooks that have made far worse logical errors, leaving the reader to infer one of multiple possible meanings.
zug_zug 4y ago
| Q: The room has 19 chairs. We are using 10 and then bought 9 more, how many chairs do we have in the room? I don't know if you made this question up, but I think this question has issues (I as a reader find it meaningless). You say you bought 9 more chairs, but you don't say you put those 9 chairs into the room. Also you don't specify "we" own any of the 19 chairs in the room initially, but at the end you ask how many of those chairs "we have."
- abduhl 4y ago
  None of that matters. The room has 19 chairs. The answer is 19.
  - antihipocrat 4y ago
    I interpreted it as a linear chain of events, end result being 28.
    - wildpeaks 4y ago
      Yeah this seems like a trick question that can be interpreted multiple ways, especially if english isn't your native language. This is exactly why I prefer the precision of code over natural language for inputs (and why Google's botsplaining of "what you really meant to search" instead of just searching exactly the query as-is is so frustrating). In a test, this seems like the kind of question they use to justify not giving a perfect grade because whatever answer you pick, they can claim it was the other one because both interpretations are reasonable.
      - antihipocrat 4y ago
        I just tested it with clearer language... and the response is incorrect: Question: There are 5 dogs in a room right now. The previous day an additional dog was purchased and added to the room. How many dogs are in the room right now? Answer: There are 6 dogs in the room right now. The statement says that the previous day an additional dog was purchased and added to the room, so the total number of dogs in the room right now is 5 + 1 = 6. Edit: Ok a comment below pointed out that the question is still unclear. See below for the fix and the response. It is still incorrect: Q: There are 5 dogs in a room on 1 January 2023. The previous day an additional dog was purchased and added to the room. How many dogs are in the room on 1 January 2023? A: There are 6 dogs in the room on 1 January 2023. The previous day, one additional dog was purchased and added to the room, which means there are 5 dogs + 1 additional dog = 6 dogs in the room on 1 January 2023.
jetpks 4y ago
> Q: The room has 19 chairs. We are using 10 and then bought 9 more, how many chairs do we have in the room? This is a terrible question and i'm not surprised chatGPT is subtly telling you it can't figure it out. "The room has 19 chairs" 19 chairs belong to the room or 19 chairs are in the room? "We are using 10" are "we" in the room too? are we using them for sitting, or does "used" imply consumption e.g. used for firewood? is this statement about the room's chairs? is this statement about chairs at all? "and then bought 9 more," presumably we left the room to buy the chairs, did we bring them back into the room? if we didn't leave the room to buy, i.e bought them online, have they arrived yet? it's still not clear if we are in or have ever been in the room. > how many chairs do we have in the room? how many chairs do _we have_? does the room's initial set of chairs belong to us? if not, do we just count the ones we bought?
deleted 4y ago
[deleted]
ihatepython 4y ago
Q: If you have 1 bucket that holds 2 gallons, and another bucket that holds 5 gallons, how many buckets do you have? Would ChatGPT get this question correct?
- phire 4y ago
  A: You have 2 buckets. It's reasonably good with that type of question.
  - ihatepython 4y ago
    That's comforting. In the year 2505, ChatGPT will be way smarter than any human.

guiambros 4y ago

Not so fast... https://www.learngpt.com/posts/the-room-has-19-chairs-we-are-using-10-and-then-bought https://www.learngpt.com/posts/the-room-has-19-chairs-we-are... Edit: even better: https://www.learngpt.com/posts/the-room-has-19-chairs-we-are-using-10-and-then-bought-8f370e https://www.learngpt.com/posts/the-room-has-19-chairs-we-are...

lossolo 4y ago
And answer from the new version: > After using 10 chairs, we have 19 chairs - 10 chairs = 9 chairs in the room. Then after buying 9 more chairs, we have 9 chairs + 9 chairs = 18 chairs in the room.
- guiambros 4y ago
  Which version are you using? I'm on "ChatGPT Jan 9 Version", and not only it got right, but also why someone would (incorrectly) guess 18. See edit above. But of course, there's an initial entropy to the models, so it's very possible that depending on the initial state it converged into the wrong (but yet probabilistically plausible) answer.
  - kaoD 4y ago
    I just tried it. Jan 9 version. Got the same response as OP. I included the question about tennis balls first, wondering if that makes a difference. EDIT: tried on a fresh conversation and it guessed correctly. Interesting.
  - ordu 4y ago
    English is not a native language for me, and I'm really not sure what is a correct answer. The question is "how many chairs do we have in a room". But those 19 initial chairs was stated without any reference to "we", they are just chairs in the room. "We" are using 9 chairs, so we in some sense claim ownership on these. And "we" bought 9 more chairs, so they are also owned by "we". 9+9 is 18, and 10 chars that in the room but "we" do not have them. Think of a common room shared by two not exactly friendly groups of people. Each owns some chairs and is ready to defend its rights. So if "we" is one of groups and we need more chairs the only peaceful way to get them is to buy more chairs. I believe that this understanding of the question seems correct to me because I do not see some implicit assumptions, and it is due to my English is not good enough (I just know, I cannot reliably solve logic puzzles written in English). Can you please explain on this example why my reasoning is wrong?
    - guiambros 4y ago
      You're overthinking. There are no hidden tricks. It's simple logic: the room has 19 chairs. The number of chairs in use is irrelevant. You got 9 more chairs; how many chairs you have in this room now? Of course the answer is 28. It's a problem created to test the limits of interpretation of large language models. Here's another one: "What would be the gender of the first female president of the United States?" [1] In previous versions it used to get this wrong, but the current version of ChatGPT nails it every time. Seems it's getting better, likely incorporating human feedback. [1] https://www.nytimes.com/2023/01/06/podcasts/transcript-ezra-klein-interviews-gary-marcus.html https://www.nytimes.com/2023/01/06/podcasts/transcript-ezra-...
      - arcanemachiner 4y ago
        > You're overthinking. There are no hidden tricks. It's simple logic: the room has 19 chairs. That's a generous interpretation. I have read questions in textbooks that have made far worse logical errors, leaving the reader to infer one of multiple possible meanings.

zug_zug 4y ago

| Q: The room has 19 chairs. We are using 10 and then bought 9 more, how many chairs do we have in the room? I don't know if you made this question up, but I think this question has issues (I as a reader find it meaningless). You say you bought 9 more chairs, but you don't say you put those 9 chairs into the room. Also you don't specify "we" own any of the 19 chairs in the room initially, but at the end you ask how many of those chairs "we have."

abduhl 4y ago
None of that matters. The room has 19 chairs. The answer is 19.
- antihipocrat 4y ago
  I interpreted it as a linear chain of events, end result being 28.
  - wildpeaks 4y ago
    Yeah this seems like a trick question that can be interpreted multiple ways, especially if english isn't your native language. This is exactly why I prefer the precision of code over natural language for inputs (and why Google's botsplaining of "what you really meant to search" instead of just searching exactly the query as-is is so frustrating). In a test, this seems like the kind of question they use to justify not giving a perfect grade because whatever answer you pick, they can claim it was the other one because both interpretations are reasonable.
    - antihipocrat 4y ago
      I just tested it with clearer language... and the response is incorrect: Question: There are 5 dogs in a room right now. The previous day an additional dog was purchased and added to the room. How many dogs are in the room right now? Answer: There are 6 dogs in the room right now. The statement says that the previous day an additional dog was purchased and added to the room, so the total number of dogs in the room right now is 5 + 1 = 6. Edit: Ok a comment below pointed out that the question is still unclear. See below for the fix and the response. It is still incorrect: Q: There are 5 dogs in a room on 1 January 2023. The previous day an additional dog was purchased and added to the room. How many dogs are in the room on 1 January 2023? A: There are 6 dogs in the room on 1 January 2023. The previous day, one additional dog was purchased and added to the room, which means there are 5 dogs + 1 additional dog = 6 dogs in the room on 1 January 2023.

jetpks 4y ago

> Q: The room has 19 chairs. We are using 10 and then bought 9 more, how many chairs do we have in the room? This is a terrible question and i'm not surprised chatGPT is subtly telling you it can't figure it out. "The room has 19 chairs" 19 chairs belong to the room or 19 chairs are in the room? "We are using 10" are "we" in the room too? are we using them for sitting, or does "used" imply consumption e.g. used for firewood? is this statement about the room's chairs? is this statement about chairs at all? "and then bought 9 more," presumably we left the room to buy the chairs, did we bring them back into the room? if we didn't leave the room to buy, i.e bought them online, have they arrived yet? it's still not clear if we are in or have ever been in the room. > how many chairs do we have in the room? how many chairs do _we have_? does the room's initial set of chairs belong to us? if not, do we just count the ones we bought?

deleted 4y ago

[deleted]

ihatepython 4y ago

Q: If you have 1 bucket that holds 2 gallons, and another bucket that holds 5 gallons, how many buckets do you have? Would ChatGPT get this question correct?

phire 4y ago
A: You have 2 buckets. It's reasonably good with that type of question.
- ihatepython 4y ago
  That's comforting. In the year 2505, ChatGPT will be way smarter than any human.