5 ms·
GPT4 API and ChatGPT both get it wrong: "Sally has 2 sisters. Each of her brothers has 2 sisters because Sally and her 2 sisters make up the group of siblings e
by yaj54 3y ago
GPT4 API and ChatGPT both get it wrong:
"Sally has 2 sisters. Each of her brothers has 2 sisters because Sally and her 2 sisters make up the group of siblings each brother has."
GPT4 w/ CoT system prompting gets it right:
SYS: "You are a helpful assistant. Think through your work step by step before providing your answer."
USER: "Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?"
ASSIST: "Let's break down the information given:
1. Sally has 3 brothers.
2. Each brother has 2 sisters.
Given that Sally is a girl, she is one of the sisters. Since each brother has 2 sisters and we know Sally is one of them, there must be one more sister to make the statement true for each brother.
Therefore, Sally has 1 sister."
The importance of prompting makes it quite difficult to compare model peak performance. Especially since different models have different styles of prompts that generate peak performance.
- COAGULOPATH 3y agoGemini Ultra gets this right. (Usually it's worse at GPT4 at these sorts of questions.)
- fragmede 3y agoThanks. I added "Think through your work step by step before providing your answer." to my custom prompt and it fixes my gpt 4. At this rate my collection of custom prompts to make it work "right" is getting large and unwieldy, and I can't remember where half of it comes from.
- auggierose 3y agoIs w/ supposed to mean "with". And if so, why not just write "with"? Also, I guess CoT means Chain of Thought?
- MeImCounting 3y agoWhy not just write you are instead of youre? Why shorten any word ever? Yes CoT is chain of thought
- sigmoid10 3y agoDid you use GPT3.5 for chat? I just tried it on vanilla ChatGPT using GPT4 with no extra stuff and it immediately gets the correct answer: "Sally has 3 brothers, and each of them has 2 sisters. The description implies that Sally's brothers are her only siblings. Therefore, the two sisters each brother has must be Sally and one other sister. This means Sally has just one sister."
- littlestymaar 3y agoThat's the problem with nondeterministic generative stuff: sometimes it get things right, and sometimes it doesn't and you cannot rely on any behavior.
- sigmoid10 3y agoI tried it 10 times and while the wording is different, the answer remained correct every time. I used the exact question from the comment above, nothing else. While determinism is a possible source of error, I find that in these cases people usually just use the wrong model on ChatGPT for whatever reason. And unless you set the temperature way too high, it is pretty unlikely that you will end up outside of correct responses as far as the internal world model is concerned. It just mixes up wording by using the next most likely tokens. So if the correct answer is "one", you might find "single" or "1" as similarly likely tokens, but not "two." For that to happen something must be seriously wrong either in the model or in the temperature setting.
- kenjackson 3y agoI got an answer with GPT-4 that is mostly wrong: "Sally has 2 sisters. Since each of her brothers has 2 sisters, that includes Sally and one additional sister." I think said, "wait, how many sisters does Sally have?" And then it answered it fully correctly.
- sigmoid10 3y agoThe only way I can get it to consistently generate wrong answers (i.e. two sisters) is by switching to GPT3.5. That one just doesn't seem capable of answering correctly on the first try (and sometimes not even with careful nudging).
- compumetrika 3y agoDo you have a custom "intro prompt?" I have one that describes a lot of statistical work I want GPT to help me with. I got this result the first try: > Sally has 2 sisters. Since each brother has 2 sisters, and Sally is one of them, there must be one other sister making it two sisters in total. >