4 ms·

From the ChatGPT app, using GPT-4 model, using my own custom prompt for such puzzles: “You are better than Marilyn Vos Savant at solving brain teaser logic puzz

From the ChatGPT app, using GPT-4 model, using my own custom prompt for such puzzles: “You are better than Marilyn Vos Savant at solving brain teaser logic puzzles step by step. Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?” Certainly! Let's break it down step by step: 1. Sally has 3 brothers. 2. Each of those brothers has two sisters. Since all the brothers share the same two sisters, we are essentially hearing about the same sisters each time, not accumulating more sisters with each brother mentioned. 3. We know that Sally is a sister to her brothers, so she is one of the two sisters each brother has. By putting it all together, we find that there must be another sister. Therefore, Sally has one sister. --- New chats with no history, no custom prompt, and no system prompt, first interaction in the chat this question: “Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?” All Answers (not cherry picked): Sally has 1 sister. In the description, Sally is counted as one of the two sisters that each brother has. This means the brothers have another sister apart from Sally. . . . Sally has 1 sister. In the description, it mentioned that "Each brother has 2 sisters." This implies that there is Sally and another sister, since we know that all the brothers share the same sisters. . . . Sally has one sister. Each of her brothers has two sisters, counting Sally and the other sister. . . . Sally has 1 sister. Here's how we figure it out: Sally has 3 brothers, and each of these brothers has 2 sisters. Since the brothers share the same sisters, we know that the two sisters must be Sally and another girl. So, Sally has 1 sister. . . . I have gotten zero wrong answers from GPT-4 in OpenAI’s ChatGPT app.

deleted 3y ago
[deleted]
facu17y 3y ago
It might be trained on this question or a variant of it.
- civilitty 3y ago
  It's certainly RLHFed. All of the logic puzzles I use for evaluation that used to fail months ago now pass no problem and I've even had a hard time modifying them to fail.
  - appplication 3y ago
    This is sort of a bummer because it’s not actually an improvement to the model, but just a patch job to artificially inflate performance. All it does is make true evaluation more difficult. Classic “you get what you measure”.
    - civilitty 3y ago
      Maybe, maybe not. The magic of LLMs is their ability to generalize both from the human language in the data set and examples in the prompt. If RLHF training improves on that generalization, then it's just a matter of getting a big enough high quality dataset (and not crippling it with censorship). This is probably what's given OpenAI their initial advantage. Time will tell I guess.
    - FrustratedMonky 3y ago
      Are you implying that to counter these logic puzzles that GPT4 was specifically trained on logic puzzles so it would know the answers? In that case, just make new problems. If it is being 'patched' to pass specific known problems, then the new ones would fail. If it is able to answer them, then maybe it is actually analyzing them and working out the solution. Not sure how you can assume there was no underlying improvement, and these are cases of feeding it the answers.
      - thaumasiotes 3y ago
        > Not sure how you can assume there was no underlying improvement, and these are cases of feeding it the answers. Compare > And it's only fixed for the stated case, but if you reverse the genders, GPT-4 gets it wrong.
    - ruszki 3y ago
      I don’t think that it’s not an improvement. It’s not an improvement in context of finding new genuine solutions, sure. But that’s definitely not needed most of the time in real life for an average person, just like it’s not needed for an average developer anymore.
ddol 3y ago
I first asked GPT-4 “james (a boy) has four sisters, and each sister has three brothers. how many brothers does james have?”[0] and it told me 3 brothers. I then asked it to think though step by step and it said 2. [0] https://chat.openai.com/share/859baa94-3edb-40f0-9c74-8b1403ea84bc https://chat.openai.com/share/859baa94-3edb-40f0-9c74-8b1403...
d0mine 3y ago
I got a wrong answer (via API): "Sally has 2 sisters. The three brothers are all siblings, so when it's said that each brother has 2 sisters, it means they all share the same two sisters - one of them being Sally."

deleted 3y ago

[deleted]

facu17y 3y ago

It might be trained on this question or a variant of it.

civilitty 3y ago
It's certainly RLHFed. All of the logic puzzles I use for evaluation that used to fail months ago now pass no problem and I've even had a hard time modifying them to fail.
- appplication 3y ago
  This is sort of a bummer because it’s not actually an improvement to the model, but just a patch job to artificially inflate performance. All it does is make true evaluation more difficult. Classic “you get what you measure”.
  - civilitty 3y ago
    Maybe, maybe not. The magic of LLMs is their ability to generalize both from the human language in the data set and examples in the prompt. If RLHF training improves on that generalization, then it's just a matter of getting a big enough high quality dataset (and not crippling it with censorship). This is probably what's given OpenAI their initial advantage. Time will tell I guess.
  - FrustratedMonky 3y ago
    Are you implying that to counter these logic puzzles that GPT4 was specifically trained on logic puzzles so it would know the answers? In that case, just make new problems. If it is being 'patched' to pass specific known problems, then the new ones would fail. If it is able to answer them, then maybe it is actually analyzing them and working out the solution. Not sure how you can assume there was no underlying improvement, and these are cases of feeding it the answers.
    - thaumasiotes 3y ago
      > Not sure how you can assume there was no underlying improvement, and these are cases of feeding it the answers. Compare > And it's only fixed for the stated case, but if you reverse the genders, GPT-4 gets it wrong.
  - ruszki 3y ago
    I don’t think that it’s not an improvement. It’s not an improvement in context of finding new genuine solutions, sure. But that’s definitely not needed most of the time in real life for an average person, just like it’s not needed for an average developer anymore.

ddol 3y ago

I first asked GPT-4 “james (a boy) has four sisters, and each sister has three brothers. how many brothers does james have?”[0] and it told me 3 brothers. I then asked it to think though step by step and it said 2. [0] https://chat.openai.com/share/859baa94-3edb-40f0-9c74-8b1403ea84bc https://chat.openai.com/share/859baa94-3edb-40f0-9c74-8b1403...

d0mine 3y ago

I got a wrong answer (via API): "Sally has 2 sisters. The three brothers are all siblings, so when it's said that each brother has 2 sisters, it means they all share the same two sisters - one of them being Sally."