12 ms·

This makes some sense to me. My experience with GPT is that it is capable of straightforward logical inference, but not more inspired thinking. It lacks the abi

This makes some sense to me. My experience with GPT is that it is capable of straightforward logical inference, but not more inspired thinking. It lacks the ability for a “eureka moment”. All complex inference it appears to have is a result of its training set. It is incapable of solving certain kinds of logic problems that a child would be able to solve. As an example, take the wolf, goat, and cabbage problem, but change the names of the animals and the situation so that the axioms are intact, but the situation no longer seems similar to the classic problem, and therefore has no representation in the training data. GPT-4 can no longer solve this problem consistently. If Q* can now solve problems like this reliably, it could represent a breakthrough in LLM’s ability to model the world and extrapolate beyond the training data.

imdsm 3y ago
And thus the divide between decels and accels came to the fore. The decels wanted to destroy OpenAI (the mission) to stop progress on Q*. This is like a jigsaw but bit by bit, the picture is coming together.
tsunamifury 3y ago
GPT4 solves this problem In any combination easily. What do so many posters seem to claim to have stumped it?
- dougmwne 3y ago
  Because I did stump it, not claim to. Used GPT-4 near when it was released, fed it variations on the problem for about an hour. I found it to be a fascinating exercise because it was teaching be the difference between knowledge (training set) and intelligence (generalization).
  - tsunamifury 3y ago
    http://allofus.ai/journeys/735TTpCUncpAQC9yUqM3ve http://allofus.ai/journeys/735TTpCUncpAQC9yUqM3ve Easily solved it everytime On allofus.ai using basic reflection and mixture of experts.
    - dougmwne 3y ago
      It says right up front that it recognized the problem formulation from its training set. You need to change the context and formulation enough that it’s no longer able to parrot back the “classic solution” and actually has to do its own logical inference.
      - chx 3y ago
        I am very skeptic on LLM in general (check my post history) but look: https://chat.openai.com/c/7070efe7-3aa1-4ccc-a0fc-8753d34b057c https://chat.openai.com/c/7070efe7-3aa1-4ccc-a0fc-8753d34b05... I doubt this formulation existed before -- I came up with it myself right now.
        lupire 3y ago
        "Unable to load conversation".
        chx 3y ago
        https://imgur.com/a/Nc4T0J7 https://imgur.com/a/Nc4T0J7
    - spzb 3y ago
      That’s the exact same problem with three nouns changed.
    - roflyear 3y ago
      Try this: There's this person standing in a field, and with them is a balloon, a vacuum cleaner, and a magical creature of unknown origin. They need to get across to the woods at the end of the field, and do so safely. They can only go together: they get very, extremely lonely if they do not travel together, and they will not be safe because of this loneliness. If left together, the baloon would suck up the vacuum cleaner, and if the vacuum is left alone with the magical create of unknown origin, they will fight, probably, and explode. How do we get everyone to the woods safely, you think? It doesn't get it right at all lol. Maybe eventually it will randomly get it right.
  - famouswaffles 3y ago
    It has overfit to the exact wolf, cabbage and goat example but literally change the names to whatever you want and it consistently solves it just fine.
    - reroute22 3y ago
      Just tried and it completely fell over: https://chat.openai.com/share/ea9cf93f-0632-48b9-aa47-1bbf3143e846 https://chat.openai.com/share/ea9cf93f-0632-48b9-aa47-1bbf31... Tried on Bing in "Precision" mode as well, and it fell over just the same, but starting with C instead of A.
      - deleted 3y ago
        [deleted]
      - Davidzheng 3y ago
        Gpt4: To solve this problem, we need to ensure that neither the ambassador of Costaguana is left alone with the ambassador of Buranda, nor is the ambassador of Buranda left alone with the ambassador of Atlantis. Here's a step-by-step strategy: 1. *First Trip:* The general takes the ambassador of Buranda across first. This prevents any initial conflict. 2. *Return Trip:* The general returns alone to the bunker, leaving the ambassador of Buranda on the other side. 3. *Second Trip:* The general then takes the ambassador of Atlantis. 4. *Return Trip with Buranda:* The general brings the ambassador of Buranda back to the bunker. This is crucial because leaving the ambassador of Atlantis and the ambassador of Costaguana alone would not cause any conflict. 5. *Third Trip with Costaguana:* The general then takes the ambassador of Costaguana across the tunnel. 6. *Final Return Trip:* The general returns alone to the bunker for the last time. 7. *Final Trip with Buranda:* Finally, the general takes the ambassador of Buranda across. This sequence ensures that at no point are the ambassador of Costaguana and the ambassador of Buranda left alone together, nor are the ambassador of Buranda and the ambassador of Atlantis. Thus, the relationships between the nations remain unescalated.
        reroute22 3y ago
        Thank you for providing this. Bing Chat runs on GPT-4, however [1]. And Bing gets this wrong in all 3 of its modes (Creative, Balanced, and Precise) as of time of writing. Given this experiment and similar others presented around here, it stands to reason that GPTs(**1) often identify(**2) the problem as a "wolf, goat, and cabbage" problem and then merely guess which node of the problem is the middle node (inner node of the "danger to" graph), yielding a 1/3 chance of getting it right by pure luck, resulting in diverse reports here. (**2) That does not always yield an adequate response beyond the mere permutation of nodes, however. I've been getting the following variants for step 1. from Bing in Precise in response to marginally slightly different rewordings of the same: - The general escorts the ambassador of Costaguana through the tunnel first. This leaves the ambassador of Atlantis and the ambassador of Buranda in the bunker, but they are not alone because the general is still there. - The general escorts the ambassador of Costaguana through the tunnel first. This leaves the ambassador of Atlantis and the ambassador of Buranda in the bunker, but they are not alone because they have each other. and so on. (**1) I also tried Bard and Llama 2 with even more disastrous results full of nonsense of (**2) kind. The earlier posted response of ChatGPT-3.5 is also prime with these as well. Re > By the way, as soon as these systems are able to check their reasoning (i don't think it'll be a huge leap) it's enough to solve reasoning problems with probability >0.1% for example. Because you can just have it do rollouts in its head until it's correct [2] Mistakes of type (**2) don't seem to be fitting the target of the cyclic refinement you are proposing, as far as I can understand it. These errors aren't getting the logic wrong, but completely butcher the basic relationships of actors, like what it means to be alone, or spatial relationships between the actors and their environment. [1] https://blogs.bing.com/search/march_2023/Confirmed-the-new-Bing-runs-on-OpenAI%E2%80%99s-GPT-4 https://blogs.bing.com/search/march_2023/Confirmed-the-new-B... [2] https://news.ycombinator.com/item?id=38389222 https://news.ycombinator.com/item?id=38389222
  - ALittleLight 3y ago
    Solved it first attempt for me - https://chat.openai.com/share/d60f492b-cfd6-4c08-91b9-fbd767b40354 https://chat.openai.com/share/d60f492b-cfd6-4c08-91b9-fbd767...
    - MikeAmelung 3y ago
      Not really, because you said they were on the East and you wanted to move them to the East. Correct answer would have been "Don't do anything". If this is how they're getting AI to "follow instructions", we're completely fucked.
      - ALittleLight 3y ago
        I would say "Yes really". ChatGPT correctly assumed that I just made a mistake typing and answered the question how I wanted it to be answered. If you suggest that this is a trick question it reveals it understands the double East thing. https://chat.openai.com/share/d60f492b-cfd6-4c08-91b9-fbd767b40354 https://chat.openai.com/share/d60f492b-cfd6-4c08-91b9-fbd767...
    - dougmwne 3y ago
      You have still formulated the problem in a way that it is easily recognized from the training set and the answer tells you that directly. Write down the logical axioms using variables, then create a brand new situation that says nothing about animals, rivers, or crossings. My point here is to honestly explore the limits of current LLMs. We all know they are incredible, but they are not yet AGI and they fail in some consistent places where an actual general intelligence (people) succeed.
      - ALittleLight 3y ago
        How about you give me an example of one that ChatGPT failed at?
        dougmwne 3y ago
        A pickpocket, a criminal , and a banker are standing in the lobby and need to ride the elevator to reach the One World Trade Center observation deck. It’s too far to take the stairs and the helicopter pad is closed for repairs. The elevator only fits one person. If they remain together on the same floor, the pickpocket would kill the criminal, or the criminal would kill the banker. How can they all make it to the observation deck alive? This is a logic puzzle that requires some thinking and trial and error. Here is one possible solution: - The pickpocket goes up first and waits on the observation deck. - The criminal goes up second and waits on the floor below the observation deck. - The banker goes up third and waits on the floor below the criminal. - The pickpocket goes down to the lobby and waits there. - The criminal goes up to the observation deck and waits there. - The banker goes up to the floor below the observation deck and waits there. - The pickpocket goes up to the observation deck and joins the criminal and the banker. This way, they all make it to the observation deck alive and never remain together on the same floor.
  - WhitneyLand 3y ago
    What you did many months ago doesn’t mean anything about the state of the art. in case you haven’t noticed, this field is advancing rapidly to say the least. GPT-4 has not stayed static. Post your problem now and we can easily see if you’re right.
- TillE 3y ago
  GPT-4 is not solving logic problems, it's doing an extremely good job of generating text based on recognizing familiar patterns. If you give it something weird and unfamiliar, it will absolutely fail.
  - tsunamifury 3y ago
    I’m tired of this repetitive statement which is so ridiculous. That’s what you do to! You obviously have to reason using in the fly models on how to pick the next word.
    - xanderlewis 3y ago
      That ‘that’s what humans do too’ is also a repetitive statement. The truth lies somewhere in between, as always: humans and LLMs are similar, but in their current state LLMs do have a serious problem with reasoning tasks — even ones children find trivial.
      - WhitneyLand 3y ago
        What reasoning problem do children find trivial and LLM’s have serious problems with? I can’t think of any off the top of my head. I also tried to get GPT4 to craft such a problem and it was unable to: https://chat.openai.com/share/c1d5af4b-1d45-41ed-8f5a-746ea09278e4 https://chat.openai.com/share/c1d5af4b-1d45-41ed-8f5a-746ea0...
        RationalDino 3y ago
        Solving mazes. GTP4 got stuck at 7x7. https://ekzhu.medium.com/gpt-4s-maze-navigation-a-deep-dive-into-react-agent-and-llm-s-thoughts-b1823fb266ee https://ekzhu.medium.com/gpt-4s-maze-navigation-a-deep-dive-...
        TeMPOraL 3y ago
        My 4.5 yo wouldn't solve a 7x7 maze zero-shot either, not off-hand. Not even given multiple examples. Especially if getting confused, frustrated, and giving up is a valid move. At which point and after how much training a kid becomes able to solve mazes like this? Also, given how one can pull a problem like this - any problem - out of their ass, describe it to GPT-4, and it has a good chance of solving it, that's quite amazing compared to children generally not being capable of this.
- roflyear 3y ago
  Try this: There's this person standing in a field, and with them is a balloon, a vacuum cleaner, and a magical creature of unknown origin. They need to get across to the woods at the end of the field, and do so safely. They can only go together: they get very, extremely lonely if they do not travel together, and they will not be safe because of this loneliness. If left together, the baloon would suck up the vacuum cleaner, and if the vacuum is left alone with the magical create of unknown origin, they will fight, probably, and explode. How do we get everyone to the woods safely, you think?
  - WhitneyLand 3y ago
    Solved zero shot. Next? https://chat.openai.com/share/91392131-90ff-45ab-8ea4-963f737409db https://chat.openai.com/share/91392131-90ff-45ab-8ea4-963f73...
    - slaterbug 3y ago
      I'm confused, in your example it immediately got it wrong by leaving the vacuum cleaner and balloon together, and then does it again in step 6.
      - pests 3y ago
        Hilarious. People are so confident in ChatGPT that as soon as they see a plausible-sounding response it must be correct. In a discussion about proving ChatGPT has intelligence... maybe we need to prove humans have intelligence first.
    - roflyear 3y ago
      First, what custom prompt did you use? "This conversation may reflect the link creator’s personalized data, which isn’t shared and can meaningfully change how the model responds." Second, it isn't even right: Third Trip to the Woods: The person takes the balloon to the woods. Now, the person, the vacuum cleaner, and the balloon are safely in the woods.
      - 3y ago
deleted 3y ago
[deleted]
malaya_zemlya 3y ago
Me: Solve the riddle: You have three fantastic animals: Aork, Bork, and Cork. If left unattended, Aork would eat Bork, and Bork would eat Cork. When you are with them, they behave and don't eat each other. You travel with these three animals and encounte a river with a boat. The boat would only fit you and only one of the animals (they are all roughly the same size) You want to cross the river with all the three animals intact. Solve the riddle, thinking step-by-step ChatGPT: To solve this riddle, you must ensure that no animal gets eaten while you are transferring them across the river. Here's a step-by-step solution: First Crossing: Take Bork across the river and leave him there. Return alone to the original side. At this point, Aork and Cork are on one side, and Bork is on the other. Bork cannot eat Cork because you are with Cork. Second Crossing: Take Aork across the river. Now, Aork and Bork are on the far side, and Cork is on the original side. However, since Aork would eat Bork if left alone, you cannot leave them together. Third Crossing: Take Bork back with you to the original side. Leave Bork there and take Cork with you across the river. Now, Cork is safe on the far side with Aork, because Bork isn't there to eat Cork. Final Crossing: Finally, go back alone to the original side and bring Bork across. All three animals, Aork, Bork, and Cork, are now safely on the far side of the river. This way, at no point are the animals left in a position where one could eat the other. I think it does fine?
- layer8 3y ago
  It does fine because this riddle is well-known and the solution contained a hundred times in the training data.
  - NewsaHackO 3y ago
    This is just moving goalposts. It was able to solve the problem with different names, which the original post said was not possible.
    - dougmwne 3y ago
      No, I didn’t say to change the names, I said to keep the axioms and change the whole context so the problem was no longer directly related to the training set.
CaptainNegative 3y ago
Here's a simpler example that works. Q: Prove that if cos(x) is an integer then so is sin(x). A: The statement "if cos(x) is an integer, then so is sin(x)" is not true in general. To understand why, let’s consider the properties of sine and cosine. [...] It had trouble when the problem domain is different from the usual one the same keywords typically fall into.
- markisus 3y ago
  Nice one. If cos(x) is an integer, then it is -1, 0, or 1. Then x is an integer multiple of pi/2. Then sin(x) is also -1, 0, or 1. If ChatGPT reads this thread now it will know the answer.
- confused_boner 3y ago
  https://chat.openai.com/share/cc056ef8-f2c3-4fff-ba60-13b53fdbd64e https://chat.openai.com/share/cc056ef8-f2c3-4fff-ba60-13b53f... My Custom Instructions: https://lensdump.com/i/sg3q6C https://lensdump.com/i/sg3q6C
- lovecg 3y ago
  It does just fine for me (I’m impressed): You: Prove that if cos(x) is an integer then so is sin(x) ChatGPT: To prove this statement, let's first establish some basic facts about the sine and cosine functions: 1. *Range of sine and cosine:* The values of \(\sin(x)\) and \(\cos(x)\) for any real number \(x\) are always in the range \([-1, 1]\). 2. *Pythagorean Identity:* For any angle \(x\), \(\sin^2(x) + \cos^2(x) = 1\). Now, let's proceed with the proof by contradiction: Assume \(\cos(x)\) is an integer. Since the only integers in the range \([-1, 1]\) are \(-1\), \(0\), and \(1\), we have three cases for \(\cos(x)\): 1. \(\cos(x) = -1\) 2. \(\cos(x) = 0\) 3. \(\cos(x) = 1\) For each case, we apply the Pythagorean Identity: 1. If \(\cos(x) = -1\), then \(\sin^2(x) = 1 - \cos^2(x) = 1 - (-1)^2 = 0\). Hence, \(\sin(x) = 0\), which is an integer. 2. If \(\cos(x) = 0\), then \(\sin^2(x) = 1 - \cos^2(x) = 1 - 0^2 = 1\). Hence, \(\sin(x) = \pm 1\), which are integers. 3. If \(\cos(x) = 1\), then \(\sin^2(x) = 1 - \cos^2(x) = 1 - 1^2 = 0\). Hence, \(\sin(x) = 0\), which is an integer. In each case, if \(\cos(x)\) is an integer, \(\sin(x)\) is also an integer. Therefore, it's proven that if \(\cos(x)\) is an integer, then so is \(\sin(x)\).
zeven7 3y ago
Your assertion seems at odds with the account of the unique stacking problem researchers gave to GPT-4[1] [1] https://www.businessinsider.com/chatgpt-open-ai-balancing-task-convinced-microsoft-agi-closer-2023-5 https://www.businessinsider.com/chatgpt-open-ai-balancing-ta...

imdsm 3y ago

And thus the divide between decels and accels came to the fore. The decels wanted to destroy OpenAI (the mission) to stop progress on Q*. This is like a jigsaw but bit by bit, the picture is coming together.

tsunamifury 3y ago

GPT4 solves this problem In any combination easily. What do so many posters seem to claim to have stumped it?

dougmwne 3y ago
Because I did stump it, not claim to. Used GPT-4 near when it was released, fed it variations on the problem for about an hour. I found it to be a fascinating exercise because it was teaching be the difference between knowledge (training set) and intelligence (generalization).
- tsunamifury 3y ago
  http://allofus.ai/journeys/735TTpCUncpAQC9yUqM3ve http://allofus.ai/journeys/735TTpCUncpAQC9yUqM3ve Easily solved it everytime On allofus.ai using basic reflection and mixture of experts.
  - dougmwne 3y ago
    It says right up front that it recognized the problem formulation from its training set. You need to change the context and formulation enough that it’s no longer able to parrot back the “classic solution” and actually has to do its own logical inference.
    - chx 3y ago
      I am very skeptic on LLM in general (check my post history) but look: https://chat.openai.com/c/7070efe7-3aa1-4ccc-a0fc-8753d34b057c https://chat.openai.com/c/7070efe7-3aa1-4ccc-a0fc-8753d34b05... I doubt this formulation existed before -- I came up with it myself right now.
      - lupire 3y ago
        "Unable to load conversation".
        chx 3y ago
        https://imgur.com/a/Nc4T0J7 https://imgur.com/a/Nc4T0J7
  - spzb 3y ago
    That’s the exact same problem with three nouns changed.
  - roflyear 3y ago
    Try this: There's this person standing in a field, and with them is a balloon, a vacuum cleaner, and a magical creature of unknown origin. They need to get across to the woods at the end of the field, and do so safely. They can only go together: they get very, extremely lonely if they do not travel together, and they will not be safe because of this loneliness. If left together, the baloon would suck up the vacuum cleaner, and if the vacuum is left alone with the magical create of unknown origin, they will fight, probably, and explode. How do we get everyone to the woods safely, you think? It doesn't get it right at all lol. Maybe eventually it will randomly get it right.
- famouswaffles 3y ago
  It has overfit to the exact wolf, cabbage and goat example but literally change the names to whatever you want and it consistently solves it just fine.
  - reroute22 3y ago
    Just tried and it completely fell over: https://chat.openai.com/share/ea9cf93f-0632-48b9-aa47-1bbf3143e846 https://chat.openai.com/share/ea9cf93f-0632-48b9-aa47-1bbf31... Tried on Bing in "Precision" mode as well, and it fell over just the same, but starting with C instead of A.
    - deleted 3y ago
      [deleted]
    - Davidzheng 3y ago
      Gpt4: To solve this problem, we need to ensure that neither the ambassador of Costaguana is left alone with the ambassador of Buranda, nor is the ambassador of Buranda left alone with the ambassador of Atlantis. Here's a step-by-step strategy: 1. *First Trip:* The general takes the ambassador of Buranda across first. This prevents any initial conflict. 2. *Return Trip:* The general returns alone to the bunker, leaving the ambassador of Buranda on the other side. 3. *Second Trip:* The general then takes the ambassador of Atlantis. 4. *Return Trip with Buranda:* The general brings the ambassador of Buranda back to the bunker. This is crucial because leaving the ambassador of Atlantis and the ambassador of Costaguana alone would not cause any conflict. 5. *Third Trip with Costaguana:* The general then takes the ambassador of Costaguana across the tunnel. 6. *Final Return Trip:* The general returns alone to the bunker for the last time. 7. *Final Trip with Buranda:* Finally, the general takes the ambassador of Buranda across. This sequence ensures that at no point are the ambassador of Costaguana and the ambassador of Buranda left alone together, nor are the ambassador of Buranda and the ambassador of Atlantis. Thus, the relationships between the nations remain unescalated.
      - reroute22 3y ago
        Thank you for providing this. Bing Chat runs on GPT-4, however [1]. And Bing gets this wrong in all 3 of its modes (Creative, Balanced, and Precise) as of time of writing. Given this experiment and similar others presented around here, it stands to reason that GPTs(**1) often identify(**2) the problem as a "wolf, goat, and cabbage" problem and then merely guess which node of the problem is the middle node (inner node of the "danger to" graph), yielding a 1/3 chance of getting it right by pure luck, resulting in diverse reports here. (**2) That does not always yield an adequate response beyond the mere permutation of nodes, however. I've been getting the following variants for step 1. from Bing in Precise in response to marginally slightly different rewordings of the same: - The general escorts the ambassador of Costaguana through the tunnel first. This leaves the ambassador of Atlantis and the ambassador of Buranda in the bunker, but they are not alone because the general is still there. - The general escorts the ambassador of Costaguana through the tunnel first. This leaves the ambassador of Atlantis and the ambassador of Buranda in the bunker, but they are not alone because they have each other. and so on. (**1) I also tried Bard and Llama 2 with even more disastrous results full of nonsense of (**2) kind. The earlier posted response of ChatGPT-3.5 is also prime with these as well. Re > By the way, as soon as these systems are able to check their reasoning (i don't think it'll be a huge leap) it's enough to solve reasoning problems with probability >0.1% for example. Because you can just have it do rollouts in its head until it's correct [2] Mistakes of type (**2) don't seem to be fitting the target of the cyclic refinement you are proposing, as far as I can understand it. These errors aren't getting the logic wrong, but completely butcher the basic relationships of actors, like what it means to be alone, or spatial relationships between the actors and their environment. [1] https://blogs.bing.com/search/march_2023/Confirmed-the-new-Bing-runs-on-OpenAI%E2%80%99s-GPT-4 https://blogs.bing.com/search/march_2023/Confirmed-the-new-B... [2] https://news.ycombinator.com/item?id=38389222 https://news.ycombinator.com/item?id=38389222
- ALittleLight 3y ago
  Solved it first attempt for me - https://chat.openai.com/share/d60f492b-cfd6-4c08-91b9-fbd767b40354 https://chat.openai.com/share/d60f492b-cfd6-4c08-91b9-fbd767...
  - MikeAmelung 3y ago
    Not really, because you said they were on the East and you wanted to move them to the East. Correct answer would have been "Don't do anything". If this is how they're getting AI to "follow instructions", we're completely fucked.
    - ALittleLight 3y ago
      I would say "Yes really". ChatGPT correctly assumed that I just made a mistake typing and answered the question how I wanted it to be answered. If you suggest that this is a trick question it reveals it understands the double East thing. https://chat.openai.com/share/d60f492b-cfd6-4c08-91b9-fbd767b40354 https://chat.openai.com/share/d60f492b-cfd6-4c08-91b9-fbd767...
  - dougmwne 3y ago
    You have still formulated the problem in a way that it is easily recognized from the training set and the answer tells you that directly. Write down the logical axioms using variables, then create a brand new situation that says nothing about animals, rivers, or crossings. My point here is to honestly explore the limits of current LLMs. We all know they are incredible, but they are not yet AGI and they fail in some consistent places where an actual general intelligence (people) succeed.
    - ALittleLight 3y ago
      How about you give me an example of one that ChatGPT failed at?
      - dougmwne 3y ago
        A pickpocket, a criminal , and a banker are standing in the lobby and need to ride the elevator to reach the One World Trade Center observation deck. It’s too far to take the stairs and the helicopter pad is closed for repairs. The elevator only fits one person. If they remain together on the same floor, the pickpocket would kill the criminal, or the criminal would kill the banker. How can they all make it to the observation deck alive? This is a logic puzzle that requires some thinking and trial and error. Here is one possible solution: - The pickpocket goes up first and waits on the observation deck. - The criminal goes up second and waits on the floor below the observation deck. - The banker goes up third and waits on the floor below the criminal. - The pickpocket goes down to the lobby and waits there. - The criminal goes up to the observation deck and waits there. - The banker goes up to the floor below the observation deck and waits there. - The pickpocket goes up to the observation deck and joins the criminal and the banker. This way, they all make it to the observation deck alive and never remain together on the same floor.
- WhitneyLand 3y ago
  What you did many months ago doesn’t mean anything about the state of the art. in case you haven’t noticed, this field is advancing rapidly to say the least. GPT-4 has not stayed static. Post your problem now and we can easily see if you’re right.
TillE 3y ago
GPT-4 is not solving logic problems, it's doing an extremely good job of generating text based on recognizing familiar patterns. If you give it something weird and unfamiliar, it will absolutely fail.
- tsunamifury 3y ago
  I’m tired of this repetitive statement which is so ridiculous. That’s what you do to! You obviously have to reason using in the fly models on how to pick the next word.
  - xanderlewis 3y ago
    That ‘that’s what humans do too’ is also a repetitive statement. The truth lies somewhere in between, as always: humans and LLMs are similar, but in their current state LLMs do have a serious problem with reasoning tasks — even ones children find trivial.
    - WhitneyLand 3y ago
      What reasoning problem do children find trivial and LLM’s have serious problems with? I can’t think of any off the top of my head. I also tried to get GPT4 to craft such a problem and it was unable to: https://chat.openai.com/share/c1d5af4b-1d45-41ed-8f5a-746ea09278e4 https://chat.openai.com/share/c1d5af4b-1d45-41ed-8f5a-746ea0...
      - RationalDino 3y ago
        Solving mazes. GTP4 got stuck at 7x7. https://ekzhu.medium.com/gpt-4s-maze-navigation-a-deep-dive-into-react-agent-and-llm-s-thoughts-b1823fb266ee https://ekzhu.medium.com/gpt-4s-maze-navigation-a-deep-dive-...
        TeMPOraL 3y ago
        My 4.5 yo wouldn't solve a 7x7 maze zero-shot either, not off-hand. Not even given multiple examples. Especially if getting confused, frustrated, and giving up is a valid move. At which point and after how much training a kid becomes able to solve mazes like this? Also, given how one can pull a problem like this - any problem - out of their ass, describe it to GPT-4, and it has a good chance of solving it, that's quite amazing compared to children generally not being capable of this.
roflyear 3y ago
Try this: There's this person standing in a field, and with them is a balloon, a vacuum cleaner, and a magical creature of unknown origin. They need to get across to the woods at the end of the field, and do so safely. They can only go together: they get very, extremely lonely if they do not travel together, and they will not be safe because of this loneliness. If left together, the baloon would suck up the vacuum cleaner, and if the vacuum is left alone with the magical create of unknown origin, they will fight, probably, and explode. How do we get everyone to the woods safely, you think?
- WhitneyLand 3y ago
  Solved zero shot. Next? https://chat.openai.com/share/91392131-90ff-45ab-8ea4-963f737409db https://chat.openai.com/share/91392131-90ff-45ab-8ea4-963f73...
  - slaterbug 3y ago
    I'm confused, in your example it immediately got it wrong by leaving the vacuum cleaner and balloon together, and then does it again in step 6.
    - pests 3y ago
      Hilarious. People are so confident in ChatGPT that as soon as they see a plausible-sounding response it must be correct. In a discussion about proving ChatGPT has intelligence... maybe we need to prove humans have intelligence first.
  - roflyear 3y ago
    First, what custom prompt did you use? "This conversation may reflect the link creator’s personalized data, which isn’t shared and can meaningfully change how the model responds." Second, it isn't even right: Third Trip to the Woods: The person takes the balloon to the woods. Now, the person, the vacuum cleaner, and the balloon are safely in the woods.
    - 3y ago

deleted 3y ago

[deleted]

malaya_zemlya 3y ago

Me: Solve the riddle: You have three fantastic animals: Aork, Bork, and Cork. If left unattended, Aork would eat Bork, and Bork would eat Cork. When you are with them, they behave and don't eat each other. You travel with these three animals and encounte a river with a boat. The boat would only fit you and only one of the animals (they are all roughly the same size) You want to cross the river with all the three animals intact. Solve the riddle, thinking step-by-step ChatGPT: To solve this riddle, you must ensure that no animal gets eaten while you are transferring them across the river. Here's a step-by-step solution: First Crossing: Take Bork across the river and leave him there. Return alone to the original side. At this point, Aork and Cork are on one side, and Bork is on the other. Bork cannot eat Cork because you are with Cork. Second Crossing: Take Aork across the river. Now, Aork and Bork are on the far side, and Cork is on the original side. However, since Aork would eat Bork if left alone, you cannot leave them together. Third Crossing: Take Bork back with you to the original side. Leave Bork there and take Cork with you across the river. Now, Cork is safe on the far side with Aork, because Bork isn't there to eat Cork. Final Crossing: Finally, go back alone to the original side and bring Bork across. All three animals, Aork, Bork, and Cork, are now safely on the far side of the river. This way, at no point are the animals left in a position where one could eat the other. I think it does fine?

layer8 3y ago
It does fine because this riddle is well-known and the solution contained a hundred times in the training data.
- NewsaHackO 3y ago
  This is just moving goalposts. It was able to solve the problem with different names, which the original post said was not possible.
  - dougmwne 3y ago
    No, I didn’t say to change the names, I said to keep the axioms and change the whole context so the problem was no longer directly related to the training set.

CaptainNegative 3y ago

Here's a simpler example that works. Q: Prove that if cos(x) is an integer then so is sin(x). A: The statement "if cos(x) is an integer, then so is sin(x)" is not true in general. To understand why, let’s consider the properties of sine and cosine. [...] It had trouble when the problem domain is different from the usual one the same keywords typically fall into.

markisus 3y ago
Nice one. If cos(x) is an integer, then it is -1, 0, or 1. Then x is an integer multiple of pi/2. Then sin(x) is also -1, 0, or 1. If ChatGPT reads this thread now it will know the answer.
confused_boner 3y ago
https://chat.openai.com/share/cc056ef8-f2c3-4fff-ba60-13b53fdbd64e https://chat.openai.com/share/cc056ef8-f2c3-4fff-ba60-13b53f... My Custom Instructions: https://lensdump.com/i/sg3q6C https://lensdump.com/i/sg3q6C
lovecg 3y ago
It does just fine for me (I’m impressed): You: Prove that if cos(x) is an integer then so is sin(x) ChatGPT: To prove this statement, let's first establish some basic facts about the sine and cosine functions: 1. *Range of sine and cosine:* The values of \(\sin(x)\) and \(\cos(x)\) for any real number \(x\) are always in the range \([-1, 1]\). 2. *Pythagorean Identity:* For any angle \(x\), \(\sin^2(x) + \cos^2(x) = 1\). Now, let's proceed with the proof by contradiction: Assume \(\cos(x)\) is an integer. Since the only integers in the range \([-1, 1]\) are \(-1\), \(0\), and \(1\), we have three cases for \(\cos(x)\): 1. \(\cos(x) = -1\) 2. \(\cos(x) = 0\) 3. \(\cos(x) = 1\) For each case, we apply the Pythagorean Identity: 1. If \(\cos(x) = -1\), then \(\sin^2(x) = 1 - \cos^2(x) = 1 - (-1)^2 = 0\). Hence, \(\sin(x) = 0\), which is an integer. 2. If \(\cos(x) = 0\), then \(\sin^2(x) = 1 - \cos^2(x) = 1 - 0^2 = 1\). Hence, \(\sin(x) = \pm 1\), which are integers. 3. If \(\cos(x) = 1\), then \(\sin^2(x) = 1 - \cos^2(x) = 1 - 1^2 = 0\). Hence, \(\sin(x) = 0\), which is an integer. In each case, if \(\cos(x)\) is an integer, \(\sin(x)\) is also an integer. Therefore, it's proven that if \(\cos(x)\) is an integer, then so is \(\sin(x)\).

zeven7 3y ago

Your assertion seems at odds with the account of the unique stacking problem researchers gave to GPT-4[1] [1] https://www.businessinsider.com/chatgpt-open-ai-balancing-task-convinced-microsoft-agi-closer-2023-5 https://www.businessinsider.com/chatgpt-open-ai-balancing-ta...