3 ms·

I'd put it this way: LLMs may reason in the sense that sampling from the distribution of human generated text sometimes reproduces chains of thought that are va

I'd put it this way: LLMs may reason in the sense that sampling from the distribution of human generated text sometimes reproduces chains of thought that are valid, but this kind of changed sampling is the only compelling model of how they might reason (in my opinion). There might be some kind of "reasoning" in the weights and activations but by construction this is a single pass, feed forward, process calculating outputs from inputs, so if there is reasoning there, it can only be a approximately a single step.

onemoresoop 2y ago
That's right I did this on chat gpt and went off the rails quickly: a man with a wolf and a cabbage wants to cross a river/ There is a boat but the man can only bring one more item beside himself at a time. What are the steps the man needs to do in order to cross the river? The response was: To solve the puzzle where a man needs to cross a river with a wolf and a cabbage, while ensuring none of them are left together unsupervised (since the wolf will eat the sheep, and the sheep will eat the cabbage), follow these steps: 1.Man takes the sheep across the river (leaving the wolf and the cabbage on the original side). 2.Man returns alone. 3.Man takes the cabbage across the river. 4.Man leaves the cabbage on the other side and takes the sheep back 5.Man takes the wolf across the river. 6.Man returns alone. 7.Man takes the sheep across the river. 8.This way, at no point are the wolf and the sheep left alone together, nor are the sheep and the cabbage left alone together. at which I prompted: I didn't mention a sheep The response was corrected but wrongly: Oops, my mistake! Let’s solve the puzzle correctly with a man, a wolf, and a cabbage: 1.Man takes the wolf across the river (leaving the cabbage on the original side). 2.Man returns alone. 3.Man takes the cabbage across the river. 4.Man leaves the cabbage on the other side and takes the wolf back. 5.Man takes the man across the river. 6.Man returns alone. 7.Man takes the wolf across the river. 8.Now, both the man and the wolf are across the river, with no danger to the cabbage. At step 3 the problem was solved but it kept on going
- bckr 2y ago
  Oh wow I got similar results with GPT4o. I’m surprised! It even messed up the second round in a similar way.
  - swatcoder 2y ago
    Presumably, because river crossing challenges are an important benchmark in satisfying AI hype, it's been aggressively trained to output solutions to them and now overfits their three-object/eight-crossing structure onto prompts that approximate the river cross prompt. Parlor tricks are lurking everywhere, and especially in ChatGPT, because of it's high profile and high stakes.
    - onemoresoop 2y ago
      > Parlor tricks are lurking everywhere, and especially in ChatGPT, because of it's high profile and high stakes. The only way for LLMs to get this right is to be trained on all possible paths of all possible problems, which is not very feasible. That's not really reasoning, you may as well calculate all permutations and add it to a DB and call it a day.
    - Ekaros 2y ago
      I think it is more so just very common data in any dataset used in training. And thus you would expect model to produce it when relevant tokens appear. These models store these common patterns and then repeat slight variations of them as that is natural flow of the tokens. Humans do it similarly with the same problem if they have heard it before. Challenge is that coming up with actually novel reasoning problems is not that easy. So generating one to test models is non-trivial task.
- JustBreath 2y ago
  Yeah, exactly, it isn't reasoning or creating in the way that humans reason, it's calculating that statistically speaking, when a human says what you said here's roughly what often comes next. It's like debating your political other. You can often predict what they will say not because you understand their reasoning, but simply because you've seen multiple times how they react to a given argument in the past.
  - Kim_Bruning 2y ago
    Well it actually CAN execute the reasoning in natural language (see my other post). You just need to set very strict constraints, because as you point out: especially 4o will really prefer to hare off at the first nearest pattern match. (I hate 4o's tuning). I mean, we'd get a similar outcome if we first give a human child this problem, and then subsequently hand them an ice-cream. "Clearly the child isn't reasoning or creating the way that adults reason, it's off eating an ice cream". :-P We can debate whether the machine is smart, or whether natural language requires some level of reasoning as part of the <syntax>. (compare: the type system in rust) . We can even debate whether knowing natural language thus makes the machine smart. %-/ -- edit: Turns out GPT-4 (plain) can solve the question if you warn it upfront, and Claude actually recognizes the trap and then solves it. So I guess we've demonstrated that both AI and humans can jump to conclusions based on incomplete or assumed information.
- Kim_Bruning 2y ago
  https://chatgpt.com/share/715eb79d-f6e4-4487-95ba-9bfad0d1178e https://chatgpt.com/share/715eb79d-f6e4-4487-95ba-9bfad0d117... Geeeeh, 4o really took a lot of work. It's like getting it to win at tic-tac-toe. It CAN be done, and possibly even repeatably, but it's non-trivial.
  - onemoresoop 2y ago
    Well, it does quite poorly to be honest. The wolf doesn't eat the cabbage, it's just a vestige of the full problem where the sheep existed. If pointed out it will fail regardless. Try removing the wolf such that there is only a man and see how well it does. I did with but without 4o and it kept on making unnecessary trips where all could be solved in one step. Try removing the wolf but add the sheep and the cabbage. It seems to me that if falls of the rails rather quickly.
    - Kim_Bruning 2y ago
      No. I don't wanna. Did you see how I had to debug and structure the question until I got it to actually execute the reasoning we actually wanted it to? I tell you, 4o is like a child and ice-cream. You need to eliminate any hint of the ice cream before it'll even deign to do what you ask. %-/ -- Ok fine. I ran out of 4 and 4o responses, so instead I actually got GPT-3 to solve the man and the cabbage problem. But it was tough again, since 3 is bad at meta-evaluation for sure. I keep getting the feeling that I'm doing "maths" with natural language instead of mathematical notation or code. Not helped by the tendency of the AI to want to hare off into the wide blue yonder instead of stop and consider its own answers. I shan't bother you with the transcript unless you really insist. (I had to convince it to check its own answers for logical deficiencies. I can't quite rule out a kluger hans effect[1] , though that's one heck of a kluger hans if so, having written out the entire sequence :-P ) -- Ok so a friend of mine tried this question with Claude 3.5 Sonnet, and Claude was almost offended by the question, and solved it perfectly. [2] I don't have Claude, but I do realize that the web version of GPT-4 is actually a little smarter than the web version of 4o. Now that I have some 4 credits again: https://chatgpt.com/share/e60d5237-c59d-442c-a59c-28a0dc12bd1c https://chatgpt.com/share/e60d5237-c59d-442c-a59c-28a0dc12bd... It got the answer wrong at first because it was over-eager, but when I told it it was wrong, it then actually took the correct approach. -- Score so far: GPT-3.5 : Can't manage the cabbage problem without a lot of help GPT-4 : Needs to be told that it's walking into a trap, but then can solve it pretty much on its own after that. GPT-4o: Fails. (I didn't try walking it through the reasoning, 4o is too tiring for me) Claude-3.5: Insulted by the simplicity of the question, answers it, points out that this is not the man-wolf-goat question, and asks if it should answer that one too? -- Mewpmewp2 figured out a way to avoid pattern matching, this solution actually works on all the GPTs I tried it on. * https://news.ycombinator.com/item?id=40947296 https://news.ycombinator.com/item?id=40947296 * https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d5053265e https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d505326... == [1] Here the part of the effect is : "Keep asking, and stopping because I got the right answer" [2] https://usercontent.irccloud-cdn.com/file/01Al3FZQ/1720800803.JPG https://usercontent.irccloud-cdn.com/file/01Al3FZQ/172080080... <- extra edit: Claude 3.5 answer
- mewpmewp2 2y ago
  If it doesn't pattern match to that specific problem, then it's possible to have it "reason" through a similar setup: This one I used objects A, B and C, where A and C must move together with B. https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d5053265e https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d505326... Do you think that is also from a trained dataset? Because I wonder if it might have some sort of internal router in mind where certain signals will trigger it to pattern match to a very common problem whereas if it's not overfit there it will try to fallback to some form of "reasoning".
  - Kim_Bruning 2y ago
    Oh, duh! * GPT-4o can solve the problem with this design. * GPT-3 in my case actually discovered a lacking constraint, and transports A and C all at once. (I needed to add that B can only carry 1 object at a time) https://chatgpt.com/share/9aa2efa1-ac89-4412-86b3-47fbce4107a3 https://chatgpt.com/share/9aa2efa1-ac89-4412-86b3-47fbce4107...

onemoresoop 2y ago

That's right I did this on chat gpt and went off the rails quickly: a man with a wolf and a cabbage wants to cross a river/ There is a boat but the man can only bring one more item beside himself at a time. What are the steps the man needs to do in order to cross the river? The response was: To solve the puzzle where a man needs to cross a river with a wolf and a cabbage, while ensuring none of them are left together unsupervised (since the wolf will eat the sheep, and the sheep will eat the cabbage), follow these steps: 1.Man takes the sheep across the river (leaving the wolf and the cabbage on the original side). 2.Man returns alone. 3.Man takes the cabbage across the river. 4.Man leaves the cabbage on the other side and takes the sheep back 5.Man takes the wolf across the river. 6.Man returns alone. 7.Man takes the sheep across the river. 8.This way, at no point are the wolf and the sheep left alone together, nor are the sheep and the cabbage left alone together. at which I prompted: I didn't mention a sheep The response was corrected but wrongly: Oops, my mistake! Let’s solve the puzzle correctly with a man, a wolf, and a cabbage: 1.Man takes the wolf across the river (leaving the cabbage on the original side). 2.Man returns alone. 3.Man takes the cabbage across the river. 4.Man leaves the cabbage on the other side and takes the wolf back. 5.Man takes the man across the river. 6.Man returns alone. 7.Man takes the wolf across the river. 8.Now, both the man and the wolf are across the river, with no danger to the cabbage. At step 3 the problem was solved but it kept on going

bckr 2y ago
Oh wow I got similar results with GPT4o. I’m surprised! It even messed up the second round in a similar way.
- swatcoder 2y ago
  Presumably, because river crossing challenges are an important benchmark in satisfying AI hype, it's been aggressively trained to output solutions to them and now overfits their three-object/eight-crossing structure onto prompts that approximate the river cross prompt. Parlor tricks are lurking everywhere, and especially in ChatGPT, because of it's high profile and high stakes.
  - onemoresoop 2y ago
    > Parlor tricks are lurking everywhere, and especially in ChatGPT, because of it's high profile and high stakes. The only way for LLMs to get this right is to be trained on all possible paths of all possible problems, which is not very feasible. That's not really reasoning, you may as well calculate all permutations and add it to a DB and call it a day.
  - Ekaros 2y ago
    I think it is more so just very common data in any dataset used in training. And thus you would expect model to produce it when relevant tokens appear. These models store these common patterns and then repeat slight variations of them as that is natural flow of the tokens. Humans do it similarly with the same problem if they have heard it before. Challenge is that coming up with actually novel reasoning problems is not that easy. So generating one to test models is non-trivial task.
JustBreath 2y ago
Yeah, exactly, it isn't reasoning or creating in the way that humans reason, it's calculating that statistically speaking, when a human says what you said here's roughly what often comes next. It's like debating your political other. You can often predict what they will say not because you understand their reasoning, but simply because you've seen multiple times how they react to a given argument in the past.
- Kim_Bruning 2y ago
  Well it actually CAN execute the reasoning in natural language (see my other post). You just need to set very strict constraints, because as you point out: especially 4o will really prefer to hare off at the first nearest pattern match. (I hate 4o's tuning). I mean, we'd get a similar outcome if we first give a human child this problem, and then subsequently hand them an ice-cream. "Clearly the child isn't reasoning or creating the way that adults reason, it's off eating an ice cream". :-P We can debate whether the machine is smart, or whether natural language requires some level of reasoning as part of the <syntax>. (compare: the type system in rust) . We can even debate whether knowing natural language thus makes the machine smart. %-/ -- edit: Turns out GPT-4 (plain) can solve the question if you warn it upfront, and Claude actually recognizes the trap and then solves it. So I guess we've demonstrated that both AI and humans can jump to conclusions based on incomplete or assumed information.
Kim_Bruning 2y ago
https://chatgpt.com/share/715eb79d-f6e4-4487-95ba-9bfad0d1178e https://chatgpt.com/share/715eb79d-f6e4-4487-95ba-9bfad0d117... Geeeeh, 4o really took a lot of work. It's like getting it to win at tic-tac-toe. It CAN be done, and possibly even repeatably, but it's non-trivial.
- onemoresoop 2y ago
  Well, it does quite poorly to be honest. The wolf doesn't eat the cabbage, it's just a vestige of the full problem where the sheep existed. If pointed out it will fail regardless. Try removing the wolf such that there is only a man and see how well it does. I did with but without 4o and it kept on making unnecessary trips where all could be solved in one step. Try removing the wolf but add the sheep and the cabbage. It seems to me that if falls of the rails rather quickly.
  - Kim_Bruning 2y ago
    No. I don't wanna. Did you see how I had to debug and structure the question until I got it to actually execute the reasoning we actually wanted it to? I tell you, 4o is like a child and ice-cream. You need to eliminate any hint of the ice cream before it'll even deign to do what you ask. %-/ -- Ok fine. I ran out of 4 and 4o responses, so instead I actually got GPT-3 to solve the man and the cabbage problem. But it was tough again, since 3 is bad at meta-evaluation for sure. I keep getting the feeling that I'm doing "maths" with natural language instead of mathematical notation or code. Not helped by the tendency of the AI to want to hare off into the wide blue yonder instead of stop and consider its own answers. I shan't bother you with the transcript unless you really insist. (I had to convince it to check its own answers for logical deficiencies. I can't quite rule out a kluger hans effect[1] , though that's one heck of a kluger hans if so, having written out the entire sequence :-P ) -- Ok so a friend of mine tried this question with Claude 3.5 Sonnet, and Claude was almost offended by the question, and solved it perfectly. [2] I don't have Claude, but I do realize that the web version of GPT-4 is actually a little smarter than the web version of 4o. Now that I have some 4 credits again: https://chatgpt.com/share/e60d5237-c59d-442c-a59c-28a0dc12bd1c https://chatgpt.com/share/e60d5237-c59d-442c-a59c-28a0dc12bd... It got the answer wrong at first because it was over-eager, but when I told it it was wrong, it then actually took the correct approach. -- Score so far: GPT-3.5 : Can't manage the cabbage problem without a lot of help GPT-4 : Needs to be told that it's walking into a trap, but then can solve it pretty much on its own after that. GPT-4o: Fails. (I didn't try walking it through the reasoning, 4o is too tiring for me) Claude-3.5: Insulted by the simplicity of the question, answers it, points out that this is not the man-wolf-goat question, and asks if it should answer that one too? -- Mewpmewp2 figured out a way to avoid pattern matching, this solution actually works on all the GPTs I tried it on. * https://news.ycombinator.com/item?id=40947296 https://news.ycombinator.com/item?id=40947296 * https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d5053265e https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d505326... == [1] Here the part of the effect is : "Keep asking, and stopping because I got the right answer" [2] https://usercontent.irccloud-cdn.com/file/01Al3FZQ/1720800803.JPG https://usercontent.irccloud-cdn.com/file/01Al3FZQ/172080080... <- extra edit: Claude 3.5 answer
mewpmewp2 2y ago
If it doesn't pattern match to that specific problem, then it's possible to have it "reason" through a similar setup: This one I used objects A, B and C, where A and C must move together with B. https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d5053265e https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d505326... Do you think that is also from a trained dataset? Because I wonder if it might have some sort of internal router in mind where certain signals will trigger it to pattern match to a very common problem whereas if it's not overfit there it will try to fallback to some form of "reasoning".
- Kim_Bruning 2y ago
  Oh, duh! * GPT-4o can solve the problem with this design. * GPT-3 in my case actually discovered a lacking constraint, and transports A and C all at once. (I needed to add that B can only carry 1 object at a time) https://chatgpt.com/share/9aa2efa1-ac89-4412-86b3-47fbce4107a3 https://chatgpt.com/share/9aa2efa1-ac89-4412-86b3-47fbce4107...