5 ms·

Well, it does quite poorly to be honest. The wolf doesn't eat the cabbage, it's just a vestige of the full problem where the sheep existed. If pointed out it wi

Well, it does quite poorly to be honest. The wolf doesn't eat the cabbage, it's just a vestige of the full problem where the sheep existed. If pointed out it will fail regardless. Try removing the wolf such that there is only a man and see how well it does. I did with but without 4o and it kept on making unnecessary trips where all could be solved in one step. Try removing the wolf but add the sheep and the cabbage. It seems to me that if falls of the rails rather quickly.

Kim_Bruning 2y ago
No. I don't wanna. Did you see how I had to debug and structure the question until I got it to actually execute the reasoning we actually wanted it to? I tell you, 4o is like a child and ice-cream. You need to eliminate any hint of the ice cream before it'll even deign to do what you ask. %-/ -- Ok fine. I ran out of 4 and 4o responses, so instead I actually got GPT-3 to solve the man and the cabbage problem. But it was tough again, since 3 is bad at meta-evaluation for sure. I keep getting the feeling that I'm doing "maths" with natural language instead of mathematical notation or code. Not helped by the tendency of the AI to want to hare off into the wide blue yonder instead of stop and consider its own answers. I shan't bother you with the transcript unless you really insist. (I had to convince it to check its own answers for logical deficiencies. I can't quite rule out a kluger hans effect[1] , though that's one heck of a kluger hans if so, having written out the entire sequence :-P ) -- Ok so a friend of mine tried this question with Claude 3.5 Sonnet, and Claude was almost offended by the question, and solved it perfectly. [2] I don't have Claude, but I do realize that the web version of GPT-4 is actually a little smarter than the web version of 4o. Now that I have some 4 credits again: https://chatgpt.com/share/e60d5237-c59d-442c-a59c-28a0dc12bd1c https://chatgpt.com/share/e60d5237-c59d-442c-a59c-28a0dc12bd... It got the answer wrong at first because it was over-eager, but when I told it it was wrong, it then actually took the correct approach. -- Score so far: GPT-3.5 : Can't manage the cabbage problem without a lot of help GPT-4 : Needs to be told that it's walking into a trap, but then can solve it pretty much on its own after that. GPT-4o: Fails. (I didn't try walking it through the reasoning, 4o is too tiring for me) Claude-3.5: Insulted by the simplicity of the question, answers it, points out that this is not the man-wolf-goat question, and asks if it should answer that one too? -- Mewpmewp2 figured out a way to avoid pattern matching, this solution actually works on all the GPTs I tried it on. * https://news.ycombinator.com/item?id=40947296 https://news.ycombinator.com/item?id=40947296 * https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d5053265e https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d505326... == [1] Here the part of the effect is : "Keep asking, and stopping because I got the right answer" [2] https://usercontent.irccloud-cdn.com/file/01Al3FZQ/1720800803.JPG https://usercontent.irccloud-cdn.com/file/01Al3FZQ/172080080... <- extra edit: Claude 3.5 answer
- onemoresoop 2y ago
  I appreciate your response but I wouldn't blame you if you didn't want to wrangle and coerce LLMs to do proper reasoning, it becomes boring quickly.
  - Kim_Bruning 2y ago
    I was lazy and did it with GPT-4, since that actually does manage to solve the problem. GPT-4 is -of course- slower. https://chatgpt.com/share/2ac767d9-2a6b-4631-b071-7266e001a14d https://chatgpt.com/share/2ac767d9-2a6b-4631-b071-7266e001a1...
    - onemoresoop 2y ago
      Ok, I guess it appears to work with a bit of nudging on your part. But if you look carefully it still fails and the wolf hallucinates itself back into existence in the last prompt, step 4. My conclusion with LLMs is "Close but no cigar" when it comes to reasoning deviated from the script. Maybe it will get better but something has to change to make this work, more parameters or training will only make it appear to work better but still fail. Even flawed I still find LLMS and GenAI useful for some tasks such as brainstorming though.
- mewpmewp2 2y ago
  What do you think of this one, where the problem is the same however it would not trigger its pattern matching with the classic puzzle? https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d5053265e https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d505326...
  - Kim_Bruning 2y ago
    Ah you posted it twice, including upthread. Neat! I upvoted it there.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]

Kim_Bruning 2y ago

No. I don't wanna. Did you see how I had to debug and structure the question until I got it to actually execute the reasoning we actually wanted it to? I tell you, 4o is like a child and ice-cream. You need to eliminate any hint of the ice cream before it'll even deign to do what you ask. %-/ -- Ok fine. I ran out of 4 and 4o responses, so instead I actually got GPT-3 to solve the man and the cabbage problem. But it was tough again, since 3 is bad at meta-evaluation for sure. I keep getting the feeling that I'm doing "maths" with natural language instead of mathematical notation or code. Not helped by the tendency of the AI to want to hare off into the wide blue yonder instead of stop and consider its own answers. I shan't bother you with the transcript unless you really insist. (I had to convince it to check its own answers for logical deficiencies. I can't quite rule out a kluger hans effect[1] , though that's one heck of a kluger hans if so, having written out the entire sequence :-P ) -- Ok so a friend of mine tried this question with Claude 3.5 Sonnet, and Claude was almost offended by the question, and solved it perfectly. [2] I don't have Claude, but I do realize that the web version of GPT-4 is actually a little smarter than the web version of 4o. Now that I have some 4 credits again: https://chatgpt.com/share/e60d5237-c59d-442c-a59c-28a0dc12bd1c https://chatgpt.com/share/e60d5237-c59d-442c-a59c-28a0dc12bd... It got the answer wrong at first because it was over-eager, but when I told it it was wrong, it then actually took the correct approach. -- Score so far: GPT-3.5 : Can't manage the cabbage problem without a lot of help GPT-4 : Needs to be told that it's walking into a trap, but then can solve it pretty much on its own after that. GPT-4o: Fails. (I didn't try walking it through the reasoning, 4o is too tiring for me) Claude-3.5: Insulted by the simplicity of the question, answers it, points out that this is not the man-wolf-goat question, and asks if it should answer that one too? -- Mewpmewp2 figured out a way to avoid pattern matching, this solution actually works on all the GPTs I tried it on. * https://news.ycombinator.com/item?id=40947296 https://news.ycombinator.com/item?id=40947296 * https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d5053265e https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d505326... == [1] Here the part of the effect is : "Keep asking, and stopping because I got the right answer" [2] https://usercontent.irccloud-cdn.com/file/01Al3FZQ/1720800803.JPG https://usercontent.irccloud-cdn.com/file/01Al3FZQ/172080080... <- extra edit: Claude 3.5 answer

onemoresoop 2y ago
I appreciate your response but I wouldn't blame you if you didn't want to wrangle and coerce LLMs to do proper reasoning, it becomes boring quickly.
- Kim_Bruning 2y ago
  I was lazy and did it with GPT-4, since that actually does manage to solve the problem. GPT-4 is -of course- slower. https://chatgpt.com/share/2ac767d9-2a6b-4631-b071-7266e001a14d https://chatgpt.com/share/2ac767d9-2a6b-4631-b071-7266e001a1...
  - onemoresoop 2y ago
    Ok, I guess it appears to work with a bit of nudging on your part. But if you look carefully it still fails and the wolf hallucinates itself back into existence in the last prompt, step 4. My conclusion with LLMs is "Close but no cigar" when it comes to reasoning deviated from the script. Maybe it will get better but something has to change to make this work, more parameters or training will only make it appear to work better but still fail. Even flawed I still find LLMS and GenAI useful for some tasks such as brainstorming though.
mewpmewp2 2y ago
What do you think of this one, where the problem is the same however it would not trigger its pattern matching with the classic puzzle? https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d5053265e https://chatgpt.com/share/c3f744e4-199a-4c45-a332-c40d505326...
- Kim_Bruning 2y ago
  Ah you posted it twice, including upthread. Neat! I upvoted it there.

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]