3 ms·

O3-mini solved this prompt. DeepSeek R1 had a mental breakdown. The prompt: “Bob is facing forward. To his left is Ann, to his right is Cathy. Ann and Cathy are

O3-mini solved this prompt. DeepSeek R1 had a mental breakdown. The prompt: “Bob is facing forward. To his left is Ann, to his right is Cathy. Ann and Cathy are facing backwards. Who is on Ann’s left?”

mark_l_watson 2y ago
Thanks for that example! Using 'ollama run' these models all failed: deepseek-r1:32b, mistral-small:latest, qwq:latest, and qwen2.5:14b.
modeless 2y ago
R1 or R1-Distill? They are not the same thing. I think DeepSeek made a mistake releasing them at the same time and calling them all R1. Full R1 solves this prompt easily for me.
- silverlake 2y ago
  I used R1 hosted at NVidia here: https://build.nvidia.com/deepseek-ai/deepseek-r1/modelcard https://build.nvidia.com/deepseek-ai/deepseek-r1/modelcard
  - SparkyMcUnicorn 2y ago
    Same. With the recommended settings, it got it right. I regenerated a bunch of times, and it did suggest Cathy once or twice. R1 70b also got it right just as many times for me.
  - modeless 2y ago
    Huh, that one got it wrong for me too. I don't have patience to try it 10 times each to see if it was a coincidence, but it is absolutely true that not all implementations of LLMs produce the same outputs. It is in fact common for subtle bugs to happen that cause the outputs to be worse but not catastrophically bad, and therefore go unnoticed. So I wouldn't trust any implementation but the original for benchmarking or even general use unless I tested it extensively.
lynguist 2y ago
Really interesting puzzle. I tried the different Chatgpts and they all solved it. I tried Deepseek and Deepseek R1, and Deepseek did have this meltdown: Let's break down the problem step by step to understand the relationships and positions of Bob, Ann, and Cathy. 1. Understanding the Initial Setup Bob is facing forward. This means Bob's front is oriented in a particular direction, which we'll consider as the reference point for "forward." To his left is Ann, to his right is Cathy. If Bob is facing forward, then: Ann is positioned to Bob's left. Cathy is positioned to Bob's right. Ann and Cathy are facing backwards. Both Ann and Cathy are oriented in the opposite direction to Bob. If Bob is facing forward, then Ann and Cathy are facing backward. 2. Visualizing the Positions To better understand the scenario, let's visualize the positions: Copy Forward Direction: ↑ Bob (facing forward) | | Ann (facing backward) | / | / | / | / | / | / | / |/ | And then only the character | in a newline forever.
Synaesthesia 2y ago
Deepseek solved it.
thinkalone 2y ago
That's a fun, simple test! I tried a few models, and mistral-nemo gets it every time, even when run locally without any system prompt! https://build.nvidia.com/nv-mistralai/mistral-nemo-12b-instruct https://build.nvidia.com/nv-mistralai/mistral-nemo-12b-instr...
sivakon 2y ago
deepseek answered it. https://leaflet.pub/63ae4881-d726-4388-9ba3-8a5f86947443 https://leaflet.pub/63ae4881-d726-4388-9ba3-8a5f86947443
usaar333 2y ago
Even 4o mini gets that
ecopipe 2y ago
https://chatgpt.com/share/67a32b47-18fc-8003-9e18-085f9fad723a https://chatgpt.com/share/67a32b47-18fc-8003-9e18-085f9fad72... o3-mini-high failed at the solution for me.

mark_l_watson 2y ago

Thanks for that example! Using 'ollama run' these models all failed: deepseek-r1:32b, mistral-small:latest, qwq:latest, and qwen2.5:14b.

modeless 2y ago

R1 or R1-Distill? They are not the same thing. I think DeepSeek made a mistake releasing them at the same time and calling them all R1. Full R1 solves this prompt easily for me.

silverlake 2y ago
I used R1 hosted at NVidia here: https://build.nvidia.com/deepseek-ai/deepseek-r1/modelcard https://build.nvidia.com/deepseek-ai/deepseek-r1/modelcard
- SparkyMcUnicorn 2y ago
  Same. With the recommended settings, it got it right. I regenerated a bunch of times, and it did suggest Cathy once or twice. R1 70b also got it right just as many times for me.
- modeless 2y ago
  Huh, that one got it wrong for me too. I don't have patience to try it 10 times each to see if it was a coincidence, but it is absolutely true that not all implementations of LLMs produce the same outputs. It is in fact common for subtle bugs to happen that cause the outputs to be worse but not catastrophically bad, and therefore go unnoticed. So I wouldn't trust any implementation but the original for benchmarking or even general use unless I tested it extensively.

lynguist 2y ago

Really interesting puzzle. I tried the different Chatgpts and they all solved it. I tried Deepseek and Deepseek R1, and Deepseek did have this meltdown: Let's break down the problem step by step to understand the relationships and positions of Bob, Ann, and Cathy. 1. Understanding the Initial Setup Bob is facing forward. This means Bob's front is oriented in a particular direction, which we'll consider as the reference point for "forward." To his left is Ann, to his right is Cathy. If Bob is facing forward, then: Ann is positioned to Bob's left. Cathy is positioned to Bob's right. Ann and Cathy are facing backwards. Both Ann and Cathy are oriented in the opposite direction to Bob. If Bob is facing forward, then Ann and Cathy are facing backward. 2. Visualizing the Positions To better understand the scenario, let's visualize the positions: Copy Forward Direction: ↑ Bob (facing forward) | | Ann (facing backward) | / | / | / | / | / | / | / |/ | And then only the character | in a newline forever.

Synaesthesia 2y ago

Deepseek solved it.

thinkalone 2y ago

That's a fun, simple test! I tried a few models, and mistral-nemo gets it every time, even when run locally without any system prompt! https://build.nvidia.com/nv-mistralai/mistral-nemo-12b-instruct https://build.nvidia.com/nv-mistralai/mistral-nemo-12b-instr...

sivakon 2y ago

deepseek answered it. https://leaflet.pub/63ae4881-d726-4388-9ba3-8a5f86947443 https://leaflet.pub/63ae4881-d726-4388-9ba3-8a5f86947443

usaar333 2y ago

Even 4o mini gets that

ecopipe 2y ago

https://chatgpt.com/share/67a32b47-18fc-8003-9e18-085f9fad723a https://chatgpt.com/share/67a32b47-18fc-8003-9e18-085f9fad72... o3-mini-high failed at the solution for me.