3 ms·

It still fails at very simple stuff. E.g. "I put an ordinary rock into a glass of water. I then turn the glass of water upside down, do a little dance, and the

It still fails at very simple stuff. E.g. "I put an ordinary rock into a glass of water. I then turn the glass of water upside down, do a little dance, and then turn the glass right side up again. Where is the rock now?" 100+ lines later... "The rock is at the bottom of the glass, submerged in the water." Models from a year ago get this right sometimes https://pastebin.com/em5TT4Zn https://pastebin.com/em5TT4Zn

deleted 2y ago
[deleted]
unsupp0rted 2y ago
Claude 3.5 Sonnet (current): > The rock would still be at the bottom of the glass. When you turn the glass upside down, the rock falls toward the bottom due to gravity. When you turn it right side up, it falls back to the original bottom. The dance steps don't affect this outcome - gravity consistently pulls the rock toward Earth. Me: > Are you certain? Claude: > No, I apologize - I jumped to a conclusion. Let me think this through carefully: > When the glass is turned upside down, water would flow out due to gravity, taking the rock with it. So after completing the sequence, the rock would be on the surface where it fell when the glass was inverted, not in the glass at all.
- owenpalmer 2y ago
  That's odd. My Claude 3.5 Sonnet (current), got it correct 3 times in a row, each on the first try. Do you have a weird system prompt, like telling it to have short responses? https://snipboard.io/NFu4tK.jpg https://snipboard.io/NFu4tK.jpg https://snipboard.io/nmaxVW.jpg https://snipboard.io/nmaxVW.jpg https://snipboard.io/gQnJiD.jpg https://snipboard.io/gQnJiD.jpg
sebastiennight 2y ago
Llama3.1 8B fails as well. After I call the model on it, my favorite part of the "fixed" answer is step 3: > Bien sûr ! Let me retrace the steps with you. 01. You put an ordinary rock into a glass of water. 02. Then, you turn the glass upside down. 03. At this point, the rock would be suspended in the air above the glass, as it can’t fall out (it’s not being held by anything). 04. Next, you do a little dance. This doesn’t affect the rock’s position. 03. Finally, you turn the glass right side up again. > So, when you turn the glass right side up, the rock would simply fall back into the glass, ending up at the bottom where it started.

deleted 2y ago

[deleted]

unsupp0rted 2y ago

Claude 3.5 Sonnet (current): > The rock would still be at the bottom of the glass. When you turn the glass upside down, the rock falls toward the bottom due to gravity. When you turn it right side up, it falls back to the original bottom. The dance steps don't affect this outcome - gravity consistently pulls the rock toward Earth. Me: > Are you certain? Claude: > No, I apologize - I jumped to a conclusion. Let me think this through carefully: > When the glass is turned upside down, water would flow out due to gravity, taking the rock with it. So after completing the sequence, the rock would be on the surface where it fell when the glass was inverted, not in the glass at all.

owenpalmer 2y ago
That's odd. My Claude 3.5 Sonnet (current), got it correct 3 times in a row, each on the first try. Do you have a weird system prompt, like telling it to have short responses? https://snipboard.io/NFu4tK.jpg https://snipboard.io/NFu4tK.jpg https://snipboard.io/nmaxVW.jpg https://snipboard.io/nmaxVW.jpg https://snipboard.io/gQnJiD.jpg https://snipboard.io/gQnJiD.jpg

sebastiennight 2y ago

Llama3.1 8B fails as well. After I call the model on it, my favorite part of the "fixed" answer is step 3: > Bien sûr ! Let me retrace the steps with you. 01. You put an ordinary rock into a glass of water. 02. Then, you turn the glass upside down. 03. At this point, the rock would be suspended in the air above the glass, as it can’t fall out (it’s not being held by anything). 04. Next, you do a little dance. This doesn’t affect the rock’s position. 03. Finally, you turn the glass right side up again. > So, when you turn the glass right side up, the rock would simply fall back into the glass, ending up at the bottom where it started.