4 ms·

Those are really sweeping conclusions, considering the experiment is just a single iteration of a single prompt! FWIW, Claude Opus got this for me on the first

Those are really sweeping conclusions, considering the experiment is just a single iteration of a single prompt! FWIW, Claude Opus got this for me on the first try: "In the Gilligan's Island episode "Seer Gilligan" (season 3, episode 8), Gilligan gains the ability to read minds after being hit on the head with a coconut. At first, the castaways are excited about Gilligan's new power and try to use it to their advantage. However, his mind-reading abilities soon cause chaos and misunderstandings among the group. In the end, Gilligan gets hit on the head again and loses his mind-reading powers, much to everyone's relief." (the season number and episode number are wrong, but the name is right, suggesting that this is just lack of sufficient memorization rather than some deep statement about reasoning. The episode only has ~4,000 Google hits, so it's not super widely known.) More rigorously, Claude Opus gets 60% on GPQA, which very smart humans only get 34% on, even if you give them half an hour per question and full Internet access. It seems implausible that you could do that without some sort of reasoning: https://arxiv.org/pdf/2311.12022.pdf https://arxiv.org/pdf/2311.12022.pdf

HarHarVeryFunny 2y ago
The free version of Claude (Haiku?) failed, and also failed to give the episode names for all seasons, although it hallucinated some type of answer.
WheatMillington 2y ago
I just tried it with a handful of different episodes of different shows, on both chatGPT and Bard, and almost every answer was wrong, with some episodes made up entirely.
layer8 2y ago
The point is that the limitations of LLMs regarding self-reflection lead to them misrepresenting their knowledge, as in the “(season 3, episode 8)” in your test. It doesn’t matter which parts any particular LLM gets right because you can’t rely on them telling you if they are unable to get some part right.
jhbadger 2y ago
>Gilligan gains the ability to read minds after being hit on the head with a coconut. Partial credit. The title is right, but the explanation is a hallucination. It isn't a coconut but special sunflower seeds that give Gilligan (and eventually the other characters) mind-reading powers.
latexr 2y ago
> Claude Opus got this for me on the first try > (…) > the season number and episode number are wrong Then it didn’t really “get it”. That’s indistinguishable from a lucky guess. Because you know what else it got wrong? The description of the episode. There’s no coconut falling on head, it was eating sunflower seeds. And everyone gets to do it and that causes the conflict, it’s not about using Gilligan’s power to their advantage. https://gilligan.fandom.com/wiki/Seer_Gilligan https://gilligan.fandom.com/wiki/Seer_Gilligan Claude bullshitted a whole premise based on outdated generic cartoon ideas and you didn’t even notice.
- _wire_ 2y ago
  Truly LOL!

HarHarVeryFunny 2y ago

The free version of Claude (Haiku?) failed, and also failed to give the episode names for all seasons, although it hallucinated some type of answer.

WheatMillington 2y ago

I just tried it with a handful of different episodes of different shows, on both chatGPT and Bard, and almost every answer was wrong, with some episodes made up entirely.

layer8 2y ago

The point is that the limitations of LLMs regarding self-reflection lead to them misrepresenting their knowledge, as in the “(season 3, episode 8)” in your test. It doesn’t matter which parts any particular LLM gets right because you can’t rely on them telling you if they are unable to get some part right.

jhbadger 2y ago

>Gilligan gains the ability to read minds after being hit on the head with a coconut. Partial credit. The title is right, but the explanation is a hallucination. It isn't a coconut but special sunflower seeds that give Gilligan (and eventually the other characters) mind-reading powers.

latexr 2y ago

> Claude Opus got this for me on the first try > (…) > the season number and episode number are wrong Then it didn’t really “get it”. That’s indistinguishable from a lucky guess. Because you know what else it got wrong? The description of the episode. There’s no coconut falling on head, it was eating sunflower seeds. And everyone gets to do it and that causes the conflict, it’s not about using Gilligan’s power to their advantage. https://gilligan.fandom.com/wiki/Seer_Gilligan https://gilligan.fandom.com/wiki/Seer_Gilligan Claude bullshitted a whole premise based on outdated generic cartoon ideas and you didn’t even notice.

_wire_ 2y ago
Truly LOL!