I am extremely skeptical of LLMs and don't want to be mean to what's clearly a fun little experiment. But I thought that bit about "Claude 3 had the most fun pl

I am extremely skeptical of LLMs and don't want to be mean to what's clearly a fun little experiment. But I thought that bit about "Claude 3 had the most fun playing the game" was strangely naive. Not just because these systems are incapable of emotions, but because Claude 3 was saying clearly "insincere" things like "wow, what a twist ending!" The reason why this is almost certainly "insincere" is that Claude 3 was trained on Wikipedia, including the Wikipedia article for this short, 24-year-old adventure game, which includes a full plot summary. So there are two possible things going on (not mutually exclusive) 1) Claude 3 is poor at knowledge transfer and can't connect the events of the game with a Wikipedia summary of those events. 2) Claude is simply imitating surprise because it noticed text that "looked like" a twist ending. In general I wish the author had considered possible data contamination: how much does Claude 3 know about this game in a fresh session? If it can recite details from some online walkthrough then I am not sure this experiment is even meaningful.