I also think it's kind of a silly example- smart people can be functionally illiterate, after all. It doesn't tell you that much. My favorite LLM stumper is as

I also think it's kind of a silly example- smart people can be functionally illiterate, after all. It doesn't tell you that much. My favorite LLM stumper is asking them to solve the farmer/wolf/chicken/grain river-crossing puzzle but with only a chicken. They usually either insist on pointless extra trips or hallucinate a wolf or grain. Liquid somehow does both and also loses track of what's where. > The farmer can take the chicken across the river first. Then, he can go back to the original side and bring the boat back. He can then take the chicken back to the original side and leave it there. Next, he can take a bag of grain across the river. He can then go back to the original side and bring the chicken across the river. Finally, he can go back to the original side one last time and bring the chicken across the river. One generation ended like this: > the farmer and the chicken are both on the other side of the river, and the chicken is safe from the grain.