Here's a better example: a common riddle they are trained on, but the prompter changed the genders to something that doesn't puzzle people and write models stil

Here's a better example: a common riddle they are trained on, but the prompter changed the genders to something that doesn't puzzle people and write models still assume it is a puzzle, presumably from text statistics, and get it wrong. https://www.reddit.com/r/singularity/comments/1hek6ai/everyone_says_theres_something_different_about/ https://www.reddit.com/r/singularity/comments/1hek6ai/everyo... But Claude nails it. There is a lot more than base LLM training involved since RLHF and other techniques were added, but there are still problems like sycophancy and these kind of crafted misdirections that aren't misdirecrions. Or Claude might have just been trained on someone else doing the same trick to show limitations of LLMs, the internet is vast and I've seen similar experiments it might have seen since it is more recent.