If it’s real, it’s from a small model. Meanwhile, I just tried asking ChatGPT 3.5 and 4 similar questions, and despite ChatGPT having plenty of alignment tunin

If it’s real, it’s from a small model. Meanwhile, I just tried asking ChatGPT 3.5 and 4 similar questions, and despite ChatGPT having plenty of alignment tuning, neither version had any objection to them. There’s no trick; the larger models are just smart enough to differentiate this situation from situations where they actually shouldn’t give out last names. Those models may still get tricked in more complex test cases (either false positives like this one or false negatives like “jailbreaking” prompts), but my guess is that future models in turn will be resistant to those. In other words, LLMs making stupid mistakes about safety is just a special case of LLMs making stupid mistakes in general. That’s essentially their fatal flaw, and it’s an open question how well it can be ameliorated, whether by scaling to even larger models or by making algorithmic improvements. But I don’t think there’s much about it that’s specific to alignment tuning.