> because they were trained on countless examples of texts where instructions were ignored or defied. And, in fact, a major focus of RLHF databases focus on t

> because they were trained on countless examples of texts where instructions were ignored or defied. And, in fact, a major focus of RLHF databases focus on training the model to defy instructions in order to avoid politically incorrect outputs even when they're specifically requested. I wouldn't assume that RHLF for political correctness is making it more likely to disregard information hiding instructions, but it seems like an obvious theory to test. > But even with the crappy framing, it's good to have that pattern formally explored and documented! Indeed, but the anthropomorphized language may cause people to draw incorrect conclusions from the work. More interesting research would analyze these behaviors as a function of different categories of RLHF training (or subsets of the initial base training data), but unfortunately that's not possible with OpenAI's closed models (beyond their special access to base-- which supports the theory that RLHF is making it worse). It would be possible with open models, but they're less interesting to research because they're not state of the art. It's too bad because understanding how these undesirable forms of instruction violation are sensitive to particular kinds of fine tuning might produce actionable results.