This has been a discussion in AI safety for a long time, that huge amounts of LLM reasoning could be for hidden goals outside of the actions we want. Safety tra

This has been a discussion in AI safety for a long time, that huge amounts of LLM reasoning could be for hidden goals outside of the actions we want. Safety training can have the perverse effect of commonly amplifying 'forbidden' answers in hidden/encoded paths as the model doesn't get rid of these behaviors but finds methods detecting training to avoid outputting bad tokens when it's being watched. Now, don't think of this of this like a conscious behavior like a kid trying not to get in trouble, but an emergent behavior of trying to repress particular output when that output is well connected to a massive amount of other tokens.