That article explains that the quirk was introduced during supervised fine-tuning. > We unknowingly gave particularly high rewards for metaphors with creatures

That article explains that the quirk was introduced during supervised fine-tuning. > We unknowingly gave particularly high rewards for metaphors with creatures It was the human feedback that caused the bias, not a change in training data.