I am not surprised to see grok failing on this. Practically everything gets trained on extracts from other LLMs, I assume this is true for Grok too. The issue

I am not surprised to see grok failing on this. Practically everything gets trained on extracts from other LLMs, I assume this is true for Grok too. The issue is that even if you manually cull 'biased' (for whatever definition you like) output, the training data can still hide bias in high dimensional noise. So for example, you train some LLM to hate men. Then you generate from it training data for another LLM but carefully cull any mention of men or women. But other word choices like, say, "this" vs "that" in a sentence may bias the training of the "hate men" weights. I think this is particularly effective because a lot of the LLM's tone change in fine tuning as picking the character that the LLM is play acting as... and so you can pack a lot of bias into a fairly small change. This also explains how some of the bias got in there in the first place-- it's not unreasonably charitable to assume that they didn't explicitly train in the misandrist behavior, but they probably did train in other behavior, perfectly reasonable behavior, that is correlated online with misandry. The same behavior happens with adversarial examples for image classifiers, where they're robust to truncation and generalize against different models. And I've seen people give plenty of examples from grok where it produces the same negative nanny refusals that open ai models produce, -- but just on more obscure areas where it presumably wasn't spot-fixed.