It's not just biased data, though, it's an objective function optimising a biased metric. We've picked a metric, recidivism rate, that is believed to be inhere

It's not just biased data, though, it's an objective function optimising a biased metric. We've picked a metric, recidivism rate, that is believed to be inherently biased because cops arrest a lot of protected minorities. The model has correctly predicted that cops will arrest a lot of protected minorities. The general public has then turned around and shot the messenger rather than hold cops accountable for all that arresting they're doing.