The metrics are bad. And since the metrics are believed to be good, they further entrench winners. It's like a ML algo. You take all people who have been put i

The metrics are bad. And since the metrics are believed to be good, they further entrench winners. It's like a ML algo. You take all people who have been put in prison, how long, and their faces. You also take a big subset of non-prison people - same thing. (You know where this is going). You build a classifier that, for a given face, estimates yes/no for prison and how long if yes. Lo and behold, you show it darker face images, and prison estimation ratchets up and the estimated time served up up up. Is the ML racist? Technically, yes. But it's observing racist underlying societal things. Yeah, it's ugly. No, I don't have good answers, other than directly addressing when we catch stuff. But We (royal) don't see it as a problem. The status quo changes slowly.