I recall reading a story in the book Weapons of Math Destruction [1] about a system used to assess people’s recidivism risk which judges were relying on for sen

I recall reading a story in the book Weapons of Math Destruction [1] about a system used to assess people’s recidivism risk which judges were relying on for sentencing hearings. The problem is that the system showed a clear racial disparity in the risk scores yet at the same time it was more accurate. That means if we try to tune the system to make it less racist, we’ll be making it less accurate. In essence, the system isn’t really racist, it’s a reflection of the racism in society which is leading to these outcomes. Ultimately, the problem is that putting someone in jail increases the likelihood that their relatives will commit crimes. It increases the likelihood that both they and their friends and family will reoffend. It’s a vicious cycle and it doesn’t appear to have any technical solution. [1] https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction