> This assumes the data sets of the criminal punishment system are not inherently biased, which is of course a false assumption. It doesn't even require that a

> This assumes the data sets of the criminal punishment system are not inherently biased, which is of course a false assumption. It doesn't even require that assumption to go awry. Assume there exists a stereotype, say "people with freckles are more likely to be criminals". It doesn't actually matter if the stereotype has any basis in reality, just that it is widely believed. People will, on the margin, be less likely to hire freckled people. This reduces the legitimate employment opportunities available to a freckled individual, which tends to make the illegitimate opportunities more attractive by comparison. So the stereotype becomes a self-fulfilling prophecy: by assuming that freckled people are more likely to commit crimes, society actually causes freckled people to be more likely to commit crimes. An expert system will notice this and begin using "has freckles" as a weighting factor in predicting recidivism. It's important to note that the expert system is not wrong. Freckles are in fact, at this point, statistically correlated to recidivism. But the expert system can't know why the correlation exists. All it can do is tighten the vicious feedback loop, noticing a statistical correlation that strengthens the existing stereotype, which exacerbates the real world impacts, which increases the statistical correlation, which strengthens the stereotype, and so on.