Another issue is that the classification can change future behavior. For example, if a new system says you're at high risk of a heart attack then you're likely

Another issue is that the classification can change future behavior. For example, if a new system says you're at high risk of a heart attack then you're likely to want to take medication, change your diet, etc. From that point of view the algorithm makes itself invalid, but it is still useful. There are ways to control for these sorts of effects, but I don't see that done with the COMPAS data. In that case researchers got the data years after the fact so it would be hard to separate misclassification due to bad biases vs those from feedback of the system working correctly. I think it is fair to be suspicious of systems like COMPAS, but I don't think the misclassifications identified are a slam dunk against COMPAS. For reference, here is the overview of their process: https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm https://www.propublica.org/article/how-we-analyzed-the-compa...