5 ms·

That is not correct; risk is quantifiable. If an algorithm defines "high risk" as an 80% chance of having a heart attack, and only 60% of individuals classified

That is not correct; risk is quantifiable. If an algorithm defines "high risk" as an 80% chance of having a heart attack, and only 60% of individuals classified as such end up having one, then we can say that 25% of whom the algorithm classifies as "high risk" are misclassified.

kgwgk 8y ago
That’s different: you can’t say that a particular person was misclassified, you are making a statement about the calibration of the algorithm. And I can agree with that, but the argument in the article is not about miscalibration.
- jakelazaroff 8y ago
  Yes it is? The article is about bias in algorithms, which skews results. In the case of the ProPublica example, the claim is that a particular subset of the results of an algorithm is more skewed than the full set. Nowhere is it argued that individual samples can be explained probabilistically.
  - kgwgk 8y ago
    I'm sorry, I don't follow you. > In the case of the ProPublica example, the claim is that a particular subset of the results of an algorithm is more skewed than the full set. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm https://www.propublica.org/article/how-we-analyzed-the-compa... "black defendants who scored higher did recidivate slightly more often than white defendants (63 percent vs. 59 percent)." If we define "high risk" as 60% chance of recidivism, the algorithm is getting it essentially right and according to your previous comment there are no misclassifications (if we assume that the small variation is due to statistical variability). If we want to be hyperprecise there is a small bias: white defendants are getting classified as high risk more than they should and black defendants not as often as they should. Their claim is not that 60% is too low. They never define what the desired precision is, but their use of the terms "misclassify" and "false positive" makes sense only if one expects "high risk" to predict recidivism with 100% precision. If their argument was about miscalibration and they considered that getting 60% is too low, the threshold to define "high risk" could be raised. The score is fine-grained, their analysis is based on the score deciles and they consider the binary partition Low=1,2,3,4 and High=5,6,7,8,9,10. > Nowhere is it argued that individual samples can be explained probabilistically. I don't understand what you mean.
    - jakelazaroff 8y ago
      You said > you can’t say that a particular person was misclassified, you are making a statement about the calibration of the algorithm But no one is arguing that a particular person was misclassified; both the article and the study are about results in aggregate being skewed in some way. > They never define what the desired precision is, but their use of the terms "misclassify" and "false positive" makes sense only if one expects "high risk" to predict recidivism with 100% precision. They're not trying to find an absolute number for accuracy, they're comparing data for white and black defendants. Excerpt from the ProPublica article below: > Our analysis found that black defendants who did not recidivate over a two-year period were nearly twice as likely to be misclassified as higher risk compared to their white counterparts (45 percent vs. 23 percent). In this case, the equation for each race is (defendants who were labeled high risk but didn't recidivate / defendants who didn't recidivate). The accuracy in absolute terms isn't in question here; the claim they are making is that the algorithm mistakenly classifies black defendants as high risk at twice the rate it does white ones. Note that testing for true positives is different from testing for false positives, and both can be true simultaneously. If of 15 defendants, the algorithm predicts ten will recidivate, of whom six actually do, that gives 60% accuracy with ~45% of those who didn't misclassified as high risk. (I'm also not sure if you're making the semantic argument that "it just said recidivism was likely, not guaranteed, so even if they don't it's not technically wrong." That's a weird assertion to make; if every high-risk classification never recidivated but every low-risk one did, by that definition the algorithm still wouldn't be "wrong", but it would obviously be broken. Classifying a defendant as high risk is a prediction that they will recidivate, and we can judge the algorithm by whether it turned out to be true.) I'll also note that other replies to my original comment have been good rebuttals to the ProPublica study specifically, but what they're attempting to do — draw conclusions about false positives based on the "high risk" classifier — makes perfect sense.
      - bobcostas55 8y ago
        Consider the following situation: There are two groups, A and B. For both groups, when a person is labeled high risk they have a 75% risk of reoffending (ie no bias). 10% of Group A people are labeled high risk. Which means that (1 - 75%) * 10% = 2.5% are misclassified. 20% of Group B people are labeled high risk. Which means that (1 - 75%) * 20% = 5% are misclassified. As you can see, this is simply an artifact of the higher reoffending rate of Group B. If 1) the two groups have different reoffending rates, and 2) the algorithm for labeling people high risk is unbiased, it necessarily follows that the two groups will have different misclassification rates. The entire ProPublica piece (and your arguments) is based on misunderstanding this. Part of the issue is that extremely misleading table titled "Prediction Fails Differently for Black Defendants", where "Labeled Higher Risk, But Didn’t Re-Offend" actually refers to P(labeled higher risk | didn't re-offend).
Jwarder 8y ago
Another issue is that the classification can change future behavior. For example, if a new system says you're at high risk of a heart attack then you're likely to want to take medication, change your diet, etc. From that point of view the algorithm makes itself invalid, but it is still useful. There are ways to control for these sorts of effects, but I don't see that done with the COMPAS data. In that case researchers got the data years after the fact so it would be hard to separate misclassification due to bad biases vs those from feedback of the system working correctly. I think it is fair to be suspicious of systems like COMPAS, but I don't think the misclassifications identified are a slam dunk against COMPAS. For reference, here is the overview of their process: https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm https://www.propublica.org/article/how-we-analyzed-the-compa...

kgwgk 8y ago

That’s different: you can’t say that a particular person was misclassified, you are making a statement about the calibration of the algorithm. And I can agree with that, but the argument in the article is not about miscalibration.

jakelazaroff 8y ago
Yes it is? The article is about bias in algorithms, which skews results. In the case of the ProPublica example, the claim is that a particular subset of the results of an algorithm is more skewed than the full set. Nowhere is it argued that individual samples can be explained probabilistically.
- kgwgk 8y ago
  I'm sorry, I don't follow you. > In the case of the ProPublica example, the claim is that a particular subset of the results of an algorithm is more skewed than the full set. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm https://www.propublica.org/article/how-we-analyzed-the-compa... "black defendants who scored higher did recidivate slightly more often than white defendants (63 percent vs. 59 percent)." If we define "high risk" as 60% chance of recidivism, the algorithm is getting it essentially right and according to your previous comment there are no misclassifications (if we assume that the small variation is due to statistical variability). If we want to be hyperprecise there is a small bias: white defendants are getting classified as high risk more than they should and black defendants not as often as they should. Their claim is not that 60% is too low. They never define what the desired precision is, but their use of the terms "misclassify" and "false positive" makes sense only if one expects "high risk" to predict recidivism with 100% precision. If their argument was about miscalibration and they considered that getting 60% is too low, the threshold to define "high risk" could be raised. The score is fine-grained, their analysis is based on the score deciles and they consider the binary partition Low=1,2,3,4 and High=5,6,7,8,9,10. > Nowhere is it argued that individual samples can be explained probabilistically. I don't understand what you mean.
  - jakelazaroff 8y ago
    You said > you can’t say that a particular person was misclassified, you are making a statement about the calibration of the algorithm But no one is arguing that a particular person was misclassified; both the article and the study are about results in aggregate being skewed in some way. > They never define what the desired precision is, but their use of the terms "misclassify" and "false positive" makes sense only if one expects "high risk" to predict recidivism with 100% precision. They're not trying to find an absolute number for accuracy, they're comparing data for white and black defendants. Excerpt from the ProPublica article below: > Our analysis found that black defendants who did not recidivate over a two-year period were nearly twice as likely to be misclassified as higher risk compared to their white counterparts (45 percent vs. 23 percent). In this case, the equation for each race is (defendants who were labeled high risk but didn't recidivate / defendants who didn't recidivate). The accuracy in absolute terms isn't in question here; the claim they are making is that the algorithm mistakenly classifies black defendants as high risk at twice the rate it does white ones. Note that testing for true positives is different from testing for false positives, and both can be true simultaneously. If of 15 defendants, the algorithm predicts ten will recidivate, of whom six actually do, that gives 60% accuracy with ~45% of those who didn't misclassified as high risk. (I'm also not sure if you're making the semantic argument that "it just said recidivism was likely, not guaranteed, so even if they don't it's not technically wrong." That's a weird assertion to make; if every high-risk classification never recidivated but every low-risk one did, by that definition the algorithm still wouldn't be "wrong", but it would obviously be broken. Classifying a defendant as high risk is a prediction that they will recidivate, and we can judge the algorithm by whether it turned out to be true.) I'll also note that other replies to my original comment have been good rebuttals to the ProPublica study specifically, but what they're attempting to do — draw conclusions about false positives based on the "high risk" classifier — makes perfect sense.
    - bobcostas55 8y ago
      Consider the following situation: There are two groups, A and B. For both groups, when a person is labeled high risk they have a 75% risk of reoffending (ie no bias). 10% of Group A people are labeled high risk. Which means that (1 - 75%) * 10% = 2.5% are misclassified. 20% of Group B people are labeled high risk. Which means that (1 - 75%) * 20% = 5% are misclassified. As you can see, this is simply an artifact of the higher reoffending rate of Group B. If 1) the two groups have different reoffending rates, and 2) the algorithm for labeling people high risk is unbiased, it necessarily follows that the two groups will have different misclassification rates. The entire ProPublica piece (and your arguments) is based on misunderstanding this. Part of the issue is that extremely misleading table titled "Prediction Fails Differently for Black Defendants", where "Labeled Higher Risk, But Didn’t Re-Offend" actually refers to P(labeled higher risk | didn't re-offend).

Jwarder 8y ago

Another issue is that the classification can change future behavior. For example, if a new system says you're at high risk of a heart attack then you're likely to want to take medication, change your diet, etc. From that point of view the algorithm makes itself invalid, but it is still useful. There are ways to control for these sorts of effects, but I don't see that done with the COMPAS data. In that case researchers got the data years after the fact so it would be hard to separate misclassification due to bad biases vs those from feedback of the system working correctly. I think it is fair to be suspicious of systems like COMPAS, but I don't think the misclassifications identified are a slam dunk against COMPAS. For reference, here is the overview of their process: https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm https://www.propublica.org/article/how-we-analyzed-the-compa...