5 ms·

No. Take the example of the algorithm designed to tell whether people charged with a crime would commit another crime within two years: > Comparing black and w

No. Take the example of the algorithm designed to tell whether people charged with a crime would commit another crime within two years: > Comparing black and white defendants, the journalists found that a disproportionate number of black defendants were ‘false positives’: they were classified by COMPAS as high risk but subsequently not charged with another crime. It goes on to note that there are other potential measures of bias, such as false negatives, and that external factors such as different arrest rates may bias the data. But it's clear that this issue is more complex than "researchers are trying to get the algorithms to be politically correct".

kgwgk 8y ago
> ‘false positives’: [...] classified by COMPAS as high risk but subsequently not charged with another crime It doesn't make sense to say that someone classified as "high risk" which is not charged again was misclassfied. Unless "high risk" is supposed to mean "100% probability of being charged with another crime in the next two years". We wouldn't say that patients with "high risk" of a heart attack were misclassified because they didn't have a heart attack. If we applied a similar reasoning, we would conclude that cardiology guidelines are biased against obese people.
- jakelazaroff 8y ago
  That is not correct; risk is quantifiable. If an algorithm defines "high risk" as an 80% chance of having a heart attack, and only 60% of individuals classified as such end up having one, then we can say that 25% of whom the algorithm classifies as "high risk" are misclassified.
  - kgwgk 8y ago
    That’s different: you can’t say that a particular person was misclassified, you are making a statement about the calibration of the algorithm. And I can agree with that, but the argument in the article is not about miscalibration.
    - jakelazaroff 8y ago
      Yes it is? The article is about bias in algorithms, which skews results. In the case of the ProPublica example, the claim is that a particular subset of the results of an algorithm is more skewed than the full set. Nowhere is it argued that individual samples can be explained probabilistically.
      - kgwgk 8y ago
        I'm sorry, I don't follow you. > In the case of the ProPublica example, the claim is that a particular subset of the results of an algorithm is more skewed than the full set. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm https://www.propublica.org/article/how-we-analyzed-the-compa... "black defendants who scored higher did recidivate slightly more often than white defendants (63 percent vs. 59 percent)." If we define "high risk" as 60% chance of recidivism, the algorithm is getting it essentially right and according to your previous comment there are no misclassifications (if we assume that the small variation is due to statistical variability). If we want to be hyperprecise there is a small bias: white defendants are getting classified as high risk more than they should and black defendants not as often as they should. Their claim is not that 60% is too low. They never define what the desired precision is, but their use of the terms "misclassify" and "false positive" makes sense only if one expects "high risk" to predict recidivism with 100% precision. If their argument was about miscalibration and they considered that getting 60% is too low, the threshold to define "high risk" could be raised. The score is fine-grained, their analysis is based on the score deciles and they consider the binary partition Low=1,2,3,4 and High=5,6,7,8,9,10. > Nowhere is it argued that individual samples can be explained probabilistically. I don't understand what you mean.
        jakelazaroff 8y ago
        You said > you can’t say that a particular person was misclassified, you are making a statement about the calibration of the algorithm But no one is arguing that a particular person was misclassified; both the article and the study are about results in aggregate being skewed in some way. > They never define what the desired precision is, but their use of the terms "misclassify" and "false positive" makes sense only if one expects "high risk" to predict recidivism with 100% precision. They're not trying to find an absolute number for accuracy, they're comparing data for white and black defendants. Excerpt from the ProPublica article below: > Our analysis found that black defendants who did not recidivate over a two-year period were nearly twice as likely to be misclassified as higher risk compared to their white counterparts (45 percent vs. 23 percent). In this case, the equation for each race is (defendants who were labeled high risk but didn't recidivate / defendants who didn't recidivate). The accuracy in absolute terms isn't in question here; the claim they are making is that the algorithm mistakenly classifies black defendants as high risk at twice the rate it does white ones. Note that testing for true positives is different from testing for false positives, and both can be true simultaneously. If of 15 defendants, the algorithm predicts ten will recidivate, of whom six actually do, that gives 60% accuracy with ~45% of those who didn't misclassified as high risk. (I'm also not sure if you're making the semantic argument that "it just said recidivism was likely, not guaranteed, so even if they don't it's not technically wrong." That's a weird assertion to make; if every high-risk classification never recidivated but every low-risk one did, by that definition the algorithm still wouldn't be "wrong", but it would obviously be broken. Classifying a defendant as high risk is a prediction that they will recidivate, and we can judge the algorithm by whether it turned out to be true.) I'll also note that other replies to my original comment have been good rebuttals to the ProPublica study specifically, but what they're attempting to do — draw conclusions about false positives based on the "high risk" classifier — makes perfect sense.
  - Jwarder 8y ago
    Another issue is that the classification can change future behavior. For example, if a new system says you're at high risk of a heart attack then you're likely to want to take medication, change your diet, etc. From that point of view the algorithm makes itself invalid, but it is still useful. There are ways to control for these sorts of effects, but I don't see that done with the COMPAS data. In that case researchers got the data years after the fact so it would be hard to separate misclassification due to bad biases vs those from feedback of the system working correctly. I think it is fair to be suspicious of systems like COMPAS, but I don't think the misclassifications identified are a slam dunk against COMPAS. For reference, here is the overview of their process: https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm https://www.propublica.org/article/how-we-analyzed-the-compa...
bobcostas55 8y ago
The COMPAS analysis they refer to is exactly one of those cases where there is no bias whatsoever and they're trying to force the algorithm to conform to PC instead of reality. What the ProPublica analysis showed was that a black person and a white person with the same risk score have the same probability of recidivism. There is no racial bias. Black people have higher rates of recidivism, which means that they will also have a higher rate of false positives. This is mathematically unavoidable. I recommend creating a toy example in excel and playing around with some numbers to get an intuition for it. Also check out this analysis: https://www.chrisstucchio.com/blog/2016/propublica_is_lying.html https://www.chrisstucchio.com/blog/2016/propublica_is_lying....
- denzil_correa 8y ago
  If an algorithm matches or models an inherently unfair real world situation, it doesn't make the algorithm fair.

kgwgk 8y ago

> ‘false positives’: [...] classified by COMPAS as high risk but subsequently not charged with another crime It doesn't make sense to say that someone classified as "high risk" which is not charged again was misclassfied. Unless "high risk" is supposed to mean "100% probability of being charged with another crime in the next two years". We wouldn't say that patients with "high risk" of a heart attack were misclassified because they didn't have a heart attack. If we applied a similar reasoning, we would conclude that cardiology guidelines are biased against obese people.

jakelazaroff 8y ago
That is not correct; risk is quantifiable. If an algorithm defines "high risk" as an 80% chance of having a heart attack, and only 60% of individuals classified as such end up having one, then we can say that 25% of whom the algorithm classifies as "high risk" are misclassified.
- kgwgk 8y ago
  That’s different: you can’t say that a particular person was misclassified, you are making a statement about the calibration of the algorithm. And I can agree with that, but the argument in the article is not about miscalibration.
  - jakelazaroff 8y ago
    Yes it is? The article is about bias in algorithms, which skews results. In the case of the ProPublica example, the claim is that a particular subset of the results of an algorithm is more skewed than the full set. Nowhere is it argued that individual samples can be explained probabilistically.
    - kgwgk 8y ago
      I'm sorry, I don't follow you. > In the case of the ProPublica example, the claim is that a particular subset of the results of an algorithm is more skewed than the full set. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm https://www.propublica.org/article/how-we-analyzed-the-compa... "black defendants who scored higher did recidivate slightly more often than white defendants (63 percent vs. 59 percent)." If we define "high risk" as 60% chance of recidivism, the algorithm is getting it essentially right and according to your previous comment there are no misclassifications (if we assume that the small variation is due to statistical variability). If we want to be hyperprecise there is a small bias: white defendants are getting classified as high risk more than they should and black defendants not as often as they should. Their claim is not that 60% is too low. They never define what the desired precision is, but their use of the terms "misclassify" and "false positive" makes sense only if one expects "high risk" to predict recidivism with 100% precision. If their argument was about miscalibration and they considered that getting 60% is too low, the threshold to define "high risk" could be raised. The score is fine-grained, their analysis is based on the score deciles and they consider the binary partition Low=1,2,3,4 and High=5,6,7,8,9,10. > Nowhere is it argued that individual samples can be explained probabilistically. I don't understand what you mean.
      - jakelazaroff 8y ago
        You said > you can’t say that a particular person was misclassified, you are making a statement about the calibration of the algorithm But no one is arguing that a particular person was misclassified; both the article and the study are about results in aggregate being skewed in some way. > They never define what the desired precision is, but their use of the terms "misclassify" and "false positive" makes sense only if one expects "high risk" to predict recidivism with 100% precision. They're not trying to find an absolute number for accuracy, they're comparing data for white and black defendants. Excerpt from the ProPublica article below: > Our analysis found that black defendants who did not recidivate over a two-year period were nearly twice as likely to be misclassified as higher risk compared to their white counterparts (45 percent vs. 23 percent). In this case, the equation for each race is (defendants who were labeled high risk but didn't recidivate / defendants who didn't recidivate). The accuracy in absolute terms isn't in question here; the claim they are making is that the algorithm mistakenly classifies black defendants as high risk at twice the rate it does white ones. Note that testing for true positives is different from testing for false positives, and both can be true simultaneously. If of 15 defendants, the algorithm predicts ten will recidivate, of whom six actually do, that gives 60% accuracy with ~45% of those who didn't misclassified as high risk. (I'm also not sure if you're making the semantic argument that "it just said recidivism was likely, not guaranteed, so even if they don't it's not technically wrong." That's a weird assertion to make; if every high-risk classification never recidivated but every low-risk one did, by that definition the algorithm still wouldn't be "wrong", but it would obviously be broken. Classifying a defendant as high risk is a prediction that they will recidivate, and we can judge the algorithm by whether it turned out to be true.) I'll also note that other replies to my original comment have been good rebuttals to the ProPublica study specifically, but what they're attempting to do — draw conclusions about false positives based on the "high risk" classifier — makes perfect sense.
- Jwarder 8y ago
  Another issue is that the classification can change future behavior. For example, if a new system says you're at high risk of a heart attack then you're likely to want to take medication, change your diet, etc. From that point of view the algorithm makes itself invalid, but it is still useful. There are ways to control for these sorts of effects, but I don't see that done with the COMPAS data. In that case researchers got the data years after the fact so it would be hard to separate misclassification due to bad biases vs those from feedback of the system working correctly. I think it is fair to be suspicious of systems like COMPAS, but I don't think the misclassifications identified are a slam dunk against COMPAS. For reference, here is the overview of their process: https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm https://www.propublica.org/article/how-we-analyzed-the-compa...

bobcostas55 8y ago

The COMPAS analysis they refer to is exactly one of those cases where there is no bias whatsoever and they're trying to force the algorithm to conform to PC instead of reality. What the ProPublica analysis showed was that a black person and a white person with the same risk score have the same probability of recidivism. There is no racial bias. Black people have higher rates of recidivism, which means that they will also have a higher rate of false positives. This is mathematically unavoidable. I recommend creating a toy example in excel and playing around with some numbers to get an intuition for it. Also check out this analysis: https://www.chrisstucchio.com/blog/2016/propublica_is_lying.html https://www.chrisstucchio.com/blog/2016/propublica_is_lying....

denzil_correa 8y ago
If an algorithm matches or models an inherently unfair real world situation, it doesn't make the algorithm fair.