6 ms·

The pro-publica article is self-debunking. The R script written by the authors is unable to find statistically significant evidence of bias. https://www.chriss

The pro-publica article is self-debunking. The R script written by the authors is unable to find statistically significant evidence of bias. https://www.chrisstucchio.com/blog/2016/propublica_is_lying.html https://www.chrisstucchio.com/blog/2016/propublica_is_lying.... That's probably why the article is nothing but a bunch of anecdotes.

deleted 10y ago
[deleted]
deleted 10y ago
[deleted]
maxerickson 10y ago
danso linked a different article than the one you discuss. The article they link discusses the analysis in greater detail, rather than telling stories. (which I'm not trying to weigh in on the quality of the main article, just pointing out that they did publish a discussion of their statistics. I can understand the editorial decision to break it into a separate article.)
- yummyfajitas 10y ago
  It's the same underlying analysis. Their github repo contains the same script I linked to: https://github.com/propublica/compas-analysis https://github.com/propublica/compas-analysis https://github.com/propublica/compas-analysis/blob/master/Compas%20Analysis.ipynb https://github.com/propublica/compas-analysis/blob/master/Co... It even contains the same weasel words "almost statistically significant" for one analysis.
  - maxerickson 10y ago
    Yes, the statistical discussion is linked from the main article in a couple different places.
danso 10y ago
Full disclosure: I used to work at PP so I have an obvious affinity towards my former colleagues. That said, I respect endeavors to reproduce and critique data work. I haven't worked through the numbers myself but I don't see any immediate reason to distrust the numbers or interpretation you've stated. But the fact that you keep referring to COMPAS as "machine learning" when neither COMPAS (in the various papers I've read on their site) nor the ProPublica article describes it as such makes me wonder if you're missing a few key things about the actual reality. AFAIK, COMPAS clearly involves statistical heuristics, but its not clear the data pipeline used to create those heuristics, nevermind how it would feed into a machine learning system. And that's kind of the problem here, with so much of this algorithm being proprietary, there's no way to keep it accountable, other than the imperfect ad-hoc way that ProPublica devised. You have a highly emotional reaction to their process -- calling them flat-out liars -- despite the fact that they were the ones who not only procured the COMPAS scores via FOIA, but the also wrote their own scraper to automate the collection of criminal records from the Broward court system. They then reconciled this with the databases from Florida corrections and jail records from Broward's sheriff's office, and the databases required manual reconciliation. They then packaged and posted all of this data for public use, along with their workbook, on to Github. And this is the work, in your opinion, of a group of deliberately dishonest journalists who are looking to sell clickbait. And you can't stand that they went out and talked to people (anecdotes, as if data weren't the plural of anecdotes...and as if COMPAS itself isn't built on anecdotes because it requires practitioners to go out and interview inmates in person, for fucks sake) Again, I haven't run through the numbers myself, so I can't agree or disagree whether ProPublica has conclusively argued that race is a statistically significant factor when COMPAS fucks up the prediction. Or if the "bias" that they speak of is more focused on the "hazard ratios" differentiation between racial groups, and perhaps that's not as sexy a bias as its headline warrants. But I'll be honest. Your conflation of COMPAS with "machine learning", and your simplistic view of the law and the justice system -- "we've taken a flawed and racist human system and replaced it with a much better machine learning system"...um, there's an actual Constitutional process to adjust things in our judicial system that shouldn't just be overridden because statistics -- makes me think that you come into this fight with at least as raging a bias as that of the ProPublica journalists.
- 10y ago
AnthonyMouse 10y ago
Even regardless of significance, the "analysis" is just correlation dressed up as some kind of shocking revelation. > Our analysis found that black defendants who did not recidivate over a two-year period were nearly twice as likely to be misclassified as higher risk compared to their white counterparts Which is exactly what you would expect to happen if the algorithm predicts that black defendants are more likely to recidivate. Being in the group predicted to recidivate more will make you more likely to be predicted to recidivate. It would be quite unexpected for that not to still hold for the people who didn't actually recidivate. And if being black does correlate with recidivism, what is the algorithm supposed to do?
- danso 10y ago
  > And if being black does correlate with recidivism, what is the algorithm supposed to do? Algorithms don't think for themselves, and AFAIK, the COMPAS algorithm has not been touted as one of those self-healing types. We wouldn't know either way because the algorithm is kept secret even though it is broadly applied in the public justice system. But let me get this straight, you'd be OK with an algorithm just using the heuristic of "well, he's black, so give him a -0.5, because that's what blacks tend to do" to avoid the complexity of identifying possible confounding variables that are correlated with being black? That's actually illegal in many contexts (judicial system, housing, employment).
  - yummyfajitas 10y ago
    There is an entire statistical discipline devoted to evaluating algorithms without understanding their internals, or at least with minimal knowledge thereof. It's called "machine learning" (a term you criticized me for using). According to the PP article, race is not an input to the algorithm so it can't give -0.5 to black people solely for being black.
    - deleted 10y ago
      [deleted]
  - AnthonyMouse 10y ago
TheCoelacanth 10y ago
Statistical significance is irrelevant. The law is not a peer reviewed journal. The legal standard for showing that someone has been discriminated against is that it is more likely than not that they have been discriminated against.
marketforlemmas 10y ago
So I've read the PP piece and your article, and I think your criticism is way off-base. The most technical part of your argument relies on a p-value being .057 vs .05, which is not a good one. No one seriously believes that .05 is magical number that determines true from false; things that are close to it but not quite below .05 are not automatically false. They go on to give supporting analysis in form of the false positive and false negative rates by race, which is pretty compelling evidence. You claim to not believe that because you cant find it in the notebook but its literally right underneath the Cox model section. I was intrigued by this article and went a step further to plot the ROC curves and the evidence is solid. It's messy, but you can see it here https://github.com/stoddardg/compas-analysis/blob/master/my_analysis.ipynb https://github.com/stoddardg/compas-analysis/blob/master/my_... in cell 78. Its quite clear that the algorithm is choosing a different point of optimization on the ROC for white people (a more lenient one) than for black people. A white defendant with a risk score of 5 is as likely to commit a crime as a black defendant is with a score of 7. That's an obvious case where you could simply relabel and be more fair but their algorithm chooses not to. I also hate when people abuse bad statistics and reasoning to sell page views.
- yummyfajitas 10y ago
  I have plenty of criticisms of hypothesis testing and p-values. Nevertheless, if you choose to run that type of analysis, do it right - this means sticking with your analysis and not using weasel words like "almost statistically significant" when it doesn't come out the way you want. Incidentally, the real p-value is 11.075% since they ran two hypothesis tests and didn't adjust for multiple comparisons. Your analysis might be right - if so, that's interesting. I'll take a closer look and write a followup piece if true - among other things glancing at your ROC curve suggests they are pretty close, and perform better for whites in some regions and better for blacks in others. But it's 7:30AM (pre-coffee) and I haven't looked closely yet. But since PP did not do any of this, my criticism of them holds - they ran an NHST, got the wrong result, and then spouted a bunch of anecdotes instead of admitting that their analysis went against what they wanted to find.
  - danso 10y ago
areyousure 10y ago
Your analysis is incorrect (though perhaps not in the way you'd think!): ProPublica, in line [36] of their analysis, fit a Cox proportional hazards model on pairs (score, race) with three possible scores (low, medium, high) and six possible races (Caucasian, African-American, Asian, Hispanic, Native American, and Other). Thus there are 18 total groups; each person belongs to exactly one. This model tries to estimate the relative risk of recidivism for each of these groups. The "base rate" cancels out when you fit this model, so these risks are all expressed relative to a baseline; this baseline is chosen to be (low, Caucasian). There are 17 degrees of freedom here. It doesn't matter what basis you choose (as there is no regularization), so all that's really happening is you're counting recidivism in 18 buckets. The basis ProPublica's formula chooses is 2 = 3-1 terms for score, 5 = 6-1 terms for race, and 10 = (3-1)x(6-1) terms for the "interaction"; this reflects their choice of coding using two categorical variables. But you may as well just have 17=18-1 terms each of which expresses the relative risk between each group and the baseline; this would be the equivalent of coding using one categorical variable for the entire pair. The advantage of such an expression is it'd be harder to make the mistake both you and ProPublica make: the "interaction" terms are not the droids you're looking for! In the "Background" section of your "How to lie without statistics" article, you write: "the size of the second coefficient (relative to the first) is directly proportional to the degree to which the old predictor was biased". The first coefficient is the coefficient in front of the "old_predictor" variable (here, "score_factorHigh/Medium"). The second coefficient is the coefficient in front of the "is_black" variable. I'm honestly surprised, given how clearly you seem to write here, that you then focus on the wrong coefficient later. The relevant coefficient is the one on "race_factorAfrican-American"! That coefficient is 0.27879 and very significant. In other words, the algorithm is "biased in favor of blacks". To be ever more precise, the numbers you're actually looking for are "race_factorAfrican-American + race_factorAfrican-American:score_factorHigh" and "race_factorAfrican-American + race_factorAfrican-American:score_factorMedium"; as one would expect, the interaction terms are correction terms that aren't the dominant effect. Your error is a bit more understandable given quite how much ProPublica messes up line [37]. They forget the race terms entirely!! The correct calculation for "Black High Hazard" is exp(0.27879 + -0.18976 + 1.28350) = 3.95 and "Black Medium Hazard" is exp(0.27879 + 0.84286 - 0.17261) = 2.58; note that I included the race terms! Their sentence should read "High risk white defendants are 3.61 more likely than low risk white defendants [to recidivate in a given time period], while High risk black defendants are 3.95 more likely than low [risk white defendants...]". (They wrote 2.99 for the italicized term. For medium risk, they give the numbers as 2.32 white versus 1.95 black instead of the correct 2.58 black.) To specifically isolate where your analysis is incorrect: two of your three conclusions are incorrect. Conclusion #1 is correct. Conclusion #2 is not: the analysis directly suggests that it is biased in favor of blacks and the interaction term is not the dominant effect. Conclusion #3 is also incorrect: that term is how much more likely (low risk, African Americans)s are to recidivate than (low risk, Caucasian)s. If blacks had a much higher recidivism rate but it was fully captured by the score, this term would be zero!

deleted 10y ago

[deleted]

deleted 10y ago

[deleted]

maxerickson 10y ago

danso linked a different article than the one you discuss. The article they link discusses the analysis in greater detail, rather than telling stories. (which I'm not trying to weigh in on the quality of the main article, just pointing out that they did publish a discussion of their statistics. I can understand the editorial decision to break it into a separate article.)

yummyfajitas 10y ago
It's the same underlying analysis. Their github repo contains the same script I linked to: https://github.com/propublica/compas-analysis https://github.com/propublica/compas-analysis https://github.com/propublica/compas-analysis/blob/master/Compas%20Analysis.ipynb https://github.com/propublica/compas-analysis/blob/master/Co... It even contains the same weasel words "almost statistically significant" for one analysis.
- maxerickson 10y ago
  Yes, the statistical discussion is linked from the main article in a couple different places.

danso 10y ago

Full disclosure: I used to work at PP so I have an obvious affinity towards my former colleagues. That said, I respect endeavors to reproduce and critique data work. I haven't worked through the numbers myself but I don't see any immediate reason to distrust the numbers or interpretation you've stated. But the fact that you keep referring to COMPAS as "machine learning" when neither COMPAS (in the various papers I've read on their site) nor the ProPublica article describes it as such makes me wonder if you're missing a few key things about the actual reality. AFAIK, COMPAS clearly involves statistical heuristics, but its not clear the data pipeline used to create those heuristics, nevermind how it would feed into a machine learning system. And that's kind of the problem here, with so much of this algorithm being proprietary, there's no way to keep it accountable, other than the imperfect ad-hoc way that ProPublica devised. You have a highly emotional reaction to their process -- calling them flat-out liars -- despite the fact that they were the ones who not only procured the COMPAS scores via FOIA, but the also wrote their own scraper to automate the collection of criminal records from the Broward court system. They then reconciled this with the databases from Florida corrections and jail records from Broward's sheriff's office, and the databases required manual reconciliation. They then packaged and posted all of this data for public use, along with their workbook, on to Github. And this is the work, in your opinion, of a group of deliberately dishonest journalists who are looking to sell clickbait. And you can't stand that they went out and talked to people (anecdotes, as if data weren't the plural of anecdotes...and as if COMPAS itself isn't built on anecdotes because it requires practitioners to go out and interview inmates in person, for fucks sake) Again, I haven't run through the numbers myself, so I can't agree or disagree whether ProPublica has conclusively argued that race is a statistically significant factor when COMPAS fucks up the prediction. Or if the "bias" that they speak of is more focused on the "hazard ratios" differentiation between racial groups, and perhaps that's not as sexy a bias as its headline warrants. But I'll be honest. Your conflation of COMPAS with "machine learning", and your simplistic view of the law and the justice system -- "we've taken a flawed and racist human system and replaced it with a much better machine learning system"...um, there's an actual Constitutional process to adjust things in our judicial system that shouldn't just be overridden because statistics -- makes me think that you come into this fight with at least as raging a bias as that of the ProPublica journalists.

10y ago

AnthonyMouse 10y ago

Even regardless of significance, the "analysis" is just correlation dressed up as some kind of shocking revelation. > Our analysis found that black defendants who did not recidivate over a two-year period were nearly twice as likely to be misclassified as higher risk compared to their white counterparts Which is exactly what you would expect to happen if the algorithm predicts that black defendants are more likely to recidivate. Being in the group predicted to recidivate more will make you more likely to be predicted to recidivate. It would be quite unexpected for that not to still hold for the people who didn't actually recidivate. And if being black does correlate with recidivism, what is the algorithm supposed to do?

danso 10y ago
> And if being black does correlate with recidivism, what is the algorithm supposed to do? Algorithms don't think for themselves, and AFAIK, the COMPAS algorithm has not been touted as one of those self-healing types. We wouldn't know either way because the algorithm is kept secret even though it is broadly applied in the public justice system. But let me get this straight, you'd be OK with an algorithm just using the heuristic of "well, he's black, so give him a -0.5, because that's what blacks tend to do" to avoid the complexity of identifying possible confounding variables that are correlated with being black? That's actually illegal in many contexts (judicial system, housing, employment).
- yummyfajitas 10y ago
  There is an entire statistical discipline devoted to evaluating algorithms without understanding their internals, or at least with minimal knowledge thereof. It's called "machine learning" (a term you criticized me for using). According to the PP article, race is not an input to the algorithm so it can't give -0.5 to black people solely for being black.
  - deleted 10y ago
    [deleted]
- AnthonyMouse 10y ago

TheCoelacanth 10y ago

Statistical significance is irrelevant. The law is not a peer reviewed journal. The legal standard for showing that someone has been discriminated against is that it is more likely than not that they have been discriminated against.

marketforlemmas 10y ago

So I've read the PP piece and your article, and I think your criticism is way off-base. The most technical part of your argument relies on a p-value being .057 vs .05, which is not a good one. No one seriously believes that .05 is magical number that determines true from false; things that are close to it but not quite below .05 are not automatically false. They go on to give supporting analysis in form of the false positive and false negative rates by race, which is pretty compelling evidence. You claim to not believe that because you cant find it in the notebook but its literally right underneath the Cox model section. I was intrigued by this article and went a step further to plot the ROC curves and the evidence is solid. It's messy, but you can see it here https://github.com/stoddardg/compas-analysis/blob/master/my_analysis.ipynb https://github.com/stoddardg/compas-analysis/blob/master/my_... in cell 78. Its quite clear that the algorithm is choosing a different point of optimization on the ROC for white people (a more lenient one) than for black people. A white defendant with a risk score of 5 is as likely to commit a crime as a black defendant is with a score of 7. That's an obvious case where you could simply relabel and be more fair but their algorithm chooses not to. I also hate when people abuse bad statistics and reasoning to sell page views.

yummyfajitas 10y ago
I have plenty of criticisms of hypothesis testing and p-values. Nevertheless, if you choose to run that type of analysis, do it right - this means sticking with your analysis and not using weasel words like "almost statistically significant" when it doesn't come out the way you want. Incidentally, the real p-value is 11.075% since they ran two hypothesis tests and didn't adjust for multiple comparisons. Your analysis might be right - if so, that's interesting. I'll take a closer look and write a followup piece if true - among other things glancing at your ROC curve suggests they are pretty close, and perform better for whites in some regions and better for blacks in others. But it's 7:30AM (pre-coffee) and I haven't looked closely yet. But since PP did not do any of this, my criticism of them holds - they ran an NHST, got the wrong result, and then spouted a bunch of anecdotes instead of admitting that their analysis went against what they wanted to find.
- danso 10y ago

areyousure 10y ago

Your analysis is incorrect (though perhaps not in the way you'd think!): ProPublica, in line [36] of their analysis, fit a Cox proportional hazards model on pairs (score, race) with three possible scores (low, medium, high) and six possible races (Caucasian, African-American, Asian, Hispanic, Native American, and Other). Thus there are 18 total groups; each person belongs to exactly one. This model tries to estimate the relative risk of recidivism for each of these groups. The "base rate" cancels out when you fit this model, so these risks are all expressed relative to a baseline; this baseline is chosen to be (low, Caucasian). There are 17 degrees of freedom here. It doesn't matter what basis you choose (as there is no regularization), so all that's really happening is you're counting recidivism in 18 buckets. The basis ProPublica's formula chooses is 2 = 3-1 terms for score, 5 = 6-1 terms for race, and 10 = (3-1)x(6-1) terms for the "interaction"; this reflects their choice of coding using two categorical variables. But you may as well just have 17=18-1 terms each of which expresses the relative risk between each group and the baseline; this would be the equivalent of coding using one categorical variable for the entire pair. The advantage of such an expression is it'd be harder to make the mistake both you and ProPublica make: the "interaction" terms are not the droids you're looking for! In the "Background" section of your "How to lie without statistics" article, you write: "the size of the second coefficient (relative to the first) is directly proportional to the degree to which the old predictor was biased". The first coefficient is the coefficient in front of the "old_predictor" variable (here, "score_factorHigh/Medium"). The second coefficient is the coefficient in front of the "is_black" variable. I'm honestly surprised, given how clearly you seem to write here, that you then focus on the wrong coefficient later. The relevant coefficient is the one on "race_factorAfrican-American"! That coefficient is 0.27879 and very significant. In other words, the algorithm is "biased in favor of blacks". To be ever more precise, the numbers you're actually looking for are "race_factorAfrican-American + race_factorAfrican-American:score_factorHigh" and "race_factorAfrican-American + race_factorAfrican-American:score_factorMedium"; as one would expect, the interaction terms are correction terms that aren't the dominant effect. Your error is a bit more understandable given quite how much ProPublica messes up line [37]. They forget the race terms entirely!! The correct calculation for "Black High Hazard" is exp(0.27879 + -0.18976 + 1.28350) = 3.95 and "Black Medium Hazard" is exp(0.27879 + 0.84286 - 0.17261) = 2.58; note that I included the race terms! Their sentence should read "High risk white defendants are 3.61 more likely than low risk white defendants [to recidivate in a given time period], while High risk black defendants are 3.95 more likely than low [risk white defendants...]". (They wrote 2.99 for the italicized term. For medium risk, they give the numbers as 2.32 white versus 1.95 black instead of the correct 2.58 black.) To specifically isolate where your analysis is incorrect: two of your three conclusions are incorrect. Conclusion #1 is correct. Conclusion #2 is not: the analysis directly suggests that it is biased in favor of blacks and the interaction term is not the dominant effect. Conclusion #3 is also incorrect: that term is how much more likely (low risk, African Americans)s are to recidivate than (low risk, Caucasian)s. If blacks had a much higher recidivism rate but it was fully captured by the score, this term would be zero!