3 ms·

It's tremendously disheartening to see the mainstream media repeating ProPublica's lies. They ran a statistical analysis. Their R-script said that bias was stat

It's tremendously disheartening to see the mainstream media repeating ProPublica's lies. They ran a statistical analysis. Their R-script said that bias was statistically insignificant. So they repeated a bunch of anecdotes in the story and left that part of the analysis out. Reporting null results won't get you cited by the NYT, I guess. https://www.chrisstucchio.com/blog/2016/propublica_is_lying.html https://www.chrisstucchio.com/blog/2016/propublica_is_lying.... The power of the media to push a socially convenient lie is pretty amazing.

dmreedy 10y ago
>> The Opinion Pages | Op-Ed Contributor >> Julia Angwin is a reporter at ProPublica. It looks like this is an Opinion piece from ProPublica. Not much better, I know, but the trail of bias is at least a little less obscured
obastani 10y ago
I might be misunderstanding the article you linked, but it seems very misleading to me (though I agree with the conclusion that the ProPublica article is misleading). They claim to conclude: "The predictor is probably not biased against any particular race - the race_factorAfrican-American:score_factorHigh term is not statistically significant. Or, as ProPublica puts it, it's "almost statistically significant"." I don't think you can conclude that the predictor is probably not biased against any particular race -- you can only use significance to reject the null hypothesis, not prove the null hypothesis (especially since the significance is still somewhat high). Am I misunderstanding the claim in the article?
- yummyfajitas 10y ago
  I wrote the article. It's always tricky to me to figure out how to phrase a statement about a frequentist method (since frequentist methods formally say so little). I may have phrased this incorrectly. So what I'm attempting to say is that they ran a statistical test, were unable to reject the null hypothesis, and then wrote an article phrased as if they did exactly that. But I think you are right that my phrasing is incorrectly, however.
  - GFK_of_xmaspast 10y ago
    > It's always tricky to me to figure out how to phrase a statement about a frequentist method That seems to me to be a fault whose location is not in your stars.
shanacarp 10y ago
the scores are dependent. It's talking about a subset of the data compared to the entire set, rather than a similar subset or random set. The term unlabeled Random_Race is the same as Random_Race:score_factorHigh+score_factorMedium, as opposed to each one individually. So while neither high nor medium alone have statistical significance for African Americans, together they do, a statistical puzzle. Or rather, if you had also copied line 44, less of a puzzle, since `score_factorWhatever` becomes significant in a race specific model. It isn't race alone nor the score alone. It is something else (age? another not modeled thing?) They say so in this link, https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm https://www.propublica.org/article/how-we-analyzed-the-compa... Where they explain a bit more in detail what the chart labels mean than what they are labeled in the R script. Score == dependent variable. So why you ran your blog comments as Score == potential independent variable is a little strange. As for why you want to critique the very last chart (which does appear in 36, and does as a big chart not to work at all) Who cares? Overall, every other time, if we look at a Pearsons fit (line 51) All defendants Low High Survived 2681 1282 0.55 Recidivated 1216 2035 0.45 Total: 7214.00 False positive rate: 32.35 False negative rate: 37.40 Specificity: 0.68 Sensitivity: 0.63 Prevalence: 0.45 PPV: 0.61 NPV: 0.69 LR+: 1.94 LR-: 0.55 Why the hell would we use this for sentencing guidelines? It isn't very specific, nor sensitive. It has a fairly high false positive and false negative rate. When you jail someone, you take away that person's rights and you are probably as much as not going to jail them again. This sentencing guideline basically disguises that, and does it in such a way where it isn't specific nor sensitive, and to boot, 1/3 of the time you'll be wrong in your decision. Furthermore, you'll already know in your gut which 1/3 it is going to be - probably someone young or black. Propublica's writers are totally justified to write the article. They go further than many journalists and show you what they did, and there is no rule in the 4th estate that says they have to do that. If you don't like the fact that statistics as a subject, and policies dependent on statistics, for most people, have to be framed with a story, that's a different issue altogether. But why not come out and just say that.

dmreedy 10y ago

>> The Opinion Pages | Op-Ed Contributor >> Julia Angwin is a reporter at ProPublica. It looks like this is an Opinion piece from ProPublica. Not much better, I know, but the trail of bias is at least a little less obscured

obastani 10y ago

I might be misunderstanding the article you linked, but it seems very misleading to me (though I agree with the conclusion that the ProPublica article is misleading). They claim to conclude: "The predictor is probably not biased against any particular race - the race_factorAfrican-American:score_factorHigh term is not statistically significant. Or, as ProPublica puts it, it's "almost statistically significant"." I don't think you can conclude that the predictor is probably not biased against any particular race -- you can only use significance to reject the null hypothesis, not prove the null hypothesis (especially since the significance is still somewhat high). Am I misunderstanding the claim in the article?

yummyfajitas 10y ago
I wrote the article. It's always tricky to me to figure out how to phrase a statement about a frequentist method (since frequentist methods formally say so little). I may have phrased this incorrectly. So what I'm attempting to say is that they ran a statistical test, were unable to reject the null hypothesis, and then wrote an article phrased as if they did exactly that. But I think you are right that my phrasing is incorrectly, however.
- GFK_of_xmaspast 10y ago
  > It's always tricky to me to figure out how to phrase a statement about a frequentist method That seems to me to be a fault whose location is not in your stars.

shanacarp 10y ago

the scores are dependent. It's talking about a subset of the data compared to the entire set, rather than a similar subset or random set. The term unlabeled Random_Race is the same as Random_Race:score_factorHigh+score_factorMedium, as opposed to each one individually. So while neither high nor medium alone have statistical significance for African Americans, together they do, a statistical puzzle. Or rather, if you had also copied line 44, less of a puzzle, since `score_factorWhatever` becomes significant in a race specific model. It isn't race alone nor the score alone. It is something else (age? another not modeled thing?) They say so in this link, https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm https://www.propublica.org/article/how-we-analyzed-the-compa... Where they explain a bit more in detail what the chart labels mean than what they are labeled in the R script. Score == dependent variable. So why you ran your blog comments as Score == potential independent variable is a little strange. As for why you want to critique the very last chart (which does appear in 36, and does as a big chart not to work at all) Who cares? Overall, every other time, if we look at a Pearsons fit (line 51) All defendants Low High Survived 2681 1282 0.55 Recidivated 1216 2035 0.45 Total: 7214.00 False positive rate: 32.35 False negative rate: 37.40 Specificity: 0.68 Sensitivity: 0.63 Prevalence: 0.45 PPV: 0.61 NPV: 0.69 LR+: 1.94 LR-: 0.55 Why the hell would we use this for sentencing guidelines? It isn't very specific, nor sensitive. It has a fairly high false positive and false negative rate. When you jail someone, you take away that person's rights and you are probably as much as not going to jail them again. This sentencing guideline basically disguises that, and does it in such a way where it isn't specific nor sensitive, and to boot, 1/3 of the time you'll be wrong in your decision. Furthermore, you'll already know in your gut which 1/3 it is going to be - probably someone young or black. Propublica's writers are totally justified to write the article. They go further than many journalists and show you what they did, and there is no rule in the 4th estate that says they have to do that. If you don't like the fact that statistics as a subject, and policies dependent on statistics, for most people, have to be framed with a story, that's a different issue altogether. But why not come out and just say that.