3 ms·

> When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence

> When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence for a young, Black first time offender than for an older white repeat felon. Younger people are more likely to re-offend than older people. Remove race from the situation entirely, and this is still the expected result. There's zero reason to think this algorithm was based with respect to race.

TheCoelacanth 6y ago
If you read deeper than a one line description you'll see: 1. Even after accounting for criminal history, recidivism, age and gender, black defendants were still scored as much more likely to re-offend. 2. It incorrectly predicted that black defendants would re-offend much more frequently than white defendants. 3. It incorrectly predicted that white defendants would not re-offend much more frequently than black defendants. Considering those three things and that they have refused to give any details about how the algorithm works, I see zero reason to give them the benefit of the doubt. Fire them until they can demonstrate that the algorithm is not in fact biased like it appears to be. The score also takes into account answers to questions like whether the defendant's parents were separated or whether their parents were ever arrested. Those things are completely out of the defendant's control and are highly correlated with race. Even including those things in their score is damning.
- sir_bearington 6y ago
  Phrasing it in the way you do is misleading. The defendants labeled as high risk and low risk were just as likely to re-offend. To put this in simpler numbers * Out of 100 white people, 5 were labeled high risk. * Out of 100 black people, 20 were labeled high risk. * Out of the 5 white people labeled high risk, 4 re-offended. * Out of 20 black people labeled high risk, 16 re-offended. In either case, someone labeled high risk had the same likelihood to re-offend: 80%. "It incorrectly predicted that black defendants would re-offend more frequently than white defendants." This is technically correct, but not because the algorithm was bad a predicting rates if re-offending. It's because there was higher rates of re-offending. The likelihood of re-offending among someone labeled high risk is the same. This kind of objection seems like a blanket rejection of any system that produces an inequitable outcome. But the reality is that rates of re-offending is not equal. Even a perfectly accurate prediction of re-offense is going to predict higher rates of re-offending among men. Because men re-offend at higher rates. This isn't sexism. That doesn't mean we shouldn't recognize the disparate impact of incarceration on underprivileged people. But simply concluding bias due to inequitable outcomes is simplistic.
  - TheCoelacanth 6y ago
    When you are doing crystal ball voodoo based on stuff that has no connection to the defendant's choices and no direct connection to criminality like whether or not their parents were separated and whether their parents were ever arrested, then you're damn right there should be a blanket rejection of any inequitable outcome. I'm not even convinced that you should be allowed to make a decision based on stuff like that even if it is somehow equitable.
    - Rule35 6y ago
      > no direct connection to criminality like whether or not their parents were separated and whether their parents were ever arrested Again, not right at all. These things heavily correlate with crime. A broken home is the primary indicator of someone's future success, even over a 'better' but broken home. I'm male, and not a rapist, but having a penis heavily correlates with rape. I suggest you do not pick me, or any male, to watch your children. Sure it's rude to the innocent, but oh well, a little rudeness vs potential harm. > I'm not even convinced that you should be allowed to make a decision based on stuff like that even if it is somehow equitable. Then nobody will ever follow the law. If it's illegal for me to reject a potentially bad babysitter for something I know about them that could risk my child's safety I'll happily lie and say I didn't like their haircut. You'd get further if you tried to identify these people and treat them better - grants to move out of bad neighborhoods, to get educated, to get pardons for unrelated crimes, to get counseling for abuse, etc - than with this "wrong unless it's 100% identical, equity-of-outcome" thing.
      - TheCoelacanth 6y ago
        We're talking about an algorithm being used as part of the court system to determine whether or not someone rots in jail, not about how you choose your babysitter. It needs to be held to a higher standard and punish people based on their actions, not based on what their parents did.
        Rule35 6y ago
- Rule35 6y ago
  > Fire them until they can demonstrate that the algorithm is not in fact biased like it appears to be. What do the non-black box findings from the area show? If blacks are a poor demographic in the area it might be true. (Crime tracks poverty, not race.) But yeah, certainly don't pay anyone for, or use, a black-box algorithm. > The score also takes into account answers to questions like whether the defendant's parents were separated or whether their parents were ever arrested. Those things are completely out of the defendant's control and are highly correlated with race. Even including those things in their score is damning. Nope. That's perfectly fair to look at. For instance, a broken family is another predictor. It's not fair, but being a victim increases your chance to offend. Similarly, the number one predictor of child sex crimes is to have suffered them yourself. If you have a child and are picking a babysitter, skip the one who was molested. fwiw, those things don't correlate with race, they correlate with poverty which correlates with race. Broken families are more likely to be poor and abusive.

TheCoelacanth 6y ago

If you read deeper than a one line description you'll see: 1. Even after accounting for criminal history, recidivism, age and gender, black defendants were still scored as much more likely to re-offend. 2. It incorrectly predicted that black defendants would re-offend much more frequently than white defendants. 3. It incorrectly predicted that white defendants would not re-offend much more frequently than black defendants. Considering those three things and that they have refused to give any details about how the algorithm works, I see zero reason to give them the benefit of the doubt. Fire them until they can demonstrate that the algorithm is not in fact biased like it appears to be. The score also takes into account answers to questions like whether the defendant's parents were separated or whether their parents were ever arrested. Those things are completely out of the defendant's control and are highly correlated with race. Even including those things in their score is damning.

sir_bearington 6y ago
Phrasing it in the way you do is misleading. The defendants labeled as high risk and low risk were just as likely to re-offend. To put this in simpler numbers * Out of 100 white people, 5 were labeled high risk. * Out of 100 black people, 20 were labeled high risk. * Out of the 5 white people labeled high risk, 4 re-offended. * Out of 20 black people labeled high risk, 16 re-offended. In either case, someone labeled high risk had the same likelihood to re-offend: 80%. "It incorrectly predicted that black defendants would re-offend more frequently than white defendants." This is technically correct, but not because the algorithm was bad a predicting rates if re-offending. It's because there was higher rates of re-offending. The likelihood of re-offending among someone labeled high risk is the same. This kind of objection seems like a blanket rejection of any system that produces an inequitable outcome. But the reality is that rates of re-offending is not equal. Even a perfectly accurate prediction of re-offense is going to predict higher rates of re-offending among men. Because men re-offend at higher rates. This isn't sexism. That doesn't mean we shouldn't recognize the disparate impact of incarceration on underprivileged people. But simply concluding bias due to inequitable outcomes is simplistic.
- TheCoelacanth 6y ago
  When you are doing crystal ball voodoo based on stuff that has no connection to the defendant's choices and no direct connection to criminality like whether or not their parents were separated and whether their parents were ever arrested, then you're damn right there should be a blanket rejection of any inequitable outcome. I'm not even convinced that you should be allowed to make a decision based on stuff like that even if it is somehow equitable.
  - Rule35 6y ago
    > no direct connection to criminality like whether or not their parents were separated and whether their parents were ever arrested Again, not right at all. These things heavily correlate with crime. A broken home is the primary indicator of someone's future success, even over a 'better' but broken home. I'm male, and not a rapist, but having a penis heavily correlates with rape. I suggest you do not pick me, or any male, to watch your children. Sure it's rude to the innocent, but oh well, a little rudeness vs potential harm. > I'm not even convinced that you should be allowed to make a decision based on stuff like that even if it is somehow equitable. Then nobody will ever follow the law. If it's illegal for me to reject a potentially bad babysitter for something I know about them that could risk my child's safety I'll happily lie and say I didn't like their haircut. You'd get further if you tried to identify these people and treat them better - grants to move out of bad neighborhoods, to get educated, to get pardons for unrelated crimes, to get counseling for abuse, etc - than with this "wrong unless it's 100% identical, equity-of-outcome" thing.
    - TheCoelacanth 6y ago
      We're talking about an algorithm being used as part of the court system to determine whether or not someone rots in jail, not about how you choose your babysitter. It needs to be held to a higher standard and punish people based on their actions, not based on what their parents did.
      - Rule35 6y ago
Rule35 6y ago
> Fire them until they can demonstrate that the algorithm is not in fact biased like it appears to be. What do the non-black box findings from the area show? If blacks are a poor demographic in the area it might be true. (Crime tracks poverty, not race.) But yeah, certainly don't pay anyone for, or use, a black-box algorithm. > The score also takes into account answers to questions like whether the defendant's parents were separated or whether their parents were ever arrested. Those things are completely out of the defendant's control and are highly correlated with race. Even including those things in their score is damning. Nope. That's perfectly fair to look at. For instance, a broken family is another predictor. It's not fair, but being a victim increases your chance to offend. Similarly, the number one predictor of child sex crimes is to have suffered them yourself. If you have a child and are picking a babysitter, skip the one who was molested. fwiw, those things don't correlate with race, they correlate with poverty which correlates with race. Broken families are more likely to be poor and abusive.