6 ms·

One of my big concerns is whether the algorithms are institutionalizing racism, with legal decisions that make it impossible to challenge these. After all, the

by mcherm 10y ago

One of my big concerns is whether the algorithms are institutionalizing racism, with legal decisions that make it impossible to challenge these. After all, the algorithm has been trained on information about recidivism that was collected in a world where racism skews arrest rates, conviction rates, and sentencing.[1] That means the algorithm is almost certainly baking in a racial bias. Now, I'm sure they aren't foolish enough to put "race" in as one of the input factors, but other correlated factors will allow the algorithm to continue to enforce this racism, but now with legal immunity. [1] Do I really need to footnote this? http://www.huffingtonpost.com/kim-farbota/black-crime-rates-your-st_b_8078586.html http://www.huffingtonpost.com/kim-farbota/black-crime-rates-... is one source that addresses all of these, but there are many, many other sources.

mproy 10y ago
" ... other correlated factors will allow the algorithm to continue to enforce this racism" Which correlated factors are you thinking of?
- advisedwang 10y ago
  Some proxies for race: - Where somebody lives - Their name - What sports teams somebody supports - Their finances - consumer preferences - education level - what medical conditions somebody has - political leanings I'm not saying these factors can't legitimately be used in a risk assessment, but they could be used to make a good bet on race.
  - mproy 10y ago
    I don't see how those factors are relevant at all, regardless of the race of the criminal. The problem to address is the lack of transparency. Trying to guess what inputs are being factored in is pointless.
  - Sacho 10y ago
    Finances? While it is true that blacks and native americans have the highest poverty rates, if you take a random poor person they are about 50% more likely to be white than black. Same with the many of your other "proxies". Therefore they would be a poor bet for race. If you were using these proxies to identify the race of a person jailed for a crime, they might be good predictors, but only marginally more useful than just blindly guessing black, since they constitute the majority of incarcerated people. I think, by attempting to remove the smoke screens a "real racist" would use, you give them a new one - "look, our opponents want to ignore actual data".
- vmarsy 10y ago
  Even if you are careful to remove all features you think could tell the race (let's say geographic location etc), it could still be giving harsher sentences to minorities if those are treated differently upfront. For instance, it would make sense for the model to learn that the more charges count the harsher the sentence. This sounds right: a bank robbery should be less punished than a bank robbery + a carjacking, at least in the US judicial system. But now let's say minorities gets a "resiting arrest" charge on top of their original charge more often, because of many factors such as the police bias, the bias of the minorities towards police, etc. (By bias here I mean all spectrum : racism, but also fear of the police etc). As long as minorities are treated differently upfront, then the model would treat them differently.
  - mproy 10y ago
    But if all of that is true, the model doesn't change the equation. The problems you outline need to be addressed upstream. Trying to account for and correct them at the point of sentencing is very problematic.
smnrchrds 10y ago
You are right to be worried. "It found that widely used software that assessed the risk of recidivism in criminals was twice as likely to mistakenly flag black defendants as being at a higher risk of committing future crimes. It was also twice as likely to incorrectly flag white defendants as low risk." Source: https://www.nytimes.com/2016/06/26/opinion/sunday/artificial-intelligences-white-guy-problem.html https://www.nytimes.com/2016/06/26/opinion/sunday/artificial...
humanrebar 10y ago
> ...other correlated factors will allow the algorithm to continue to enforce this racism, but now with legal immunity. I'm sympathetic, but how do we keep from causing more harm than good here? It seems like blindly ignoring all data that correlates with race is also bad. We do this in some circumstances already (i.e., IQ tests when doing hiring or admissions) and I think it's a defensible point that we're worse off for it.
- hackuser 10y ago
  > blindly ignoring all data that correlates with race is also bad This is a very theoretical, futuristic red herring. Race factors into decision-making far, far too much. It's like worrying that if someone developed a perfect, bug-free crypto library, then everyone would use it and we'd have a monoculture - not really a present concern, and in fact a good problem to have (nor is it an eloquent analogy, but I have to run).
  - humanrebar 10y ago
    I can't really tell if you are for or against algorithmic sentencing from this post. I think there are fair existing examples both for and against it, so I'm not sure calling either position futuristic or theoretical is fair.
pooya72 10y ago
Yeah, there's a good book on this very issue called "Weapons of Math Destruction."[1] The author did an interview on a podcast and discusses how algorithms used by these judges reflect our racial biases.[2] One example that I remember is that the algorithm takes into consideration one's zip code, which correlates strongly with race in the US. [1]: https://weaponsofmathdestructionbook.com/ https://weaponsofmathdestructionbook.com/ [2]: http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.html http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.ht...
cauterized 10y ago
Not to mention that discrimination in hiring decreases opportunities for minorities. Which in turn increases recidivism.
jpab 10y ago
I agree wholeheartedly with this concern, but I think there is also a counter-argument too. If you assume we're starting from a situation where there is bias, then you probably want to look for ways to reduce that bias. The software may have an advantage here too. It may be easier to objectively measure and correct for undesired (racial, or otherwise) bias in an automated system than it is to objectively measure and correct for bias in human judgement. And even if it's more difficult to correct for undesired bias in an automated system, at least that correction is more likely to be permanent and applied uniformly, compared to things like giving training courses for humans, which will vary in effectiveness across the recipients of the course, and will need to be refreshed frequently. There are a lot of qualifications and "maybe"s in that argument though.
JustSomeNobody 10y ago
So, when we train all the AI and robots that are going to take over all of our jobs in the next 5 years, we need to make sure we are very careful.
bbig3831 10y ago
There was a Washington Post article and arXiv paper on this topic several months ago. [1][2] The problem boils down to the definition of bias. Northpointe, the company that owns COMPAS, says the algorithm treats black and white defendants equally, which is true. Others claim that its predictions for blacks are wrong more often than for whites, which is also true. The main point of the paper is that you can't have your cake and eat it too. If you don't want to consider race, then the predictions will be wrong more often for blacks. If you don't want the predictions to be wrong more often for blacks, the algorithm has to consider race. (I'd recommend the articles for a better explanation.) In my mind, the ownership of the software is what's troubling. First, it's not public, so nobody can look at the guts. And second, the Loomis decision constructs even more walls around the software. It's hard for people to critically examine and change something they can't see. [1] https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/17/can-an-algorithm-be-racist-our-analysis-is-more-cautious-than-propublicas/?utm_term=.8b72300511b7 https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/1... [2] https://arxiv.org/pdf/1609.05807.pdf https://arxiv.org/pdf/1609.05807.pdf
tim333 10y ago
On the other hand you could probably find ways to make the algorithms less racist than their human equivalent. No system is totally unbiased but doing better than what we have is possible.
grinnbearit 10y ago
Chris Stucchio wrote an interesting blog post which changed my mind on this. Instead of exacerbating it, a Machine Learning model can actively correct for bias. https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_...
- mrow84 10y ago
  There are some issues with the arguments contained in that post, which I discussed here: https://news.ycombinator.com/item?id=12630980 https://news.ycombinator.com/item?id=12630980 Bias correction is possible, but, tautologically, a model whose structure cannot capture a bias cannot correct for that bias. This means that a modeller must either understand the bias, and accommodate for it in their model, or use a model that might be able to capture unknown structures and run the (serious) risk of over-fitting that model.

by mcherm 10y ago

mproy 10y ago

" ... other correlated factors will allow the algorithm to continue to enforce this racism" Which correlated factors are you thinking of?

advisedwang 10y ago
Some proxies for race: - Where somebody lives - Their name - What sports teams somebody supports - Their finances - consumer preferences - education level - what medical conditions somebody has - political leanings I'm not saying these factors can't legitimately be used in a risk assessment, but they could be used to make a good bet on race.
- mproy 10y ago
  I don't see how those factors are relevant at all, regardless of the race of the criminal. The problem to address is the lack of transparency. Trying to guess what inputs are being factored in is pointless.
- Sacho 10y ago
  Finances? While it is true that blacks and native americans have the highest poverty rates, if you take a random poor person they are about 50% more likely to be white than black. Same with the many of your other "proxies". Therefore they would be a poor bet for race. If you were using these proxies to identify the race of a person jailed for a crime, they might be good predictors, but only marginally more useful than just blindly guessing black, since they constitute the majority of incarcerated people. I think, by attempting to remove the smoke screens a "real racist" would use, you give them a new one - "look, our opponents want to ignore actual data".
vmarsy 10y ago
Even if you are careful to remove all features you think could tell the race (let's say geographic location etc), it could still be giving harsher sentences to minorities if those are treated differently upfront. For instance, it would make sense for the model to learn that the more charges count the harsher the sentence. This sounds right: a bank robbery should be less punished than a bank robbery + a carjacking, at least in the US judicial system. But now let's say minorities gets a "resiting arrest" charge on top of their original charge more often, because of many factors such as the police bias, the bias of the minorities towards police, etc. (By bias here I mean all spectrum : racism, but also fear of the police etc). As long as minorities are treated differently upfront, then the model would treat them differently.
- mproy 10y ago
  But if all of that is true, the model doesn't change the equation. The problems you outline need to be addressed upstream. Trying to account for and correct them at the point of sentencing is very problematic.

smnrchrds 10y ago

You are right to be worried. "It found that widely used software that assessed the risk of recidivism in criminals was twice as likely to mistakenly flag black defendants as being at a higher risk of committing future crimes. It was also twice as likely to incorrectly flag white defendants as low risk." Source: https://www.nytimes.com/2016/06/26/opinion/sunday/artificial-intelligences-white-guy-problem.html https://www.nytimes.com/2016/06/26/opinion/sunday/artificial...

humanrebar 10y ago

> ...other correlated factors will allow the algorithm to continue to enforce this racism, but now with legal immunity. I'm sympathetic, but how do we keep from causing more harm than good here? It seems like blindly ignoring all data that correlates with race is also bad. We do this in some circumstances already (i.e., IQ tests when doing hiring or admissions) and I think it's a defensible point that we're worse off for it.

hackuser 10y ago
> blindly ignoring all data that correlates with race is also bad This is a very theoretical, futuristic red herring. Race factors into decision-making far, far too much. It's like worrying that if someone developed a perfect, bug-free crypto library, then everyone would use it and we'd have a monoculture - not really a present concern, and in fact a good problem to have (nor is it an eloquent analogy, but I have to run).
- humanrebar 10y ago
  I can't really tell if you are for or against algorithmic sentencing from this post. I think there are fair existing examples both for and against it, so I'm not sure calling either position futuristic or theoretical is fair.

pooya72 10y ago

Yeah, there's a good book on this very issue called "Weapons of Math Destruction."[1] The author did an interview on a podcast and discusses how algorithms used by these judges reflect our racial biases.[2] One example that I remember is that the algorithm takes into consideration one's zip code, which correlates strongly with race in the US. [1]: https://weaponsofmathdestructionbook.com/ https://weaponsofmathdestructionbook.com/ [2]: http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.html http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.ht...

cauterized 10y ago

Not to mention that discrimination in hiring decreases opportunities for minorities. Which in turn increases recidivism.

jpab 10y ago

I agree wholeheartedly with this concern, but I think there is also a counter-argument too. If you assume we're starting from a situation where there is bias, then you probably want to look for ways to reduce that bias. The software may have an advantage here too. It may be easier to objectively measure and correct for undesired (racial, or otherwise) bias in an automated system than it is to objectively measure and correct for bias in human judgement. And even if it's more difficult to correct for undesired bias in an automated system, at least that correction is more likely to be permanent and applied uniformly, compared to things like giving training courses for humans, which will vary in effectiveness across the recipients of the course, and will need to be refreshed frequently. There are a lot of qualifications and "maybe"s in that argument though.

JustSomeNobody 10y ago

So, when we train all the AI and robots that are going to take over all of our jobs in the next 5 years, we need to make sure we are very careful.

bbig3831 10y ago

There was a Washington Post article and arXiv paper on this topic several months ago. [1][2] The problem boils down to the definition of bias. Northpointe, the company that owns COMPAS, says the algorithm treats black and white defendants equally, which is true. Others claim that its predictions for blacks are wrong more often than for whites, which is also true. The main point of the paper is that you can't have your cake and eat it too. If you don't want to consider race, then the predictions will be wrong more often for blacks. If you don't want the predictions to be wrong more often for blacks, the algorithm has to consider race. (I'd recommend the articles for a better explanation.) In my mind, the ownership of the software is what's troubling. First, it's not public, so nobody can look at the guts. And second, the Loomis decision constructs even more walls around the software. It's hard for people to critically examine and change something they can't see. [1] https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/17/can-an-algorithm-be-racist-our-analysis-is-more-cautious-than-propublicas/?utm_term=.8b72300511b7 https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/1... [2] https://arxiv.org/pdf/1609.05807.pdf https://arxiv.org/pdf/1609.05807.pdf

tim333 10y ago

On the other hand you could probably find ways to make the algorithms less racist than their human equivalent. No system is totally unbiased but doing better than what we have is possible.

grinnbearit 10y ago

Chris Stucchio wrote an interesting blog post which changed my mind on this. Instead of exacerbating it, a Machine Learning model can actively correct for bias. https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_...

mrow84 10y ago
There are some issues with the arguments contained in that post, which I discussed here: https://news.ycombinator.com/item?id=12630980 https://news.ycombinator.com/item?id=12630980 Bias correction is possible, but, tautologically, a model whose structure cannot capture a bias cannot correct for that bias. This means that a modeller must either understand the bias, and accommodate for it in their model, or use a model that might be able to capture unknown structures and run the (serious) risk of over-fitting that model.