4 ms·

I assume this might be an unpopular opinion, but shouldn't these programs be evaluated on some metric of success rather than their racial selection? If the sof

by casion 7y ago

I assume this might be an unpopular opinion, but shouldn't these programs be evaluated on some metric of success rather than their racial selection? If the software is accurate at picking choices that match the set goals, then it would seem that: the software is doing its job and is 'blameless', the racism is on the side of those that set the goals and/or the people claiming 'racism' because a person of a given skin colour was selected. If it could be shown that there's low success in the software, yet it's relied on despite racial bias, then I would think few rational people could argue against its removal. (edit: s/for/against) Edit: I suppose that I'm asking for better auditing of the results vs the desired outcomes. Race seems like a red-herring until it's shown that the metrics support poor performance of these systems.

deleted 7y ago
[deleted]
qtplatypus 7y ago
How do you evaluate how successful an algorithm to decide if someone should get bail is? The problem with most of these deep learning/AI systems is that they are more or less pattern detection systems. To an AI racial biases in its input data is just another pattern.
- casion 7y ago
  Measure against recidivism and skipping bail? Surely we have a baseline that can be used, such as when these systems are not in play?
  - qtplatypus 7y ago
    If skipping bail is the criteria then how do you detect when you incorrectly denied bail?
    - casion 7y ago
      Doesn't the verdict of the trial related to the bail request have impact upon that? (legitimate question. I don't know if non-guilty parties are more/less likely to skip than guilty parties?) That's assuming that we can agree that the justice system works impartially to begin with. I don't know of many people (including myself) that could agree with that.
    - chongli 7y ago
      Use a trial period where you grant bail to everyone (non violent offences only). You could run many different algorithms and models, recording all of their predictions. Since you granted everyone bail, you have a clear picture for which predictors accurately denied bail to the skippers.
MaxBarraclough 7y ago
> If the software is accurate at picking choices that match the set goals, then it would seem that: the software is doing its job and is 'blameless', the racism is on the side of those that set the goals and/or the people claiming 'racism' because a person of a given skin colour was selected. Well, no. It's not enough to simply decline to input racial data into the machine-learning system. The system may discover a proxy for race, and make decisions based on that proxy. Name and address, for instance, may be enough to give a good idea as to someone's race. We expect human judges and parole-boards, for example, to discount racial information, even if there are demonstrable correlations associated with race. If the machine-learning system is not built with that discounting in mind, it isn't going to do so of its own accord, as it was of course built to detect and exploit correlations. > If it could be shown that there's low success in the software, yet it's relied on despite racial bias, then I would think few rational people could argue for its removal. I don't follow. If the software has been unsuccessful, that seems like a good reason to remove it. If, additionally, it has been making decisions on racial grounds, that seems like another good reason to remove it. > Race seems like a red-herring until it's shown that the metrics support poor performance of these systems. As I hope I've made clear above, this isn't right. It's not just about performance, it's also about ensuring the system does not make decisions on the basis of an individual's race.
- Yessing 7y ago
  Why should the system have access to the candidates' name? You've been denied parol because someone with a name similar to you skipped bail is unfair regardless of race. You've been denied parol because you come from a rough neighborhood is also unfair, regardless of race. In fact, even if 99.9% of all people named X skipped bail, it would be unjust to deny somone bail because his name is X.
- casion 7y ago
  I'm confused here, I read the article and it was somewhat muddy about the point trying to be made other than "machine bad!". Racially biased could mean two things: despite not taking into account race the algorithm groups results racially at a statistically significant rate, or the algorithm is outright using race as a factor. I assumed the former because it was not made clear in the article what was being talked about. Half the article is about things that may/may not use racial information (it's not said), and half the article is about things that do. Are we talking about the same thing here?
lidHanteyk 7y ago
By Goodhart's Law, there does not exist such a metric of success. All success metrics are arbitrary in this situation. Software is written by people. Racist software comes from racist beliefs. A belief that a neural network will learn objectivity is a belief in objectivity, and both are wrong. Race is not a red herring; even though race may be only a social construction with no objective science behind it, the fact that people believe in and act as if race were real is a problem. In particular it often violates the Civil Rights Acts.
yabadabadoes 7y ago
Suppose racism, discrimination and bullying by the general public were the dominant cause of recidivism.. Would it make sense to allow people from the majority regularly do 3 month sentences and keep members of any minority group indefinitely on their first incident?
_0ffh 7y ago
I think the true question is how you decide if an algorithm is actually biased or not. At a first shot I would suggest something like this: Divide some otherwise randomly sampled testing data into two sets, depending on the characteristic in question. If the ROC curve is approximately the same for both sets, I'd guess you could say the algorithm is not biased.
pdkl95 7y ago
> If the software is accurate at picking choices that match the set goals > auditing of the results vs the desired outcomes The core problem is this assumption that the goal is to prioritize accuracy. When making "life-changing decisions ... in areas from job applications to immigration", many different goals need to be balanced. Who gets to define what "desired outcome" means? We already have a hard time simply defining those goals. How would you even begin to define a cost function for e.g. "immigration" decisions when simply bringing up the topic is one of the easier ways to start a heated political argument? > then it would seem that: the software is doing its job and is 'blameless' The software is doing exactly what it was told to do: find a particular signal in some set of data (without explicit race. The problem is we spent several hundred years systemically embedding race into most aspects of society, which becomes a minefields of problematic local minima/maxima traps for your ML algorithm. Let's say we have an good ML algorithm that provides some sort of evaluation about job applicants. Nothing related to race is included. The algorithm is used by a company in Chicago[1][3] or D.C.[2][3] hoping to hire the "best" people. Segregated populations means different schools/etc. Even if you didn't give the algorithm race data, most ML is practically guaranteed to find one of the many ways to infer race. Unintentionally, the ML algorithm ends up doing something similar to redlining[4], because that's what is in the data: the systemic effects of historical racism that still reverberate through society. [1] https://www.washingtonpost.com/graphics/2018/national/segregation-us-cities/img/Chicago_map_2016-600.jpg https://www.washingtonpost.com/graphics/2018/national/segreg... [2] https://www.washingtonpost.com/graphics/2018/national/segregation-us-cities/img/DC_map_2016-600.jpg https://www.washingtonpost.com/graphics/2018/national/segreg... [3] red-dots="White", blue-dots="Black", yellow-dots="Hispanic", etc. See the full article for many more examples: https://www.washingtonpost.com/graphics/2018/national/segregation-us-cities/ https://www.washingtonpost.com/graphics/2018/national/segreg... [4] https://en.wikipedia.org/wiki/Redlining https://en.wikipedia.org/wiki/Redlining

by casion 7y ago

deleted 7y ago

[deleted]

qtplatypus 7y ago

How do you evaluate how successful an algorithm to decide if someone should get bail is? The problem with most of these deep learning/AI systems is that they are more or less pattern detection systems. To an AI racial biases in its input data is just another pattern.

casion 7y ago
Measure against recidivism and skipping bail? Surely we have a baseline that can be used, such as when these systems are not in play?
- qtplatypus 7y ago
  If skipping bail is the criteria then how do you detect when you incorrectly denied bail?
  - casion 7y ago
    Doesn't the verdict of the trial related to the bail request have impact upon that? (legitimate question. I don't know if non-guilty parties are more/less likely to skip than guilty parties?) That's assuming that we can agree that the justice system works impartially to begin with. I don't know of many people (including myself) that could agree with that.
  - chongli 7y ago
    Use a trial period where you grant bail to everyone (non violent offences only). You could run many different algorithms and models, recording all of their predictions. Since you granted everyone bail, you have a clear picture for which predictors accurately denied bail to the skippers.

MaxBarraclough 7y ago

> If the software is accurate at picking choices that match the set goals, then it would seem that: the software is doing its job and is 'blameless', the racism is on the side of those that set the goals and/or the people claiming 'racism' because a person of a given skin colour was selected. Well, no. It's not enough to simply decline to input racial data into the machine-learning system. The system may discover a proxy for race, and make decisions based on that proxy. Name and address, for instance, may be enough to give a good idea as to someone's race. We expect human judges and parole-boards, for example, to discount racial information, even if there are demonstrable correlations associated with race. If the machine-learning system is not built with that discounting in mind, it isn't going to do so of its own accord, as it was of course built to detect and exploit correlations. > If it could be shown that there's low success in the software, yet it's relied on despite racial bias, then I would think few rational people could argue for its removal. I don't follow. If the software has been unsuccessful, that seems like a good reason to remove it. If, additionally, it has been making decisions on racial grounds, that seems like another good reason to remove it. > Race seems like a red-herring until it's shown that the metrics support poor performance of these systems. As I hope I've made clear above, this isn't right. It's not just about performance, it's also about ensuring the system does not make decisions on the basis of an individual's race.

Yessing 7y ago
Why should the system have access to the candidates' name? You've been denied parol because someone with a name similar to you skipped bail is unfair regardless of race. You've been denied parol because you come from a rough neighborhood is also unfair, regardless of race. In fact, even if 99.9% of all people named X skipped bail, it would be unjust to deny somone bail because his name is X.
casion 7y ago
I'm confused here, I read the article and it was somewhat muddy about the point trying to be made other than "machine bad!". Racially biased could mean two things: despite not taking into account race the algorithm groups results racially at a statistically significant rate, or the algorithm is outright using race as a factor. I assumed the former because it was not made clear in the article what was being talked about. Half the article is about things that may/may not use racial information (it's not said), and half the article is about things that do. Are we talking about the same thing here?

lidHanteyk 7y ago

By Goodhart's Law, there does not exist such a metric of success. All success metrics are arbitrary in this situation. Software is written by people. Racist software comes from racist beliefs. A belief that a neural network will learn objectivity is a belief in objectivity, and both are wrong. Race is not a red herring; even though race may be only a social construction with no objective science behind it, the fact that people believe in and act as if race were real is a problem. In particular it often violates the Civil Rights Acts.

yabadabadoes 7y ago

Suppose racism, discrimination and bullying by the general public were the dominant cause of recidivism.. Would it make sense to allow people from the majority regularly do 3 month sentences and keep members of any minority group indefinitely on their first incident?

_0ffh 7y ago

I think the true question is how you decide if an algorithm is actually biased or not. At a first shot I would suggest something like this: Divide some otherwise randomly sampled testing data into two sets, depending on the characteristic in question. If the ROC curve is approximately the same for both sets, I'd guess you could say the algorithm is not biased.

pdkl95 7y ago

> If the software is accurate at picking choices that match the set goals > auditing of the results vs the desired outcomes The core problem is this assumption that the goal is to prioritize accuracy. When making "life-changing decisions ... in areas from job applications to immigration", many different goals need to be balanced. Who gets to define what "desired outcome" means? We already have a hard time simply defining those goals. How would you even begin to define a cost function for e.g. "immigration" decisions when simply bringing up the topic is one of the easier ways to start a heated political argument? > then it would seem that: the software is doing its job and is 'blameless' The software is doing exactly what it was told to do: find a particular signal in some set of data (without explicit race. The problem is we spent several hundred years systemically embedding race into most aspects of society, which becomes a minefields of problematic local minima/maxima traps for your ML algorithm. Let's say we have an good ML algorithm that provides some sort of evaluation about job applicants. Nothing related to race is included. The algorithm is used by a company in Chicago[1][3] or D.C.[2][3] hoping to hire the "best" people. Segregated populations means different schools/etc. Even if you didn't give the algorithm race data, most ML is practically guaranteed to find one of the many ways to infer race. Unintentionally, the ML algorithm ends up doing something similar to redlining[4], because that's what is in the data: the systemic effects of historical racism that still reverberate through society. [1] https://www.washingtonpost.com/graphics/2018/national/segregation-us-cities/img/Chicago_map_2016-600.jpg https://www.washingtonpost.com/graphics/2018/national/segreg... [2] https://www.washingtonpost.com/graphics/2018/national/segregation-us-cities/img/DC_map_2016-600.jpg https://www.washingtonpost.com/graphics/2018/national/segreg... [3] red-dots="White", blue-dots="Black", yellow-dots="Hispanic", etc. See the full article for many more examples: https://www.washingtonpost.com/graphics/2018/national/segregation-us-cities/ https://www.washingtonpost.com/graphics/2018/national/segreg... [4] https://en.wikipedia.org/wiki/Redlining https://en.wikipedia.org/wiki/Redlining