5 ms·

Discrimination in the Age of Algorithms

49 points by martinlaz 8y ago

eponeponepon 8y ago
This cuts both ways, of course - in some ways, I think we want automated systems to be able to discriminate. A good example is speed cameras - in days past, a man speeding to get his wife to the maternity ward could be pulled over and then hurriedly wished good luck, maybe even getting an escort. These days, he gets a fine in the post three weeks later.
- skookumchuck 8y ago
  There should always be a human in the loop when justice is meted out. There's no way to write a law or an algorithm that covers every circumstance. Unthinking "by the book" enforcement is tyranny. (Like those wretched "zero tolerance" laws, where everyone knows they are unjust, but feel impelled to literally interpret the law.)
- tedivm 8y ago
  How would an automated system actually deal with that scenario?
  - eponeponepon 8y ago
    In theory, the same way as a policeman - assessing the situation and the context, and then making a decision not to issue the fine. But in practice, I don't think it can, without having total surveillance capabilities - not to mention essentially sci-fi quantities of AI advancement.
    - gumby 8y ago
      In principle both systems permit this kind of correction: you could contest the ticket by going to court.
- kevinventullo 8y ago
  Isn't it better now, since they arrive at the maternity ward sooner?
nostromo 8y ago
The paper's authors have the best of intentions, but to me race-aware algorithms programed for social justice sounds completely dystopian. If a black man and a white man rob a bank together, is it fair that they receive the same sentence? Or is it fair that a sentencing algorithm give the black man probation and send the white man to jail to correct for disparate outcomes based on race?
- cyberbanjo 8y ago
  Ironically I think algorithms would do the opposite, with the training data that is current justice system antics.
- raverbashing 8y ago
  It's not so much about programming for social justice, but accounting for biases that are present in the current status quo and that end up in training data There was a recent talk from a Meetup engineer about this but I can't find the link right now.
  - danieltillett 8y ago
    Then shouldn't the algorithms incorporate the unbiased data rather than incorporate something we know is biased to correct for other biases? A question. How do we know the algorithm is biased if it give a racial disparate result? Don't tell me the evidence is that the result is racial disparate.
    - raverbashing 8y ago
      There usually isn't unbiased data (think: law enforcement databases), though you could maybe remove the variables more likely to be discriminated against and train on that There are statistical analysis that can answer your second question, and it's not only based on disparate results (though that's probably part of it). And the solution is not "changing the signal" of the bias
      - 8y ago
Cynddl 8y ago
From Jon Kleinberg also, I highly watching his recommend his presentation [1] on the inherent trade-offs when trying to achieve algorithmic fairness. It's a little more technical than this paper, and presents very well why removing discrimination in algorithmic decision making is a complex task. [1] https://www.youtube.com/watch?v=4X3Z7FPwkA8 https://www.youtube.com/watch?v=4X3Z7FPwkA8
- nostromo 8y ago
  Note that this paper is arguing for race-aware algorithms, not removing discrimination. If I'm understanding the authors correctly, they're saying that blind algorithms don't have equal outcomes by race -- so they're reintroducing race to the algorithms so they can adjust for disparate outcomes. Basically they're arguing for affirmative action algorithms.
  - currymj 8y ago
    it is way more complicated than this. i know i underestimated how complicated these issues can be at first. there are many different failure modes. the race-blind models aren't necessarily really race-neutral; they could implicitly have something like "affirmative action" in either direction, and reintroducing an explicit race variable could be the only way to get rid of it. or the opposite might be true! it takes real care to get this stuff right whether notion of equality you aim for.
jondubois 8y ago
As soon as you label people based on superficial personal traits that are outside of their control and without their consent, that's discrimination. If you start giving some people special treatment just because of their ethnicity or gender, you're going to be opening up more cracks in the system for other people to fall through - and it will hurt those people even more because they will rightly feel that the whole system is working against them personally; down to the core specifics of who they are (unfortunate individuals who don't fit under any special label). Modern anti-discrimination approaches are often racist or sexist. They should not try to classify individuals into superficial groups, instead, they should be decided on an individual case-by-case basis based on the individual's history. The root of injustice is simply bad luck. Ethnicity and gender are only loosely correlated with bad luck but there is no causal relationship between them (not so much anymore at least). Anti-discrimination efforts should be aimed at averaging-out the effects of bad luck in people's lives; so that means we need a way to quantify luck on an individual basis. It's a difficult problem to solve, but it can't be solved by making gross generalizations.
- trowawee 8y ago
  > The root of injustice is simply bad luck. Ethnicity and gender are only loosely correlated with bad luck but there is no causal relationship between them (not so much anymore at least). This is wildly ignorant and ahistorical. There are literally thousands of books, articles, studies, movies, etc. disproving this. Literally just this week a study was released showing that predominantly white school districts collectively receive $23 billion more in funding than predominantly black districts, and that: "For every student enrolled, the average nonwhite school district receives $2,226 less than a white school district,".[1] That isn't bad luck; that is a system designed to reinforce a system of racial segregation, and it's just one example of thousands, all of which are a moment of research away if you actually cared about this. [1]: https://www.npr.org/2019/02/26/696794821/why-white-school-districts-have-so-much-more-money?utm_source=twitter.com&utm_medium=social&utm_campaign=npr&utm_term=nprnews&utm_content=20190226 https://www.npr.org/2019/02/26/696794821/why-white-school-di...
  - DoofusOfDeath 8y ago

49 points by martinlaz 8y ago

This cuts both ways, of course - in some ways, I think we want automated systems to be able to discriminate. A good example is speed cameras - in days past, a man speeding to get his wife to the maternity ward could be pulled over and then hurriedly wished good luck, maybe even getting an escort. These days, he gets a fine in the post three weeks later.

skookumchuck 8y ago
There should always be a human in the loop when justice is meted out. There's no way to write a law or an algorithm that covers every circumstance. Unthinking "by the book" enforcement is tyranny. (Like those wretched "zero tolerance" laws, where everyone knows they are unjust, but feel impelled to literally interpret the law.)
tedivm 8y ago
How would an automated system actually deal with that scenario?
- eponeponepon 8y ago
  In theory, the same way as a policeman - assessing the situation and the context, and then making a decision not to issue the fine. But in practice, I don't think it can, without having total surveillance capabilities - not to mention essentially sci-fi quantities of AI advancement.
  - gumby 8y ago
    In principle both systems permit this kind of correction: you could contest the ticket by going to court.
kevinventullo 8y ago
Isn't it better now, since they arrive at the maternity ward sooner?

nostromo 8y ago

The paper's authors have the best of intentions, but to me race-aware algorithms programed for social justice sounds completely dystopian. If a black man and a white man rob a bank together, is it fair that they receive the same sentence? Or is it fair that a sentencing algorithm give the black man probation and send the white man to jail to correct for disparate outcomes based on race?

cyberbanjo 8y ago
Ironically I think algorithms would do the opposite, with the training data that is current justice system antics.
raverbashing 8y ago
It's not so much about programming for social justice, but accounting for biases that are present in the current status quo and that end up in training data There was a recent talk from a Meetup engineer about this but I can't find the link right now.
- danieltillett 8y ago
  Then shouldn't the algorithms incorporate the unbiased data rather than incorporate something we know is biased to correct for other biases? A question. How do we know the algorithm is biased if it give a racial disparate result? Don't tell me the evidence is that the result is racial disparate.
  - raverbashing 8y ago
    There usually isn't unbiased data (think: law enforcement databases), though you could maybe remove the variables more likely to be discriminated against and train on that There are statistical analysis that can answer your second question, and it's not only based on disparate results (though that's probably part of it). And the solution is not "changing the signal" of the bias
    - 8y ago

Cynddl 8y ago

From Jon Kleinberg also, I highly watching his recommend his presentation [1] on the inherent trade-offs when trying to achieve algorithmic fairness. It's a little more technical than this paper, and presents very well why removing discrimination in algorithmic decision making is a complex task. [1] https://www.youtube.com/watch?v=4X3Z7FPwkA8 https://www.youtube.com/watch?v=4X3Z7FPwkA8

nostromo 8y ago
Note that this paper is arguing for race-aware algorithms, not removing discrimination. If I'm understanding the authors correctly, they're saying that blind algorithms don't have equal outcomes by race -- so they're reintroducing race to the algorithms so they can adjust for disparate outcomes. Basically they're arguing for affirmative action algorithms.
- currymj 8y ago
  it is way more complicated than this. i know i underestimated how complicated these issues can be at first. there are many different failure modes. the race-blind models aren't necessarily really race-neutral; they could implicitly have something like "affirmative action" in either direction, and reintroducing an explicit race variable could be the only way to get rid of it. or the opposite might be true! it takes real care to get this stuff right whether notion of equality you aim for.

jondubois 8y ago

As soon as you label people based on superficial personal traits that are outside of their control and without their consent, that's discrimination. If you start giving some people special treatment just because of their ethnicity or gender, you're going to be opening up more cracks in the system for other people to fall through - and it will hurt those people even more because they will rightly feel that the whole system is working against them personally; down to the core specifics of who they are (unfortunate individuals who don't fit under any special label). Modern anti-discrimination approaches are often racist or sexist. They should not try to classify individuals into superficial groups, instead, they should be decided on an individual case-by-case basis based on the individual's history. The root of injustice is simply bad luck. Ethnicity and gender are only loosely correlated with bad luck but there is no causal relationship between them (not so much anymore at least). Anti-discrimination efforts should be aimed at averaging-out the effects of bad luck in people's lives; so that means we need a way to quantify luck on an individual basis. It's a difficult problem to solve, but it can't be solved by making gross generalizations.

trowawee 8y ago
> The root of injustice is simply bad luck. Ethnicity and gender are only loosely correlated with bad luck but there is no causal relationship between them (not so much anymore at least). This is wildly ignorant and ahistorical. There are literally thousands of books, articles, studies, movies, etc. disproving this. Literally just this week a study was released showing that predominantly white school districts collectively receive $23 billion more in funding than predominantly black districts, and that: "For every student enrolled, the average nonwhite school district receives $2,226 less than a white school district,".[1] That isn't bad luck; that is a system designed to reinforce a system of racial segregation, and it's just one example of thousands, all of which are a moment of research away if you actually cared about this. [1]: https://www.npr.org/2019/02/26/696794821/why-white-school-districts-have-so-much-more-money?utm_source=twitter.com&utm_medium=social&utm_campaign=npr&utm_term=nprnews&utm_content=20190226 https://www.npr.org/2019/02/26/696794821/why-white-school-di...
- DoofusOfDeath 8y ago