5 ms·

The money quote: "I don’t care what your politics are. If you’re building a data-driven system and you’re not actively seeking to combat prejudice, you’re build

by beat 10y ago

The money quote: "I don’t care what your politics are. If you’re building a data-driven system and you’re not actively seeking to combat prejudice, you’re building a discriminatory system." This is extremely important. The weight of existing prejudice drives and reinforces future prejudice. Garbage in, garbage out. If you start with biased, flawed data, and you look for patterns like the biased, flawed data, you'll just add to the biases and flaws. You need to account for the poor data quality explicitly.

ben_jones 10y ago
I love how we work in an industry where it's ok to manipulate massive numbers of people because as engineers we know what's best for them. \s.
TruthAndDare 10y ago
I don't follow you. When do you start with biased data? Furthermore, wouldn't a advertisement system that somehow progresses to an increasingly erroneous view be broken anyway?
- rm_-rf_slash 10y ago
  If historic and contemporary data assumes American blacks live in the city and whites live in the suburbs, data alone will not "know" the histories of race-based discrimination. So if it advertises apartments to blacks and houses to whites, it does not "understand" that it is propagating a stereotype. A human has to intervene.
  - TruthAndDare 10y ago
    The point of the algorithm is simply to choose advertisements as successfully as possible.
- ubernostrum 10y ago
  Current distributions of property, wealth, access to services, etc. are all rooted in discriminatory laws and policies of the (not always distant) past. Which creates a risk that blindly pointing an algorithm at these data sets will result in an algorithm which assumes these are the expected/natural distributions and will optimize to perpetuate them, essentially becoming an algorithmic implementation of the prior discriminatory laws and policies. Thus, unless the goal is to perpetuate the effects of discrimination, algorithms need to be carefully set up and trained to understand that "this isn't how things are supposed to be".
  - selectron 10y ago
    The algorithm just tries to model the world as it actually is, not as it is supposed to be. The algorithm cares nothing about uncomfortable truths. The algorithm should not be set up and trained to reflect a wrong model of reality, but rather you should consider if you want to use an algorithm based on reality or one based on how you want the world to be.
    - Joof 10y ago
      Fair enough. In many cases we would prefer to model it how we want the world to be in order to overcome human misconceptions that have caused the world to be as it is currently.
      - prodigal_erik 10y ago
        Society would have to compensate me for turning a blind eye to today's reality, because otherwise I'm giving myself a competitive disadvantage vs. people with accurate tools.
    - cortesoft 10y ago
      It really depends on what your algorithm is determining.
    - dwaltrip 10y ago
      Yes, I'm glad we are in agreement. We shouldn't limit ourselves to current state of the world. We should design systems that help us get closer to an improved world.
    - beat 10y ago
      This isn't hard to understand. Suppose crime is equal in two neighborhoods, but the police patrol one neighborhood 500% more than the other. What would the result of the algorithm be? Five times more arrests in one neighborhood than another. Now, take that model and apply a really crude analysis. You would conclude that there is five times more crime in the more heavily patrolled neighborhood. Your algorithm, applied to policy, suggests that crime could be controlled better by patrolling the "bad" neighborhood ten times as much, rather than five times as much. Now, consider the social result. The "high crime neighborhood" then sees its property values drop, and more homes sold into the rental market. This attracts poorer, rougher people who can't afford the "good" neighborhood. Now crime actually goes up, reinforcing the idea that the bad neighborhood is high-crime. Now, imagine we've been doing this to those two neighborhoods for generations. What are the likely results of your algorithm? There's your uncomfortable truth.
- pdkl95 10y ago
  > When do you start with biased data? In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. "What are you doing?", asked Minsky. "I am training a randomly wired neural net to play Tic-Tac-Toe" Sussman replied. "Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play", Sussman said. Minsky then shut his eyes. "Why do you close your eyes?", Sussman asked his teacher. "So that the room will be empty." At that moment, Sussman was enlightened. (from: http://catb.org/jargon/html/koans.html#id3141241 http://catb.org/jargon/html/koans.html#id3141241 ) There is always bias (and other types of error) in data. There is even bias in the choice of which data to use and the type of analysis to perform. If you think that this isn't a problem, you should really read about practices like "redlining"[1]. For many decades segregation was (and still is) enforced by opaque "loan approval" methods that just happened to always deny loans to blacks. [1] http://www.theatlantic.com/magazine/archive/2014/06/the-case-for-reparations/361631/ http://www.theatlantic.com/magazine/archive/2014/06/the-case...
Forge36 10y ago
I'd go a step back from "flawed data". I'd call it a bad sample size. See [Researchers have long known that whites are more likely to use and sell drugs. And yet, who is arrested for drugs? Blacks. 13% of the US population is black, but over 60% of those in prison are black.] So how would this happen? One possibility: if a crime happens more police frequent there. So you find more crime. Because we've over-sampled black neighborhoods (including DNA testing making it more likely to find relatives of someone already in prison) the mistaken belief becomes confirmed, and reinforced by the continued arrests. I'm unaware of how to call out bad sample sizes in data. However doing so may help highlight invalid assumptions.
- danielvf 10y ago
  You said "13% of the US population is black, but over 60% of those in prison are black." Blacks have a six times greater homicide rate than the rest of the US population, according to FBI extended homicide data. Almost half of homicides in the US in the last year I checked were by African Americans. With the long prison sentences given for violent crimes, violent crimes are disproportionately represented in prison populations. As a thought experiment, imagine that each year one person with a thirty year sentance was committed, and one person with a one year sentence. One the system stabled out, 97% of the prison population would be serving 30 year terms.
  - Forge36 10y ago
    That's also very good to point out. Please note I quoted the article. You point out a bias in the statistic. Of the imprisoned population. What are the percentage breakdowns for types of crime?
    - danielvf 10y ago
      I did some research on this last year. Here's what I found: https://news.ycombinator.com/item?id=10318406 https://news.ycombinator.com/item?id=10318406
    - Nadya 10y ago
selectron 10y ago
This proposal shows more concern about being politically correct than actually correct. A truly unbiased system would just make the best analysis possible given the data available. What you propose is to make a biased system in order to give an advantage to one group of people rather than the other group. This type of political correctness fuels resentment.
- aoki 10y ago
  consider danah's example in which external studies show that blacks are convicted for drug crimes at a rate disproportionate to the actual crime incidence rate. the fact that the easily-available training data (court stats) is unrepresentative of the underlying process (crime incidence) is a problem that has to be corrected or predictive output will also over-represent them. the point is that this prior or correction is also part of the "data available" but you have to actually know something about the problem domain to know about it; you can't just grab a court data dump and pour it into a model and say, "welp, that was the data available to me, it's not biased."
- cortesoft 10y ago
  What they are saying is that, if you know your data is biased, you have the option of designing your system to compensate for that bias. Why do you seem to think that you have to just throw your hands up in the air and accept the bias? Wouldn't the 'best possible analysis given the data available' include compensating for known biases in the data? If I was designing a system for analyzing temperature data, for example, and I knew that every measurement was 10 degrees warmer than the actual temperature, shouldn't my analysis system compensate for that?
  - ZoF 10y ago
    The issue here is how am I as a software developer supposed to combat that bias? Given the contrived arrest rate example above, do I multiply every black arrest by .7 to compensate for bias? How would a system built this way be useful in practice? Why are these parallels always drawn against racial divides as well? How are we compensating for the poor white kid missing out or the rich black kid getting benefits he might not need? It's such a messy problem but I don't think building bias into software is the solution. I prefer factually correct, biased systems to incorrect, still-biased-just-in-another-way systems.
kordless 10y ago
Except: "So long as man is approaching Ultimate Wisdom, he is still dwelling in the realm of consciousness. If he is to realise Nirvana, he must pass beyond the realm of consciousness. In highest Samadhi when consciousness has been transcended, he has passed beyond discrimination and knowledge, beyond any reach of change or fear. He is already enjoying Nirvana." - Sutra of Transcendental Wisdom
teacup50 10y ago
No. I'm not obligated to subscribe to your politics, and I won't coerce or bully others, either. Programmers are not the personal army of any social cause. I work hard to accurately model engineering problems, and apply consistent ethics in doing so. If that aligns with your definition of "combating prejudice", great for you, but I need the latitude to critically analyze technical problems without the threat of bullying and shaming for wrong-think. If I need to understand how surnames work across cultures, understand how time works across calendar systems, or understand what "marriage" means to the people getting married, then that's what I'll do. If I need to support non-binary 'gender' selections because that's what users want, that's what I'll do. On the other hand, if I need to support binary gender for the purposes of simplified data analysis under the assumption of two biological genders with a low probability of intersex individuals, that's what I'll do, too -- politely, and without rancor, and with all the respect for others I can muster.

by beat 10y ago

ben_jones 10y ago

I love how we work in an industry where it's ok to manipulate massive numbers of people because as engineers we know what's best for them. \s.

TruthAndDare 10y ago

I don't follow you. When do you start with biased data? Furthermore, wouldn't a advertisement system that somehow progresses to an increasingly erroneous view be broken anyway?

rm_-rf_slash 10y ago
If historic and contemporary data assumes American blacks live in the city and whites live in the suburbs, data alone will not "know" the histories of race-based discrimination. So if it advertises apartments to blacks and houses to whites, it does not "understand" that it is propagating a stereotype. A human has to intervene.
- TruthAndDare 10y ago
  The point of the algorithm is simply to choose advertisements as successfully as possible.
ubernostrum 10y ago
Current distributions of property, wealth, access to services, etc. are all rooted in discriminatory laws and policies of the (not always distant) past. Which creates a risk that blindly pointing an algorithm at these data sets will result in an algorithm which assumes these are the expected/natural distributions and will optimize to perpetuate them, essentially becoming an algorithmic implementation of the prior discriminatory laws and policies. Thus, unless the goal is to perpetuate the effects of discrimination, algorithms need to be carefully set up and trained to understand that "this isn't how things are supposed to be".
- selectron 10y ago
  The algorithm just tries to model the world as it actually is, not as it is supposed to be. The algorithm cares nothing about uncomfortable truths. The algorithm should not be set up and trained to reflect a wrong model of reality, but rather you should consider if you want to use an algorithm based on reality or one based on how you want the world to be.
  - Joof 10y ago
    Fair enough. In many cases we would prefer to model it how we want the world to be in order to overcome human misconceptions that have caused the world to be as it is currently.
    - prodigal_erik 10y ago
      Society would have to compensate me for turning a blind eye to today's reality, because otherwise I'm giving myself a competitive disadvantage vs. people with accurate tools.
  - cortesoft 10y ago
    It really depends on what your algorithm is determining.
  - dwaltrip 10y ago
    Yes, I'm glad we are in agreement. We shouldn't limit ourselves to current state of the world. We should design systems that help us get closer to an improved world.
  - beat 10y ago
    This isn't hard to understand. Suppose crime is equal in two neighborhoods, but the police patrol one neighborhood 500% more than the other. What would the result of the algorithm be? Five times more arrests in one neighborhood than another. Now, take that model and apply a really crude analysis. You would conclude that there is five times more crime in the more heavily patrolled neighborhood. Your algorithm, applied to policy, suggests that crime could be controlled better by patrolling the "bad" neighborhood ten times as much, rather than five times as much. Now, consider the social result. The "high crime neighborhood" then sees its property values drop, and more homes sold into the rental market. This attracts poorer, rougher people who can't afford the "good" neighborhood. Now crime actually goes up, reinforcing the idea that the bad neighborhood is high-crime. Now, imagine we've been doing this to those two neighborhoods for generations. What are the likely results of your algorithm? There's your uncomfortable truth.
pdkl95 10y ago
> When do you start with biased data? In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. "What are you doing?", asked Minsky. "I am training a randomly wired neural net to play Tic-Tac-Toe" Sussman replied. "Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play", Sussman said. Minsky then shut his eyes. "Why do you close your eyes?", Sussman asked his teacher. "So that the room will be empty." At that moment, Sussman was enlightened. (from: http://catb.org/jargon/html/koans.html#id3141241 http://catb.org/jargon/html/koans.html#id3141241 ) There is always bias (and other types of error) in data. There is even bias in the choice of which data to use and the type of analysis to perform. If you think that this isn't a problem, you should really read about practices like "redlining"[1]. For many decades segregation was (and still is) enforced by opaque "loan approval" methods that just happened to always deny loans to blacks. [1] http://www.theatlantic.com/magazine/archive/2014/06/the-case-for-reparations/361631/ http://www.theatlantic.com/magazine/archive/2014/06/the-case...

Forge36 10y ago

I'd go a step back from "flawed data". I'd call it a bad sample size. See [Researchers have long known that whites are more likely to use and sell drugs. And yet, who is arrested for drugs? Blacks. 13% of the US population is black, but over 60% of those in prison are black.] So how would this happen? One possibility: if a crime happens more police frequent there. So you find more crime. Because we've over-sampled black neighborhoods (including DNA testing making it more likely to find relatives of someone already in prison) the mistaken belief becomes confirmed, and reinforced by the continued arrests. I'm unaware of how to call out bad sample sizes in data. However doing so may help highlight invalid assumptions.

danielvf 10y ago
You said "13% of the US population is black, but over 60% of those in prison are black." Blacks have a six times greater homicide rate than the rest of the US population, according to FBI extended homicide data. Almost half of homicides in the US in the last year I checked were by African Americans. With the long prison sentences given for violent crimes, violent crimes are disproportionately represented in prison populations. As a thought experiment, imagine that each year one person with a thirty year sentance was committed, and one person with a one year sentence. One the system stabled out, 97% of the prison population would be serving 30 year terms.
- Forge36 10y ago
  That's also very good to point out. Please note I quoted the article. You point out a bias in the statistic. Of the imprisoned population. What are the percentage breakdowns for types of crime?
  - danielvf 10y ago
    I did some research on this last year. Here's what I found: https://news.ycombinator.com/item?id=10318406 https://news.ycombinator.com/item?id=10318406
  - Nadya 10y ago

selectron 10y ago

This proposal shows more concern about being politically correct than actually correct. A truly unbiased system would just make the best analysis possible given the data available. What you propose is to make a biased system in order to give an advantage to one group of people rather than the other group. This type of political correctness fuels resentment.

aoki 10y ago
consider danah's example in which external studies show that blacks are convicted for drug crimes at a rate disproportionate to the actual crime incidence rate. the fact that the easily-available training data (court stats) is unrepresentative of the underlying process (crime incidence) is a problem that has to be corrected or predictive output will also over-represent them. the point is that this prior or correction is also part of the "data available" but you have to actually know something about the problem domain to know about it; you can't just grab a court data dump and pour it into a model and say, "welp, that was the data available to me, it's not biased."
cortesoft 10y ago
What they are saying is that, if you know your data is biased, you have the option of designing your system to compensate for that bias. Why do you seem to think that you have to just throw your hands up in the air and accept the bias? Wouldn't the 'best possible analysis given the data available' include compensating for known biases in the data? If I was designing a system for analyzing temperature data, for example, and I knew that every measurement was 10 degrees warmer than the actual temperature, shouldn't my analysis system compensate for that?
- ZoF 10y ago
  The issue here is how am I as a software developer supposed to combat that bias? Given the contrived arrest rate example above, do I multiply every black arrest by .7 to compensate for bias? How would a system built this way be useful in practice? Why are these parallels always drawn against racial divides as well? How are we compensating for the poor white kid missing out or the rich black kid getting benefits he might not need? It's such a messy problem but I don't think building bias into software is the solution. I prefer factually correct, biased systems to incorrect, still-biased-just-in-another-way systems.

kordless 10y ago

Except: "So long as man is approaching Ultimate Wisdom, he is still dwelling in the realm of consciousness. If he is to realise Nirvana, he must pass beyond the realm of consciousness. In highest Samadhi when consciousness has been transcended, he has passed beyond discrimination and knowledge, beyond any reach of change or fear. He is already enjoying Nirvana." - Sutra of Transcendental Wisdom

teacup50 10y ago

No. I'm not obligated to subscribe to your politics, and I won't coerce or bully others, either. Programmers are not the personal army of any social cause. I work hard to accurately model engineering problems, and apply consistent ethics in doing so. If that aligns with your definition of "combating prejudice", great for you, but I need the latitude to critically analyze technical problems without the threat of bullying and shaming for wrong-think. If I need to understand how surnames work across cultures, understand how time works across calendar systems, or understand what "marriage" means to the people getting married, then that's what I'll do. If I need to support non-binary 'gender' selections because that's what users want, that's what I'll do. On the other hand, if I need to support binary gender for the purposes of simplified data analysis under the assumption of two biological genders with a low probability of intersex individuals, that's what I'll do, too -- politely, and without rancor, and with all the respect for others I can muster.