7 ms·
I think there's two different issues here. One, if datasets are biased --- if you build your system to only work on white males --- then it may have suboptimal
by mlyle 6y ago
I think there's two different issues here.
One, if datasets are biased --- if you build your system to only work on white males --- then it may have suboptimal results for other groups. This is a common problem: you use your company's faces, or college students enrolling in data gathering exercises, etc, who are not representative of the population at large. We can fix this by being careful about dataset bias.
But the second issue hits right at the heart of a major societal problem/debate. When we use AI to make decisions about people, will the system become racist--- even with representative datasets? If you train something to predict, say, odds to default on a loan-- will it figure out things that correlate to race and be making mostly racial decisions? Different races do have different default rates, but we've decided as a society that it is unfair to use race to determine an individual's probability of default. But if we choose things that are correlates of both default rates and race, when is that fair and when is it just veiled racism (redlining)? What things are measuring a causal relationship and what things are just racism in disguise?
This second problem is much worse with ML, because we have the ability to accept a whole bunch more things into our models and explaining the rationale of why decision are made is much harder.
And of course, the first problem-- both bias in datasets during use and biases during research and development -- makes the second problem worse. It can't even necessarily be addressed by broadening the dataset and retraining: if, in this case, you do your research and training with just white faces, and report positive results, it may not generalize to work as well for everyone with a broader dataset.
- mc32 6y agoIs that attenuated by making the system "blind" to race in those decision-making processes?
- mlyle 6y agoIt's pretty hard to make the system "blind" to race. If you blind a system predicting recidivism to race, but it is able to see a lot of things that are correlates of race... it can end up being racist anyways. E.g. for a non-ML example -- redlining is theoretically "blind" to race, but makes extremely racist decisions.
- compiler-guy 6y agoThat is difficult to do because there are many things that correlate with race. So even if you remove race explicitly, things like neighborhoods, level of education and such end up being proxies. Removing race from the model is a very hard problem.
- mlyle 6y agoOne example: Credit card issuers have been known to take into account places that you have made purchases in assessing risk. If you shop at businesses that are primarily frequented by minorities, odds are you'll get a lower credit limit than if you didn't.
- mc32 6y agoI guess the question is why would that be germane to issuing credit? Is it correlated to something that indicated actual credit risk? If it makes erroneous assumptions correlations, would it not behoove them to eliminate the contamination?
- mlyle 6y agoSure: some of your purchase history definitely indicates behaviors that indicate you're a credit risk. Some of your purchase history indicates you're likely black (and if you're black, your credit risk is, empirically, a little higher-- even if it is not fair to not issue credit to people because they're black). ML will figure out both but can't explain the hidden variables: it'll figure out behaviors that directly indicate higher risk, and behaviors that indicate you're a minority that tends to be a higher risk.
- mc32 6y agoIs it too entangled to say disregard anything that leads to a path that makes inferences based on demographics? Although I suppose age is also tough to avoid in assigning risk.
- 6y ago
- Natsu 6y agoIt's really, really hard to erase biases that are deeply systemic. For example, from the outset, would you object to an AI that made decisions on how harshly to sentence someone based on age & number of prior crimes? What about hiring or admitting people to college based on the results of an IQ test? Yeah, those end up being badly biased, this first has been studied and the second is the reason for dropping SAT/ACT scores--every mental ability test correlates with IQ. The more interesting thing, IMHO, is that the opposite is also helpful. For example, if you help all poor people equally, you help even the playing field by disproportionately helping out all disadvantaged groups.
- mc32 6y agoApplication of law should always have room for judicial discretion. Mandatory minimum sentences should not exist, but then you get outrage over some light sentence here people get all riled over about and want to impeach judges and all. Education is different. IQ shouldn’t be relevant but competency and aptitude should and in addition we should recognize some people are better off going to vocational school where they may do better economically during their lifetimes.
- gold_mustard 6y ago>For example, from the outset, would you object to an AI that made decisions on how harshly to sentence someone based on age & number of prior crimes? Yes. This is an overly simplistic system for which it would be hard to justify the cost of developing. >What about hiring or admitting people to college based on the results of an IQ test? Yes. For the same reason.
- Natsu 6y agoThe first one was intentionally minimalist in order to see how little it took for racial bias to be reconstructed. For the second, any mental ability test correlates with IQ, so you end up with a different set of difficulties. There have been attempts to, e.g. correct for cultural bias in the tests, but these actually made the problems worse. I'm not presently aware of anything that makes that situation better.
- perl4ever 6y agoAssuming good faith, I really think there is a gap in understanding why an "objective" viewpoint can be seen as blinkered or prejudiced. If you're looking at things that correlate, you've got a lot of choices. Saying that your data set is representative, your algorithms are correct, doesn't mean your choices of what correlations to use are objectively or morally correct. Suppose that you have a system that accurately tells you men have higher car insurance claims on average than women. The same system might also be able to tell you people with low credit scores have higher claims on average than people with high scores. Is it right or ok to use the first correlation and ignore the second? Or vice versa? Or maybe both of them are unacceptable? There's nothing objectively inherent in the correlations that tells you it's ok to charge men who are good credit risks the same as men who are bad credit risks. Or optimal economically! Those are independent and difficult questions.
- karpierz 6y agoThis is tangential to the original thread; the point isn't that the world is biased. Obama's face exists in the world. The issue is that the model is biased in a way that the world isn't.
- mlyle 6y agoBut it's completely on point, IMO: even if the model is biased against minorities in a way that the world actually is, it can be what we consider ethnically unfair. You're worse off lending in traditionally black neighborhoods; the question is, is it ethical to make a map of all of these traditionally black neighborhoods and refuse to lend there? A model that takes this into account would be biased in the same way the actual world is, but many would consider it deeply racist and unfair.
- karpierz 6y agoSure, but "there are racial disparities in this domain, and therefore in the data" and "there aren't racial disparities in this domain, but there are in the data" are very different problems. The latter is the topic of discussion and is much less ambiguous than the former.
- deleted 6y ago[deleted]
- tomp 6y agoYou’re missing the broader picture. This is not a problem with data, AI, algorithms or whatever. This is simply a mathematical issue that is fundamentally unsolvable. Imagine you have two groups of people that have different means and distributions of “input” (could be intelligence, wealth, interests, ...), and this input is somehow correlated with some output (insurance rate, criminality, university acceptance rate). Both input and output are one-dimensional. Then you only have two choices. (1) You “unify” the distribution and use it as joint input, resulting in one group (the one with the higher mean) getting “more” - many people think this is unfair. (2) Or you can subtract the means first and so make the distributions as equal as possible, which can result in a person from the “more” group having worse outcome than an equivalent person from the “less” group (if the means are close together and the distributions are wide, there will be many such people) - many people would consider this unfair. There are plenty of examples, current and historical, of society picking one or the other, for different groups and topics: wealth/success/income (old vs young, Jews&Asians vs other Americans, men vs women, whites vs blacks), university acceptance (Jews used to be discriminated against (look up “Jewish problems”), now Asians are, men vs women [1], “positive” discrimination of blacks) - just a few. In many examples historically, we consider (1) fundamentally better than (2) - e.g. discrimination against Jews at universities - but it seems it takes the society a long time to reach this conclusion in each instance. [1] https://en.wikipedia.org/wiki/Simpson%27s_paradox#UC_Berkeley_gender_bias https://en.wikipedia.org/wiki/Simpson%27s_paradox#UC_Berkele...
- mlyle 6y agoNo, indeed, I think that's the whole point. Approach #1 is to just go to maximize expected value, using all variables, including race. Approach #2 is to use race to adjust the distributions as you suggest. Call this "affirmative action." The inbetween approach #1.5, that we usually follow, is to make a value judgment about whether any individual item is OK to use. Income? Of course it's reasonable to use income to determine whether to lend-- even if it's correlated with race. Race? No, that seems wrong. Location? Sure--- wait, you've drawn a box around all the black neighborhoods, nope! Buys certain products? Sure-- wait, you don't mean "buys products that blacks stereotypically like," do you? #1.5 is already iffy, but becomes completely untenable once model complexity gets high and we use approaches that do not do well at explaining their decision: it basically becomes #1. A major problem with #1 is it becomes self-perpetuating and reinforcing: if everyone is going to be biased against you, you're going to do worse, and hence the models are validated/more models are created with these assumptions. Everyone using #1 may improve their own expected value at the expense of a worse outcome for society as a whole.
- ALittleLight 6y agoI don't really understand what it means to not use race as a criteria. If we're comparing Italians and Irish, and Irish are mostly left handed and Italians mostly right, and we know that right handers are more likely to pay back their loans, then presumably my race/ethnicity blind program will favor right handers over left handers, but the consequence will be favoring Italians over Irish. When more variables come up and get more complicated it seems like it would be impossible to say "You're based on race" versus "You're based on a multitude of factors, several of which correlate to race, and consequently you give more or better loans to race X over Y."
- mlyle 6y agoSee https://news.ycombinator.com/item?id=23698851 https://news.ycombinator.com/item?id=23698851
- andreig2191 6y ago>I don't really understand what it means to not use race as a criteria You find criteria that are not inherent to race. Reluctantly using your twisted example, imagine the left handed are more likely to default on loans because the checks they send are illegible. Then you could use the acceptance rate of written checks as your criteria instead of handedness.
- joyeuse6701 6y ago>But the second issue hits right at the heart of a major societal problem/debate. When we use AI to make decisions about people, will the system become racist--- even with representative datasets? Yeah this is tough, answering the question of the odds a person may default on a loan will probably reflect historical bias. In fact I'd argue that it should show this if the model is accurate. The moral issue is what do you do with that information? Do you use that knowledge to improve the outcomes of the disenfranchised or do you reinforce the bias based on the information you have?