5 ms·

Let me try to clear your confusion: an input my seem innocent (e.g. zip code), but a zip code is likely to correlate to ethnicity and race in some regions. So e

Let me try to clear your confusion: an input my seem innocent (e.g. zip code), but a zip code is likely to correlate to ethnicity and race in some regions. So even if the inputs seem legal, an ML model that’s sophisticated enough can derive illegal results that discriminate against certain populations.

mieseratte 7y ago
Are there any real-world examples of an ML system causing protected-class discrimination based on non-protected criteria?
- Arcsech 7y ago
  Yes, for one example, a system used to recommend sentencing based on how likely a criminal was to re-offend was found to have been heavily biased based on race, despite not being given explicit information about race[1]. Edit: It’s unclear if this system is based on ML, but I think the point stands - if a system can be manually tuned to do this, the risk exists of this happening to a trained model as well. Given other cases of ML models learning an incorrect behavior (such as the system intended to detect skin cancer that ended up being a fancy ruler detector [2]). [1]: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm... [2]: https://www.thedailybeast.com/why-doctors-arent-afraid-of-better-more-efficient-ai-diagnosing-cancer https://www.thedailybeast.com/why-doctors-arent-afraid-of-be...
  - harryh 7y ago
    That is a fine answer to mieseratte's question but I feel compelled to mention that the propublica article you link to is...pretty bad. Here's a pretty good summary of why: https://www.chrisstucchio.com/blog/2016/propublica_is_lying.html https://www.chrisstucchio.com/blog/2016/propublica_is_lying....
- AmericanChopper 7y ago
  Using zip codes to implement discriminatory policies isn’t ML, but has almost certainly occurred. It’s referred to as redlining, and has been around for a while. The New Deal had some pretty indefensible redlining conditions. A lot of people will say the practice continues to this day, but the modern examples are a lot more open to interpretation than some of the historical ones are.
  - mieseratte 7y ago
    Was the New Deal redlining out-and-out discrimination or a matter of accidentally creating conditions, i.e. based on bad intelligence, that resulted in discrimination. I always understood it to be the former.
    - AmericanChopper 7y ago
      The level of plausible deniability regarding The New Deal is mostly a matter of opinion, but personally I think it was pretty transparent in how it intentionally discriminated. It’s the modern examples that I think are much more questionable. For example, there are more liquor stores in black neighbourhoods. Is this because: a) A conspiracy to use alcohol to suppress the black population? b) Those neighbourhoods have a greater demand for liquor stores?
- jetrink 7y ago
  A recruiting tool used by Amazon developed a bias against women despite not being told candidates' genders. It penalized candidates who were graduates of all-women's colleges and also those who had the word "women" in their resume (e.g. “women’s chess club captain.”) It had been trained on resumes submitted to Amazon during the previous ten years, so the tool's bias was likely reflective of real human bias in Amazon's recruiting process. 1. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G https://www.reuters.com/article/us-amazon-com-jobs-automatio...
  - mieseratte 7y ago
    I think that’s a great example on needing to sanitize data, but being a member of a women’s club or attending a women’s college is itself telling you the candidates gender so I don’t know that this is deriving protected information from non-protected information.
    - jetrink 7y ago
      This is a game of whack-a-mole. Given enough data, the ML system will just find other characteristics or groups of characteristics that act as proxies. It might unfairly penalize candidates in ways that are impossible to detect by human evaluators. The promise of these systems is that if you give them a pile of raw data, they will detect subtle patterns that aid in assessing individuals (be they job applicants, ex-cons, whatever.) However, some of the patterns that they detect are our biases against classes of people. If the solution is to sanitize the data to such a degree that the ML system can no longer infer that someone is a woman or that someone is over 40, etc., then the training data is probably also useless for detecting the non-obvious patterns that we want the system to discover.
      - mieseratte 7y ago
        That it is a “game of whack-a-mole” isn’t clear to me, and you are asserting so without any backing, which is why I asked if anyone has real-world examples. Citing an example where they don’t provide gender but then provide “went to an all-women college” is not an example of proving non-protected information which results in protected-class discrimination, it is an example of telling an ML system protected information in a roundabout way.
dahfizz 7y ago
This gets at the heart of one of the issues with these discrimination laws. What if, all else being equal, people from zipcode A are statistically much more likely to default than those from zipcode B? Do financial firms have to pretend like they don't know that fact? How removed from race does information have to be in order to be considered by a financial firm?
- mindslight 7y ago
  Isn't that a bit like asking how many pedestrians you can hit and still keep your license? The right answer is to try for zero. The dilemma you're describing isn't due to the law itself, but rather because the difficulty of writing law results in only the absolute worst abuses being criminalized. The abstract safe answer is to not engage in group-based discrimination at all, regardless of it seeming quite lucrative to hire a compliance department to analyze just how far you can get away with stretching it.
  - dahfizz 7y ago
    > Isn't that a bit like asking how many pedestrians you can hit and still keep your license? This is nothing like that. Hitting or not hitting a pedestrian is binary, and there is a clear way to determine fault. We often call car collisions "accidents", but they aren't. Someone did something wrong to cause a collision. The issue here is that you can be discriminatory "by accident". You can be 100% race blind and have the best intentions in the world and work very hard to be equitable but still have the slightest bias due to the nature of the data. In fact, when poverty is correlated with race so strongly, I would argue it's impossible for a bank not to accidentally discriminate. A bank isn't going to lend you someone who is unlikely to pay it back - there's nothing racist about that. But they have had a disparate impact. It's a very sticky situation.
    - mindslight 7y ago
      > there is a clear way to determine fault Yes, but criminal intent is less clear. Someone can be declared at fault for hitting a pedestrian, yet not have been criminally negligent. And even if this happens a few times from really bad luck, it's reasonable that they'd get to keep their license [0]. > You can be 100% race blind and have the best intentions in the world ... bias due to the nature of the data The question is what data? Are you feeding things in that have a clear causal relationship with your desired result? Or are you inputting everything you can hoping to discover correlations ? The latter is essentially trying to suss out informal groups, and engaging in any group-based discrimination means you cannot claim to have "the best intentions" - regardless of whether the group can be named as a legally protected one or not. If say you're try to base mortgage underwriting on credit card purchase data, it's wholly disingenuous to claim that the resulting fallout is "accidental". [0] In a society where cars are de facto mandatory to get around. I will disclaim that this casual attitude is a large part of what keeps roads so hostile for everyone else, but it currently is what it is. Also historically we haven't had such a likely hidden factor as phone use.
AnthonyMouse 7y ago
> So even if the inputs seem legal, an ML model that’s sophisticated enough can derive illegal results that discriminate against certain populations. That doesn't make any sense. The old school discrimination (e.g. redlining) worked like this. They would find some factor that correlates strongly with black people (e.g. black neighborhood zip codes), then assign a weight for that factor based on how well it correlated with what they wanted to discriminate against (black people) rather than how well it correlated with what they were supposed to be measuring (creditworthiness). You can certainly do that on purpose with ML, but the way you do it is to give the algorithm the data on which factors are associated with black people and then ask it to assign blackness scores rather than credit scores and use the blackness scores for making credit approvals. I am not aware of anybody stupid/racist enough to actually be doing that in 2019. What you're supposed to do is to weight factors based on how well they correlate with the outcome you're trying to predict (e.g. loan repayment) and use those weights rather than the ones chosen purposely to discriminate on the basis of race. That doesn't mean none of those factors will ever correlate with race. Everything correlates with everything to one degree or another. True independent variables are the exception rather than the rule. But weighting each factor based on how well it correlates with the outcome you're trying to predict rather than how well it correlates with race is maximally non-discriminatory -- doing something else would be purposely giving advantage to one race over another disproportionate to the best available information. And nobody who is not an overt racist has reason to fudge the numbers that way, because it would also make worse predictions and cause you to lose money.
dd36 7y ago
I mentioned corollaries... there’s no reason zip code or lat long should be an input.

mieseratte 7y ago

Are there any real-world examples of an ML system causing protected-class discrimination based on non-protected criteria?

Arcsech 7y ago
Yes, for one example, a system used to recommend sentencing based on how likely a criminal was to re-offend was found to have been heavily biased based on race, despite not being given explicit information about race[1]. Edit: It’s unclear if this system is based on ML, but I think the point stands - if a system can be manually tuned to do this, the risk exists of this happening to a trained model as well. Given other cases of ML models learning an incorrect behavior (such as the system intended to detect skin cancer that ended up being a fancy ruler detector [2]). [1]: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm... [2]: https://www.thedailybeast.com/why-doctors-arent-afraid-of-better-more-efficient-ai-diagnosing-cancer https://www.thedailybeast.com/why-doctors-arent-afraid-of-be...
- harryh 7y ago
  That is a fine answer to mieseratte's question but I feel compelled to mention that the propublica article you link to is...pretty bad. Here's a pretty good summary of why: https://www.chrisstucchio.com/blog/2016/propublica_is_lying.html https://www.chrisstucchio.com/blog/2016/propublica_is_lying....
AmericanChopper 7y ago
Using zip codes to implement discriminatory policies isn’t ML, but has almost certainly occurred. It’s referred to as redlining, and has been around for a while. The New Deal had some pretty indefensible redlining conditions. A lot of people will say the practice continues to this day, but the modern examples are a lot more open to interpretation than some of the historical ones are.
- mieseratte 7y ago
  Was the New Deal redlining out-and-out discrimination or a matter of accidentally creating conditions, i.e. based on bad intelligence, that resulted in discrimination. I always understood it to be the former.
  - AmericanChopper 7y ago
    The level of plausible deniability regarding The New Deal is mostly a matter of opinion, but personally I think it was pretty transparent in how it intentionally discriminated. It’s the modern examples that I think are much more questionable. For example, there are more liquor stores in black neighbourhoods. Is this because: a) A conspiracy to use alcohol to suppress the black population? b) Those neighbourhoods have a greater demand for liquor stores?
jetrink 7y ago
A recruiting tool used by Amazon developed a bias against women despite not being told candidates' genders. It penalized candidates who were graduates of all-women's colleges and also those who had the word "women" in their resume (e.g. “women’s chess club captain.”) It had been trained on resumes submitted to Amazon during the previous ten years, so the tool's bias was likely reflective of real human bias in Amazon's recruiting process. 1. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G https://www.reuters.com/article/us-amazon-com-jobs-automatio...
- mieseratte 7y ago
  I think that’s a great example on needing to sanitize data, but being a member of a women’s club or attending a women’s college is itself telling you the candidates gender so I don’t know that this is deriving protected information from non-protected information.
  - jetrink 7y ago
    This is a game of whack-a-mole. Given enough data, the ML system will just find other characteristics or groups of characteristics that act as proxies. It might unfairly penalize candidates in ways that are impossible to detect by human evaluators. The promise of these systems is that if you give them a pile of raw data, they will detect subtle patterns that aid in assessing individuals (be they job applicants, ex-cons, whatever.) However, some of the patterns that they detect are our biases against classes of people. If the solution is to sanitize the data to such a degree that the ML system can no longer infer that someone is a woman or that someone is over 40, etc., then the training data is probably also useless for detecting the non-obvious patterns that we want the system to discover.
    - mieseratte 7y ago
      That it is a “game of whack-a-mole” isn’t clear to me, and you are asserting so without any backing, which is why I asked if anyone has real-world examples. Citing an example where they don’t provide gender but then provide “went to an all-women college” is not an example of proving non-protected information which results in protected-class discrimination, it is an example of telling an ML system protected information in a roundabout way.

dahfizz 7y ago

This gets at the heart of one of the issues with these discrimination laws. What if, all else being equal, people from zipcode A are statistically much more likely to default than those from zipcode B? Do financial firms have to pretend like they don't know that fact? How removed from race does information have to be in order to be considered by a financial firm?

mindslight 7y ago
Isn't that a bit like asking how many pedestrians you can hit and still keep your license? The right answer is to try for zero. The dilemma you're describing isn't due to the law itself, but rather because the difficulty of writing law results in only the absolute worst abuses being criminalized. The abstract safe answer is to not engage in group-based discrimination at all, regardless of it seeming quite lucrative to hire a compliance department to analyze just how far you can get away with stretching it.
- dahfizz 7y ago
  > Isn't that a bit like asking how many pedestrians you can hit and still keep your license? This is nothing like that. Hitting or not hitting a pedestrian is binary, and there is a clear way to determine fault. We often call car collisions "accidents", but they aren't. Someone did something wrong to cause a collision. The issue here is that you can be discriminatory "by accident". You can be 100% race blind and have the best intentions in the world and work very hard to be equitable but still have the slightest bias due to the nature of the data. In fact, when poverty is correlated with race so strongly, I would argue it's impossible for a bank not to accidentally discriminate. A bank isn't going to lend you someone who is unlikely to pay it back - there's nothing racist about that. But they have had a disparate impact. It's a very sticky situation.
  - mindslight 7y ago
    > there is a clear way to determine fault Yes, but criminal intent is less clear. Someone can be declared at fault for hitting a pedestrian, yet not have been criminally negligent. And even if this happens a few times from really bad luck, it's reasonable that they'd get to keep their license [0]. > You can be 100% race blind and have the best intentions in the world ... bias due to the nature of the data The question is what data? Are you feeding things in that have a clear causal relationship with your desired result? Or are you inputting everything you can hoping to discover correlations ? The latter is essentially trying to suss out informal groups, and engaging in any group-based discrimination means you cannot claim to have "the best intentions" - regardless of whether the group can be named as a legally protected one or not. If say you're try to base mortgage underwriting on credit card purchase data, it's wholly disingenuous to claim that the resulting fallout is "accidental". [0] In a society where cars are de facto mandatory to get around. I will disclaim that this casual attitude is a large part of what keeps roads so hostile for everyone else, but it currently is what it is. Also historically we haven't had such a likely hidden factor as phone use.

AnthonyMouse 7y ago

> So even if the inputs seem legal, an ML model that’s sophisticated enough can derive illegal results that discriminate against certain populations. That doesn't make any sense. The old school discrimination (e.g. redlining) worked like this. They would find some factor that correlates strongly with black people (e.g. black neighborhood zip codes), then assign a weight for that factor based on how well it correlated with what they wanted to discriminate against (black people) rather than how well it correlated with what they were supposed to be measuring (creditworthiness). You can certainly do that on purpose with ML, but the way you do it is to give the algorithm the data on which factors are associated with black people and then ask it to assign blackness scores rather than credit scores and use the blackness scores for making credit approvals. I am not aware of anybody stupid/racist enough to actually be doing that in 2019. What you're supposed to do is to weight factors based on how well they correlate with the outcome you're trying to predict (e.g. loan repayment) and use those weights rather than the ones chosen purposely to discriminate on the basis of race. That doesn't mean none of those factors will ever correlate with race. Everything correlates with everything to one degree or another. True independent variables are the exception rather than the rule. But weighting each factor based on how well it correlates with the outcome you're trying to predict rather than how well it correlates with race is maximally non-discriminatory -- doing something else would be purposely giving advantage to one race over another disproportionate to the best available information. And nobody who is not an overt racist has reason to fudge the numbers that way, because it would also make worse predictions and cause you to lose money.

dd36 7y ago

I mentioned corollaries... there’s no reason zip code or lat long should be an input.