4 ms·

This gets at the heart of one of the issues with these discrimination laws. What if, all else being equal, people from zipcode A are statistically much more li

This gets at the heart of one of the issues with these discrimination laws. What if, all else being equal, people from zipcode A are statistically much more likely to default than those from zipcode B? Do financial firms have to pretend like they don't know that fact? How removed from race does information have to be in order to be considered by a financial firm?

mindslight 7y ago
Isn't that a bit like asking how many pedestrians you can hit and still keep your license? The right answer is to try for zero. The dilemma you're describing isn't due to the law itself, but rather because the difficulty of writing law results in only the absolute worst abuses being criminalized. The abstract safe answer is to not engage in group-based discrimination at all, regardless of it seeming quite lucrative to hire a compliance department to analyze just how far you can get away with stretching it.
- dahfizz 7y ago
  > Isn't that a bit like asking how many pedestrians you can hit and still keep your license? This is nothing like that. Hitting or not hitting a pedestrian is binary, and there is a clear way to determine fault. We often call car collisions "accidents", but they aren't. Someone did something wrong to cause a collision. The issue here is that you can be discriminatory "by accident". You can be 100% race blind and have the best intentions in the world and work very hard to be equitable but still have the slightest bias due to the nature of the data. In fact, when poverty is correlated with race so strongly, I would argue it's impossible for a bank not to accidentally discriminate. A bank isn't going to lend you someone who is unlikely to pay it back - there's nothing racist about that. But they have had a disparate impact. It's a very sticky situation.
  - mindslight 7y ago
    > there is a clear way to determine fault Yes, but criminal intent is less clear. Someone can be declared at fault for hitting a pedestrian, yet not have been criminally negligent. And even if this happens a few times from really bad luck, it's reasonable that they'd get to keep their license [0]. > You can be 100% race blind and have the best intentions in the world ... bias due to the nature of the data The question is what data? Are you feeding things in that have a clear causal relationship with your desired result? Or are you inputting everything you can hoping to discover correlations ? The latter is essentially trying to suss out informal groups, and engaging in any group-based discrimination means you cannot claim to have "the best intentions" - regardless of whether the group can be named as a legally protected one or not. If say you're try to base mortgage underwriting on credit card purchase data, it's wholly disingenuous to claim that the resulting fallout is "accidental". [0] In a society where cars are de facto mandatory to get around. I will disclaim that this casual attitude is a large part of what keeps roads so hostile for everyone else, but it currently is what it is. Also historically we haven't had such a likely hidden factor as phone use.
jcranmer 7y ago
> How removed from race does information have to be in order to be considered by a financial firm? In general, the standard is "disparate impact"--if you accepted 80% of all white applicants but only 20% of black applicants, then you're probably liable for racial discrimination even if you were completely race-blind.
- AnthonyMouse 7y ago
  > In general, the standard is "disparate impact" That standard is unworkable. What happens when an idealistic charity goes into a homeless shelter in a black neighborhood with a high rate of drug use and helps them all fill out mortgage applications, so that 80% of the lender's white applicants are married with stable middle class jobs and 80% of the black applicants are single, homeless, unemployed and suffering from drug addiction? The institution evaluating the applications may not even be aware of why their applicant population skews that way.
  - jcranmer 7y ago
    There is a legal defense, where you can argue that the disparate impact is caused by practical business need instead of by implicit bias. An example (using gender instead of race) is that women have less upper body strength than men, on average, so they are far less likely to meet job requirements that require being able to carry 100 pounds of equipment on their backs. Even then, however, you still have to demonstrate in your defense that the requirements are actually justified and not a backdoor proxy. In the example I gave earlier, you have to demonstrate that employees actually need to carry 100 pounds of equipment on their backs, and furthermore that there is no workable alternative to that requirement. If I were a legal compliance officer, machine learning for applicant screening would scare me, because I would have a hard time arguing in court that the results constituted a legitimate business need and not racism-by-proxy, especially if the plaintiff found that some of the metrics highly correlated to race.
    - AnthonyMouse 7y ago
      Which is why that standard is unworkable. If you're giving loans and justifying a factor is the same as showing that it correlates with repayment rate then it's that easy to show a justification, but it will also basically always produce a "disparate impact" because repayment rate itself correlates with race, so any reasonably accurate measure of repayment rate will do the same. A "disparate impact" standard is useless because it's routinely met even when discrimination is not actually occurring. In fact, finding no disparate impact would be highly indicative that someone was impermissibly taking race into consideration in order to fudge the numbers. So either you have a disparate impact because that's what naturally happens with accurate predictions, or you do the expressly prohibited thing and take race directly into consideration in order to make it go away. Imagine a human intervening to remove a factor from consideration because considering it benefits black applicants relative to other applicants, even though considering it improves accuracy overall. Would they not rightfully get their butts sued off? But what happens if it's the same thing, only this time it benefits hispanic or korean or greek applicants relative to black applicants? The only remaining option is to consider all the information you have available, which is what people are inclined to do to begin with.

mindslight 7y ago

Isn't that a bit like asking how many pedestrians you can hit and still keep your license? The right answer is to try for zero. The dilemma you're describing isn't due to the law itself, but rather because the difficulty of writing law results in only the absolute worst abuses being criminalized. The abstract safe answer is to not engage in group-based discrimination at all, regardless of it seeming quite lucrative to hire a compliance department to analyze just how far you can get away with stretching it.

dahfizz 7y ago
> Isn't that a bit like asking how many pedestrians you can hit and still keep your license? This is nothing like that. Hitting or not hitting a pedestrian is binary, and there is a clear way to determine fault. We often call car collisions "accidents", but they aren't. Someone did something wrong to cause a collision. The issue here is that you can be discriminatory "by accident". You can be 100% race blind and have the best intentions in the world and work very hard to be equitable but still have the slightest bias due to the nature of the data. In fact, when poverty is correlated with race so strongly, I would argue it's impossible for a bank not to accidentally discriminate. A bank isn't going to lend you someone who is unlikely to pay it back - there's nothing racist about that. But they have had a disparate impact. It's a very sticky situation.
- mindslight 7y ago
  > there is a clear way to determine fault Yes, but criminal intent is less clear. Someone can be declared at fault for hitting a pedestrian, yet not have been criminally negligent. And even if this happens a few times from really bad luck, it's reasonable that they'd get to keep their license [0]. > You can be 100% race blind and have the best intentions in the world ... bias due to the nature of the data The question is what data? Are you feeding things in that have a clear causal relationship with your desired result? Or are you inputting everything you can hoping to discover correlations ? The latter is essentially trying to suss out informal groups, and engaging in any group-based discrimination means you cannot claim to have "the best intentions" - regardless of whether the group can be named as a legally protected one or not. If say you're try to base mortgage underwriting on credit card purchase data, it's wholly disingenuous to claim that the resulting fallout is "accidental". [0] In a society where cars are de facto mandatory to get around. I will disclaim that this casual attitude is a large part of what keeps roads so hostile for everyone else, but it currently is what it is. Also historically we haven't had such a likely hidden factor as phone use.

jcranmer 7y ago

> How removed from race does information have to be in order to be considered by a financial firm? In general, the standard is "disparate impact"--if you accepted 80% of all white applicants but only 20% of black applicants, then you're probably liable for racial discrimination even if you were completely race-blind.

AnthonyMouse 7y ago
> In general, the standard is "disparate impact" That standard is unworkable. What happens when an idealistic charity goes into a homeless shelter in a black neighborhood with a high rate of drug use and helps them all fill out mortgage applications, so that 80% of the lender's white applicants are married with stable middle class jobs and 80% of the black applicants are single, homeless, unemployed and suffering from drug addiction? The institution evaluating the applications may not even be aware of why their applicant population skews that way.
- jcranmer 7y ago
  There is a legal defense, where you can argue that the disparate impact is caused by practical business need instead of by implicit bias. An example (using gender instead of race) is that women have less upper body strength than men, on average, so they are far less likely to meet job requirements that require being able to carry 100 pounds of equipment on their backs. Even then, however, you still have to demonstrate in your defense that the requirements are actually justified and not a backdoor proxy. In the example I gave earlier, you have to demonstrate that employees actually need to carry 100 pounds of equipment on their backs, and furthermore that there is no workable alternative to that requirement. If I were a legal compliance officer, machine learning for applicant screening would scare me, because I would have a hard time arguing in court that the results constituted a legitimate business need and not racism-by-proxy, especially if the plaintiff found that some of the metrics highly correlated to race.
  - AnthonyMouse 7y ago
    Which is why that standard is unworkable. If you're giving loans and justifying a factor is the same as showing that it correlates with repayment rate then it's that easy to show a justification, but it will also basically always produce a "disparate impact" because repayment rate itself correlates with race, so any reasonably accurate measure of repayment rate will do the same. A "disparate impact" standard is useless because it's routinely met even when discrimination is not actually occurring. In fact, finding no disparate impact would be highly indicative that someone was impermissibly taking race into consideration in order to fudge the numbers. So either you have a disparate impact because that's what naturally happens with accurate predictions, or you do the expressly prohibited thing and take race directly into consideration in order to make it go away. Imagine a human intervening to remove a factor from consideration because considering it benefits black applicants relative to other applicants, even though considering it improves accuracy overall. Would they not rightfully get their butts sued off? But what happens if it's the same thing, only this time it benefits hispanic or korean or greek applicants relative to black applicants? The only remaining option is to consider all the information you have available, which is what people are inclined to do to begin with.