5 ms·

> It really does matter how it got that way. If 20% of people of color vs. white people have a low income (a non-protected variable), I am perfectly allowed to

> It really does matter how it got that way. If 20% of people of color vs. white people have a low income (a non-protected variable), I am perfectly allowed to use that variable to deny them a loan more often, even if the result is that 20% of people of color don't receive a loan. That's not a racist algorithm, however it would be racist if I start ignoring income of certain people as to achieve racial demographic parity in loans. It's not that simple. The principle of "disparate impact" means that you can't, in general, make the decision in a way that adversely impacts people of color just because you didn't directly consider race in your decision. You can use income in your decision because considering income "is necessary to achieve one or more substantial, legitimate, non-discriminatory interests." Income is directly related to ability to repay a loan, so you need to consider it. You can't just use any old variable you want to. You can't deny people for a loan based on what genre of music they listen to, because that has no justifiable connection to whether they will repay a loan.

BickNowstrom 7y ago
Everything, including income, is eventually correlated with race. The point I was making counters "does not matter how, just the outcome matters": The model disapproved you not because you are black, but because you have a low income. There is societal racism there that needs addressing with policy and regulation, not by handicapping your model by throwing away non-protected variables. The -perfectly legal-outcome will be that fewer people of color receive a loan. I know that you can't use just any old variable, but I tried enough to know that music genre would probably be an informative feature (dibs on providing loans to classical - and Judeo-Christian religious music lovers, you are free to underwrite the dubstep - and ghetto rap fans).
- TheCoelacanth 7y ago
  Music preferences undoubtedly would be correlated with risk. You still can't use them because of disparate impact. To get away with making a decision in a way that has disparate impact, you need to have a legitimate need to be making the decision that way. In the case of income, you can justify needing to use it in your decision, because income is directly connected to ability to repay. In the case of music preferences, you can't, because there is no direct connection to loan risk.

BickNowstrom 7y ago

Everything, including income, is eventually correlated with race. The point I was making counters "does not matter how, just the outcome matters": The model disapproved you not because you are black, but because you have a low income. There is societal racism there that needs addressing with policy and regulation, not by handicapping your model by throwing away non-protected variables. The -perfectly legal-outcome will be that fewer people of color receive a loan. I know that you can't use just any old variable, but I tried enough to know that music genre would probably be an informative feature (dibs on providing loans to classical - and Judeo-Christian religious music lovers, you are free to underwrite the dubstep - and ghetto rap fans).

TheCoelacanth 7y ago
Music preferences undoubtedly would be correlated with risk. You still can't use them because of disparate impact. To get away with making a decision in a way that has disparate impact, you need to have a legitimate need to be making the decision that way. In the case of income, you can justify needing to use it in your decision, because income is directly connected to ability to repay. In the case of music preferences, you can't, because there is no direct connection to loan risk.