4 ms·

> The Apple Card bias is a real problem. Who cares whether it's about gender or not? If it's not about gender, the responsibility lies on the bad actor to prove

> The Apple Card bias is a real problem. Who cares whether it's about gender or not? If it's not about gender, the responsibility lies on the bad actor to prove otherwise. The most logical conclusion - based on the events that occurred and the info the general public has - is that it's about gender. And I work with credit risk models and can tell you there is a negligible chance that gender is causing a 20x credit limit increase. No protected variable will have so much influence, and Apple/GS are not even legally allowed to directly reference such variables. It may be the most logical conclusion for the general public, but all they see is the output of a black box. DHH jumped to conclusions that it had to be something about gender, with n=1. Who says there is a problem that needs fixing? Demanding that women and men always receive equal limits is not about fairness, it is a very radical notion that makes discriminative credit risk scoring almost impossible. I am not saying that Apple/GS did not mess up here, and could have been more transparent, but, in their defense: Oftentimes you can't be more transparent (like when Google closes an account for posting something illegal, they have to keep mum, while the user loudly complains online that they got banned for doing nothing), an help desk person is never going to be able to give an explanation for a model decision (this is usually a good thing), and DHH seemed to be barking up the wrong tree all together (what if the different limits were the result of a, completely legal, influencer program or randomized trial?). It really does matter how it got that way. If 20% of people of color vs. white people have a low income (a non-protected variable), I am perfectly allowed to use that variable to deny them a loan more often, even if the result is that 20% of people of color don't receive a loan. That's not a racist algorithm, however it would be racist if I start ignoring income of certain people as to achieve racial demographic parity in loans. Similarly, as this is an ongoing investigation now, and they probably anticipated that, Google was probably legally very limited in what they could disclose. I bet some people with all the knowledge of the data - and colleague snooping are shaking their heads at some of the public perception of this case as union-busting. The opposite of accepting a coincidence once in a while is viewing everything through the lens of intentional discrimination.

TheCoelacanth 7y ago
> It really does matter how it got that way. If 20% of people of color vs. white people have a low income (a non-protected variable), I am perfectly allowed to use that variable to deny them a loan more often, even if the result is that 20% of people of color don't receive a loan. That's not a racist algorithm, however it would be racist if I start ignoring income of certain people as to achieve racial demographic parity in loans. It's not that simple. The principle of "disparate impact" means that you can't, in general, make the decision in a way that adversely impacts people of color just because you didn't directly consider race in your decision. You can use income in your decision because considering income "is necessary to achieve one or more substantial, legitimate, non-discriminatory interests." Income is directly related to ability to repay a loan, so you need to consider it. You can't just use any old variable you want to. You can't deny people for a loan based on what genre of music they listen to, because that has no justifiable connection to whether they will repay a loan.
- BickNowstrom 7y ago
  Everything, including income, is eventually correlated with race. The point I was making counters "does not matter how, just the outcome matters": The model disapproved you not because you are black, but because you have a low income. There is societal racism there that needs addressing with policy and regulation, not by handicapping your model by throwing away non-protected variables. The -perfectly legal-outcome will be that fewer people of color receive a loan. I know that you can't use just any old variable, but I tried enough to know that music genre would probably be an informative feature (dibs on providing loans to classical - and Judeo-Christian religious music lovers, you are free to underwrite the dubstep - and ghetto rap fans).
  - TheCoelacanth 7y ago
    Music preferences undoubtedly would be correlated with risk. You still can't use them because of disparate impact. To get away with making a decision in a way that has disparate impact, you need to have a legitimate need to be making the decision that way. In the case of income, you can justify needing to use it in your decision, because income is directly connected to ability to repay. In the case of music preferences, you can't, because there is no direct connection to loan risk.

TheCoelacanth 7y ago

> It really does matter how it got that way. If 20% of people of color vs. white people have a low income (a non-protected variable), I am perfectly allowed to use that variable to deny them a loan more often, even if the result is that 20% of people of color don't receive a loan. That's not a racist algorithm, however it would be racist if I start ignoring income of certain people as to achieve racial demographic parity in loans. It's not that simple. The principle of "disparate impact" means that you can't, in general, make the decision in a way that adversely impacts people of color just because you didn't directly consider race in your decision. You can use income in your decision because considering income "is necessary to achieve one or more substantial, legitimate, non-discriminatory interests." Income is directly related to ability to repay a loan, so you need to consider it. You can't just use any old variable you want to. You can't deny people for a loan based on what genre of music they listen to, because that has no justifiable connection to whether they will repay a loan.

BickNowstrom 7y ago
Everything, including income, is eventually correlated with race. The point I was making counters "does not matter how, just the outcome matters": The model disapproved you not because you are black, but because you have a low income. There is societal racism there that needs addressing with policy and regulation, not by handicapping your model by throwing away non-protected variables. The -perfectly legal-outcome will be that fewer people of color receive a loan. I know that you can't use just any old variable, but I tried enough to know that music genre would probably be an informative feature (dibs on providing loans to classical - and Judeo-Christian religious music lovers, you are free to underwrite the dubstep - and ghetto rap fans).
- TheCoelacanth 7y ago
  Music preferences undoubtedly would be correlated with risk. You still can't use them because of disparate impact. To get away with making a decision in a way that has disparate impact, you need to have a legitimate need to be making the decision that way. In the case of income, you can justify needing to use it in your decision, because income is directly connected to ability to repay. In the case of music preferences, you can't, because there is no direct connection to loan risk.