3 ms·

The Apple Card bias is a real problem. Who cares whether it's about gender or not? If it's not about gender, the responsibility lies on the bad actor to prove o

The Apple Card bias is a real problem. Who cares whether it's about gender or not? If it's not about gender, the responsibility lies on the bad actor to prove otherwise. The most logical conclusion - based on the events that occurred and the info the general public has - is that it's about gender. They could have defused the controversy almost immediately by identifying the actual cause and fixing it. Part of the reason the story blew up was that DHH and his wife struggled at length to even get an explanation for the problem or find a human who could fix it. The opacity of that whole machine and its resistance to correction is why it became a fuss. I honestly don't care whether it turns out to be about gender, because the end result of that system is that someone got discriminated against in a way that was indistinguishable from being gender discrimination. In the same fashion, if your pricing algorithm accidentally (due to training set bias or whatever) has a 20% higher chance to overcharge people of color, it doesn't really matter how it got that way. It's a racist algorithm and that doesn't mean you're a racist: It just means you have to fix it. Similarly, if the firings were carefully vetted and legally justified, they could have done a better job of explaining that rationale to the public. As a former Google employee the public rationale was obviously deceptive in some fashion, and undermines any suspicion I might have had regarding the employees in question. I worked with plenty of people at that company who probably deserved to be fired, but when the company behaves that way in public and selects people in a way that coincidentally undermines union organizing, I know who I'm going to suspect.

lawnchair_larry 7y ago
It is not even close to reasonable to conclude that the Apple Card thing had anything to do with gender. There is no reason to believe that the end result was because of this. Especially given all of the reports of women getting higher limits than their male spouse. This is almost as logical as concluding that vaccines cause autism.
- kevingadd 7y ago
  Why is gender discrimination equivalent to autism and vaccines here? There's lots of standing evidence for gender/race discrimination in finance and zero reliable evidence to support vaccines causing autism. You can come up with a better equivalence than that... People assumed gender discrimination because you had a couple with good credit scores and the woman was discriminated against by the algorithm. The only data we have indicates that it was gender. Of course there could be some other reason, but the obvious conclusion we have based on the information we've got is gender.
  - givinguflac 7y ago
    My wife has a 3x higher limit on her Apple Card and better credit than I. Gender has nothing to do with it, it's just a new hate on Apple train.
  - lawnchair_larry 7y ago
    It’s ridiculous because there is no evidence, just misinformed anecdotes. There is no field for gender on the application. The “equivalent” factors cited (income, credit score) are all irrelevant for credit limit decisions, which are primarily based on previous revolving credit usage. Given that DHH owns basecamp as an LLC, and not a corporation - known as a disregard entity for tax purposes - means he is likely putting all business expenses on revolving credit cards, which would create a massive disparity. That’s just one of many plausible explanations. DHH didn’t even understand that he is not in fact in a community property state - although this wouldn’t have been relevant either. And there are many counter-examples that our confirmation bias doesn’t seem to be interested in: https://mobile.twitter.com/jimcopus/status/1193733746614128640 https://mobile.twitter.com/jimcopus/status/11937337466141286... Furthermore, Goldman issued a public statement confirming that they reviewed the complaints and they are not using gender as a parameter. This would be trivial for an investigator to confirm, so it’s unlikely that they’d be stupid enough to outright lie about it. Jumping to the worst possible conclusion based on a twitter anecdote is not helpful to anybody.
BickNowstrom 7y ago
> The Apple Card bias is a real problem. Who cares whether it's about gender or not? If it's not about gender, the responsibility lies on the bad actor to prove otherwise. The most logical conclusion - based on the events that occurred and the info the general public has - is that it's about gender. And I work with credit risk models and can tell you there is a negligible chance that gender is causing a 20x credit limit increase. No protected variable will have so much influence, and Apple/GS are not even legally allowed to directly reference such variables. It may be the most logical conclusion for the general public, but all they see is the output of a black box. DHH jumped to conclusions that it had to be something about gender, with n=1. Who says there is a problem that needs fixing? Demanding that women and men always receive equal limits is not about fairness, it is a very radical notion that makes discriminative credit risk scoring almost impossible. I am not saying that Apple/GS did not mess up here, and could have been more transparent, but, in their defense: Oftentimes you can't be more transparent (like when Google closes an account for posting something illegal, they have to keep mum, while the user loudly complains online that they got banned for doing nothing), an help desk person is never going to be able to give an explanation for a model decision (this is usually a good thing), and DHH seemed to be barking up the wrong tree all together (what if the different limits were the result of a, completely legal, influencer program or randomized trial?). It really does matter how it got that way. If 20% of people of color vs. white people have a low income (a non-protected variable), I am perfectly allowed to use that variable to deny them a loan more often, even if the result is that 20% of people of color don't receive a loan. That's not a racist algorithm, however it would be racist if I start ignoring income of certain people as to achieve racial demographic parity in loans. Similarly, as this is an ongoing investigation now, and they probably anticipated that, Google was probably legally very limited in what they could disclose. I bet some people with all the knowledge of the data - and colleague snooping are shaking their heads at some of the public perception of this case as union-busting. The opposite of accepting a coincidence once in a while is viewing everything through the lens of intentional discrimination.
- TheCoelacanth 7y ago
  > It really does matter how it got that way. If 20% of people of color vs. white people have a low income (a non-protected variable), I am perfectly allowed to use that variable to deny them a loan more often, even if the result is that 20% of people of color don't receive a loan. That's not a racist algorithm, however it would be racist if I start ignoring income of certain people as to achieve racial demographic parity in loans. It's not that simple. The principle of "disparate impact" means that you can't, in general, make the decision in a way that adversely impacts people of color just because you didn't directly consider race in your decision. You can use income in your decision because considering income "is necessary to achieve one or more substantial, legitimate, non-discriminatory interests." Income is directly related to ability to repay a loan, so you need to consider it. You can't just use any old variable you want to. You can't deny people for a loan based on what genre of music they listen to, because that has no justifiable connection to whether they will repay a loan.
macinjosh 7y ago
> If it's not about gender, the responsibility lies on the bad actor to prove otherwise. How do we know they are bad actor if we don't yet know what really happened? Imposing guilt on someone until they prove themselves innocent is a bad way to run a society.

lawnchair_larry 7y ago

It is not even close to reasonable to conclude that the Apple Card thing had anything to do with gender. There is no reason to believe that the end result was because of this. Especially given all of the reports of women getting higher limits than their male spouse. This is almost as logical as concluding that vaccines cause autism.

kevingadd 7y ago
Why is gender discrimination equivalent to autism and vaccines here? There's lots of standing evidence for gender/race discrimination in finance and zero reliable evidence to support vaccines causing autism. You can come up with a better equivalence than that... People assumed gender discrimination because you had a couple with good credit scores and the woman was discriminated against by the algorithm. The only data we have indicates that it was gender. Of course there could be some other reason, but the obvious conclusion we have based on the information we've got is gender.
- givinguflac 7y ago
  My wife has a 3x higher limit on her Apple Card and better credit than I. Gender has nothing to do with it, it's just a new hate on Apple train.
- lawnchair_larry 7y ago
  It’s ridiculous because there is no evidence, just misinformed anecdotes. There is no field for gender on the application. The “equivalent” factors cited (income, credit score) are all irrelevant for credit limit decisions, which are primarily based on previous revolving credit usage. Given that DHH owns basecamp as an LLC, and not a corporation - known as a disregard entity for tax purposes - means he is likely putting all business expenses on revolving credit cards, which would create a massive disparity. That’s just one of many plausible explanations. DHH didn’t even understand that he is not in fact in a community property state - although this wouldn’t have been relevant either. And there are many counter-examples that our confirmation bias doesn’t seem to be interested in: https://mobile.twitter.com/jimcopus/status/1193733746614128640 https://mobile.twitter.com/jimcopus/status/11937337466141286... Furthermore, Goldman issued a public statement confirming that they reviewed the complaints and they are not using gender as a parameter. This would be trivial for an investigator to confirm, so it’s unlikely that they’d be stupid enough to outright lie about it. Jumping to the worst possible conclusion based on a twitter anecdote is not helpful to anybody.

BickNowstrom 7y ago

> The Apple Card bias is a real problem. Who cares whether it's about gender or not? If it's not about gender, the responsibility lies on the bad actor to prove otherwise. The most logical conclusion - based on the events that occurred and the info the general public has - is that it's about gender. And I work with credit risk models and can tell you there is a negligible chance that gender is causing a 20x credit limit increase. No protected variable will have so much influence, and Apple/GS are not even legally allowed to directly reference such variables. It may be the most logical conclusion for the general public, but all they see is the output of a black box. DHH jumped to conclusions that it had to be something about gender, with n=1. Who says there is a problem that needs fixing? Demanding that women and men always receive equal limits is not about fairness, it is a very radical notion that makes discriminative credit risk scoring almost impossible. I am not saying that Apple/GS did not mess up here, and could have been more transparent, but, in their defense: Oftentimes you can't be more transparent (like when Google closes an account for posting something illegal, they have to keep mum, while the user loudly complains online that they got banned for doing nothing), an help desk person is never going to be able to give an explanation for a model decision (this is usually a good thing), and DHH seemed to be barking up the wrong tree all together (what if the different limits were the result of a, completely legal, influencer program or randomized trial?). It really does matter how it got that way. If 20% of people of color vs. white people have a low income (a non-protected variable), I am perfectly allowed to use that variable to deny them a loan more often, even if the result is that 20% of people of color don't receive a loan. That's not a racist algorithm, however it would be racist if I start ignoring income of certain people as to achieve racial demographic parity in loans. Similarly, as this is an ongoing investigation now, and they probably anticipated that, Google was probably legally very limited in what they could disclose. I bet some people with all the knowledge of the data - and colleague snooping are shaking their heads at some of the public perception of this case as union-busting. The opposite of accepting a coincidence once in a while is viewing everything through the lens of intentional discrimination.

TheCoelacanth 7y ago
> It really does matter how it got that way. If 20% of people of color vs. white people have a low income (a non-protected variable), I am perfectly allowed to use that variable to deny them a loan more often, even if the result is that 20% of people of color don't receive a loan. That's not a racist algorithm, however it would be racist if I start ignoring income of certain people as to achieve racial demographic parity in loans. It's not that simple. The principle of "disparate impact" means that you can't, in general, make the decision in a way that adversely impacts people of color just because you didn't directly consider race in your decision. You can use income in your decision because considering income "is necessary to achieve one or more substantial, legitimate, non-discriminatory interests." Income is directly related to ability to repay a loan, so you need to consider it. You can't just use any old variable you want to. You can't deny people for a loan based on what genre of music they listen to, because that has no justifiable connection to whether they will repay a loan.

macinjosh 7y ago

> If it's not about gender, the responsibility lies on the bad actor to prove otherwise. How do we know they are bad actor if we don't yet know what really happened? Imposing guilt on someone until they prove themselves innocent is a bad way to run a society.