4 ms·

Other than him being very outraged and claiming gender-based discrimination again and again, is there actually anything to suggest the algorithm is biased on th

Other than him being very outraged and claiming gender-based discrimination again and again, is there actually anything to suggest the algorithm is biased on the gender axis, as opposed to any other axis? Just to be clear: I am not dismissing the possibility that the algorithm (meaning the training data, really) is gender biased, it just isn't clear from what I have seen in the tweet storm that this is necessarily the case. E.g. I had an ex girlfriend who had a slightly lower income and slightly worse credit score (well, German Schufa score) than me, and yet she got offered like twice as credit when she applied for a card. I am guessing (just guessing) that this was due to her having paid off e.g. a car loan in the past and being generally more consumerist than me, while I never had taken out any major loans.

WA 7y ago
The whole thread is less about his wife having a lower credit limit, and more about Apple reps not understanding the fucking algorithm. The axis doesn’t really matter. Maybe it’s gender, maybe it’s not. In any case, it should be about money, credit score, trustworthiness. Apparently, it’s not. Being a women could be one indicator. It might be "being black" for other people. There seems to be something fishy going on and Apple doesn’t even know why. Hence, there is a bias.
- dkonofalski 7y ago
  >There seems to be something fishy going on and Apple doesn’t even know why That seems a little silly, though, since Goldman Sachs is the bank in charge of the card and approval process. Just like Amazon doesn't know anything about the processes that are used to determine credit worthiness for their cards (that falls to Chase and Synchrony), Apple is merely licensing services from GS. On top of that, GS representatives that work in a call center aren't going to know anything about the details or inner workings of the process either. With the amount of potential abuse for such information, it feels like it would be irresponsible for those people to have access to that info. The bigger issue is that this couple had a valid complaint and had to turn to Twitter to resolve it instead of GS having some formal method of requesting the info.
  - donarb 7y ago
    Although after this, I would think that this is a good opportunity for Apple to work with Goldman to better understand the algorithm and strive to make the process become more fair and transparent. Just like Apple how deals with 3rd parties making their devices where they can help enforce work and environmental rules.
    - dkonofalski 7y ago
      Definitely. And it sounds like this is exactly what's happening.
  - WA 7y ago
    DHH posted about this exact thing today. Basically: who cares? It has the Apple logo on it. Apple says: "It was created by Apple, not a bank." They are responsible and nobody else is to blame if the parts they outsource fail. https://mobile.twitter.com/dhh/status/1193716881351294976 https://mobile.twitter.com/dhh/status/1193716881351294976
    - dkonofalski 7y ago
      >who cares? Apple said they weren't aware of the details surrounding the decision and had the customer (DHH) work with Goldman Sachs to determine the details of the decision. I don't think that's unreasonable and it's not like Apple just wiped their hands of it and said "We don't know, not our problem." My response was directly responding to the person that said that Apple employees didn't know why the decision was made. Apple was attempting to get an answer for their customer. Especially with credit and finance information, this isn't something that would be acceptable for a low-ranking Apple employee to discuss with the customer and, if DHH is claiming that it should be because it's "Apple's card", then I think he's full of crap.
function_seven 7y ago
> is there actually anything to suggest the algorithm is biased on the gender axis, as opposed to any other axis? None of us are allowed to know that. That's the big problem here. They delegate the decision to a black box that cannot be questioned. When the process is set up like that, I think it's fair to assume the worst case scenarios and put the burden on the company to prove otherwise. Adverse inference is a sensible way to combat secret decisions like this.
- rndgermandude 7y ago
  >None of us are allowed to know that. That's the big problem here. They delegate the decision to a black box that cannot be questioned. The entire article is about questioning the black box, with regulatory force. Which is a good thing that is is going to be investigated. >When the process is set up like that, I think it's fair to assume the worst case scenarios Disagree. Consider it yes, but not assume it as a foregone conclusion. > and put the burden on the company to prove otherwise. Agree.
- xenihn 7y ago
  Black-box algorithms that affect your livelihood are bad, and I don't understand how anyone can be opposed to full transparency. I don't care what the context is. Whether its credit ratings, job applications, or college admissions.
- setpatchaddress 7y ago
  it’s notable that a lot of mid-20th-century fear surrounding the increased use of computers was centered around this exact scenario: a black box, the judgement of which cannot be questioned, deciding your fate, with no recourse. We all scoffed at this for a long time. ML makes it real, apparently.
jacquesm 7y ago
No, but neither is there anything that would justify giving a spouse such a very different limit especially considering that they file joint tax returns and her credit score is better than his (see article). It would seem that his conclusion is warranted absent evidence to the contrary, the difference is too large to explain in ways that make any sense.
- umanwizard 7y ago
  Maybe I am missing something, but wouldn't it make perfect sense if they have dramatically different income? I think even if two people have their property in common (and if the algorithm even knows about that), it is still not unreasonable to believe that there is a higher probability of the one with higher income paying off his or her loans.
  - jacquesm 7y ago
    I don't think DHH or his wife are a risk with respect to being able to pay off their loans. Any reasonably adept credit scoring algorithm should be able to pick up on that.
  - JumpCrisscross 7y ago
    > wouldn't it make perfect sense if they have dramatically different income? The implication in citing their joint filing status is they submitted identical incomes. They have the same address, assets, and she had a better credit score. She is also a woman.
    - rndgermandude 7y ago
      If she applied after him getting his application approved, this might very well explain her tiny line of credit: the algorithm didn't look at him and her alone in isolation, but at both at the same time, saw a certain combined income, saw he already has a large line of credit issued and concluded "I am not giving them another 3000 bucks or whatever, they already got that, so let it be 50 bucks or something, the minimum we can give to a customer, because really they together used all the credit we're willing to extend already when he got his card approved". If she applied first, she might have gotten the big chunk, and he might have ended up with the tiny chunk. Or it might be that the algorithm training data was just biased against women, which is entirely possible as well. The black-box-ness, even to employees, of course, is a huge problem.
thenewnewguy 7y ago
Well, it seems like both you and the twitter thread have anecdotal evidence. I'd say that the fact that an internal investigation is being launched leads credence to the belief that the discrimination is at least possible.
- nitwit005 7y ago
  Saying you're going to launch an internal investigation is often a way of calming a PR fire. For a question like this, people can often get an answer with a phone call or email.
  - ciabattabread 7y ago
    In this case, it’s a state government investigation.
sgerenser 7y ago
I haven't seen any actual evidence other than his single datapoint of him vs. his wife's experience. Of course its possible that the algorithm is designed to give lower credit limits to women, but I find it unlikely. My wife and I have applied for dozens of cards over the past 10+ years and I've consistently found that once you're score is above some reasonable threshold, the biggest predictor of what credit limit you will be offered are the credit limits on the OTHER credit cards reporting to your report. Maybe he already has much higher limits on other cards? I started using credit cards a few years before my wife, and despite us both having excellent credit the first time she started applying for cards she was getting limits around $1K or $2K while at that point I was around 10x that. But after a few years of getting more cards and requesting limit increases, both of us now have roughly the same (fairly ridiculous) limits across all of our cards. Admittedly I don't have any cards issued by Goldman Sachs bank, but I can't imagine their algorithm would be much different than Amex, Citi, Chase, etc.
ksec 7y ago
In the thread, it was pointed out that her wife actually has a better credit score than he did, and judging from the photo, it looks like ( I may be wrong ) that she only gets like $50 of credit limit. I had the same thought as you did when I first read it, for those of us not in US, it seems strange and we assume those Credit Score and Algorithm works as intended, like in your example of past loans and payment. And from experience they tend to be consistent and can be easily explained. In this case however it seems something is very wrong. I am still thinking and not sure if it is Apple to be blamed. It is easy to just point a finger to Apple, but in reality our financial and insurance system works pretty much the same way, changing these algorithm will require lots of work. Luckily GS is new to all these consumer business and changes are much easier as compared to other banks.
- dkonofalski 7y ago
  >it looks like ( I may be wrong ) that she only gets like $50 of credit limit. She actually got a higher limit than that but spent the majority of it. The complaint was that she had paid off the entire balance that she had spent already but the limit wasn't reset and wouldn't reset upon payment but would reset at the end of the billing cycle so she only had a $50 limit until that point.

WA 7y ago

The whole thread is less about his wife having a lower credit limit, and more about Apple reps not understanding the fucking algorithm. The axis doesn’t really matter. Maybe it’s gender, maybe it’s not. In any case, it should be about money, credit score, trustworthiness. Apparently, it’s not. Being a women could be one indicator. It might be "being black" for other people. There seems to be something fishy going on and Apple doesn’t even know why. Hence, there is a bias.

dkonofalski 7y ago
>There seems to be something fishy going on and Apple doesn’t even know why That seems a little silly, though, since Goldman Sachs is the bank in charge of the card and approval process. Just like Amazon doesn't know anything about the processes that are used to determine credit worthiness for their cards (that falls to Chase and Synchrony), Apple is merely licensing services from GS. On top of that, GS representatives that work in a call center aren't going to know anything about the details or inner workings of the process either. With the amount of potential abuse for such information, it feels like it would be irresponsible for those people to have access to that info. The bigger issue is that this couple had a valid complaint and had to turn to Twitter to resolve it instead of GS having some formal method of requesting the info.
- donarb 7y ago
  Although after this, I would think that this is a good opportunity for Apple to work with Goldman to better understand the algorithm and strive to make the process become more fair and transparent. Just like Apple how deals with 3rd parties making their devices where they can help enforce work and environmental rules.
  - dkonofalski 7y ago
    Definitely. And it sounds like this is exactly what's happening.
- WA 7y ago
  DHH posted about this exact thing today. Basically: who cares? It has the Apple logo on it. Apple says: "It was created by Apple, not a bank." They are responsible and nobody else is to blame if the parts they outsource fail. https://mobile.twitter.com/dhh/status/1193716881351294976 https://mobile.twitter.com/dhh/status/1193716881351294976
  - dkonofalski 7y ago
    >who cares? Apple said they weren't aware of the details surrounding the decision and had the customer (DHH) work with Goldman Sachs to determine the details of the decision. I don't think that's unreasonable and it's not like Apple just wiped their hands of it and said "We don't know, not our problem." My response was directly responding to the person that said that Apple employees didn't know why the decision was made. Apple was attempting to get an answer for their customer. Especially with credit and finance information, this isn't something that would be acceptable for a low-ranking Apple employee to discuss with the customer and, if DHH is claiming that it should be because it's "Apple's card", then I think he's full of crap.

function_seven 7y ago

> is there actually anything to suggest the algorithm is biased on the gender axis, as opposed to any other axis? None of us are allowed to know that. That's the big problem here. They delegate the decision to a black box that cannot be questioned. When the process is set up like that, I think it's fair to assume the worst case scenarios and put the burden on the company to prove otherwise. Adverse inference is a sensible way to combat secret decisions like this.

rndgermandude 7y ago
>None of us are allowed to know that. That's the big problem here. They delegate the decision to a black box that cannot be questioned. The entire article is about questioning the black box, with regulatory force. Which is a good thing that is is going to be investigated. >When the process is set up like that, I think it's fair to assume the worst case scenarios Disagree. Consider it yes, but not assume it as a foregone conclusion. > and put the burden on the company to prove otherwise. Agree.
xenihn 7y ago
Black-box algorithms that affect your livelihood are bad, and I don't understand how anyone can be opposed to full transparency. I don't care what the context is. Whether its credit ratings, job applications, or college admissions.
setpatchaddress 7y ago
it’s notable that a lot of mid-20th-century fear surrounding the increased use of computers was centered around this exact scenario: a black box, the judgement of which cannot be questioned, deciding your fate, with no recourse. We all scoffed at this for a long time. ML makes it real, apparently.

jacquesm 7y ago

No, but neither is there anything that would justify giving a spouse such a very different limit especially considering that they file joint tax returns and her credit score is better than his (see article). It would seem that his conclusion is warranted absent evidence to the contrary, the difference is too large to explain in ways that make any sense.

umanwizard 7y ago
Maybe I am missing something, but wouldn't it make perfect sense if they have dramatically different income? I think even if two people have their property in common (and if the algorithm even knows about that), it is still not unreasonable to believe that there is a higher probability of the one with higher income paying off his or her loans.
- jacquesm 7y ago
  I don't think DHH or his wife are a risk with respect to being able to pay off their loans. Any reasonably adept credit scoring algorithm should be able to pick up on that.
- JumpCrisscross 7y ago
  > wouldn't it make perfect sense if they have dramatically different income? The implication in citing their joint filing status is they submitted identical incomes. They have the same address, assets, and she had a better credit score. She is also a woman.
  - rndgermandude 7y ago
    If she applied after him getting his application approved, this might very well explain her tiny line of credit: the algorithm didn't look at him and her alone in isolation, but at both at the same time, saw a certain combined income, saw he already has a large line of credit issued and concluded "I am not giving them another 3000 bucks or whatever, they already got that, so let it be 50 bucks or something, the minimum we can give to a customer, because really they together used all the credit we're willing to extend already when he got his card approved". If she applied first, she might have gotten the big chunk, and he might have ended up with the tiny chunk. Or it might be that the algorithm training data was just biased against women, which is entirely possible as well. The black-box-ness, even to employees, of course, is a huge problem.

thenewnewguy 7y ago

Well, it seems like both you and the twitter thread have anecdotal evidence. I'd say that the fact that an internal investigation is being launched leads credence to the belief that the discrimination is at least possible.

nitwit005 7y ago
Saying you're going to launch an internal investigation is often a way of calming a PR fire. For a question like this, people can often get an answer with a phone call or email.
- ciabattabread 7y ago
  In this case, it’s a state government investigation.

sgerenser 7y ago

I haven't seen any actual evidence other than his single datapoint of him vs. his wife's experience. Of course its possible that the algorithm is designed to give lower credit limits to women, but I find it unlikely. My wife and I have applied for dozens of cards over the past 10+ years and I've consistently found that once you're score is above some reasonable threshold, the biggest predictor of what credit limit you will be offered are the credit limits on the OTHER credit cards reporting to your report. Maybe he already has much higher limits on other cards? I started using credit cards a few years before my wife, and despite us both having excellent credit the first time she started applying for cards she was getting limits around $1K or $2K while at that point I was around 10x that. But after a few years of getting more cards and requesting limit increases, both of us now have roughly the same (fairly ridiculous) limits across all of our cards. Admittedly I don't have any cards issued by Goldman Sachs bank, but I can't imagine their algorithm would be much different than Amex, Citi, Chase, etc.

ksec 7y ago

In the thread, it was pointed out that her wife actually has a better credit score than he did, and judging from the photo, it looks like ( I may be wrong ) that she only gets like $50 of credit limit. I had the same thought as you did when I first read it, for those of us not in US, it seems strange and we assume those Credit Score and Algorithm works as intended, like in your example of past loans and payment. And from experience they tend to be consistent and can be easily explained. In this case however it seems something is very wrong. I am still thinking and not sure if it is Apple to be blamed. It is easy to just point a finger to Apple, but in reality our financial and insurance system works pretty much the same way, changing these algorithm will require lots of work. Luckily GS is new to all these consumer business and changes are much easier as compared to other banks.

dkonofalski 7y ago
>it looks like ( I may be wrong ) that she only gets like $50 of credit limit. She actually got a higher limit than that but spent the majority of it. The complaint was that she had paid off the entire balance that she had spent already but the limit wasn't reset and wouldn't reset upon payment but would reset at the end of the billing cycle so she only had a $50 limit until that point.