5 ms·

Goldman faces probe after entrepreneur slams Apple Card algorithm in tweets

29 points by shakes 7y ago

felideon 7y ago
> David Heinemeier Hansson railed against the Apple Card for giving him 20 times the credit limit that his wife got Heh, railed. Good on DHH for using his platform and bringing light to an issue like this. Original thread on Twitter: https://twitter.com/dhh/status/1192540900393705474 https://twitter.com/dhh/status/1192540900393705474
- ryandrake 7y ago
  Went to the article to find all the other evidence, and no it pretty much boils down to your single quote. Some guy is not happy with his wife’s credit limit and went on a tweet-rant accusing sexism. No idea how credit limits are calculated but maybe his credit limit is higher not because he’s a man but because he’s an entrepreneur?
  - felideon 7y ago
    Sure. Someone on Twitter pointed out that GS is accustomed to catering to business accounts. (The article here also says it is GS' first credit card.) They, and presumably Apple moreso, should have considered this before using the biased algorithm for their first consumer credit card. They should have analyzed -who- would be affected by it.
    - deleted 7y ago
      [deleted]
    - FDSGSG 7y ago
      > should have considered this before using the biased algorithm for their first consumer credit card Why would such bias be bad when assigning a credit limit? Are you suggesting that this just isn't related to creditworthiness?
23B1 7y ago
What's the sample size here?
- ryanlol 7y ago
  Non-existent apparently.
- arcturus17 7y ago
  Enough to know a 20-to-1 difference is abhorrent, given they file joint tax returns and she has a better credit score?
  - derision 7y ago
    If you're married and you're filing jointly, it's really no different than the same individual applying twice in a row. They've already extended one credit line to a specific entity, so it makes perfect sense that the second line would either be denied or minimal. It baffles me that this has exploded so much
    - ryandrake 7y ago
      Surely tax filing status is not the only factor taken into account, nor is credit score. If I were a stay-at-home dad and my wife was a doctor, I’d expect her creditworthiness to be much more than mine, regardless of how we did our taxes. If both of us took out $100k in credit and then got divorced who is most likely to be able to pay it back?
      - derision 7y ago
        Theoretically both, because the lower earning spouse will be taking in alimony
  - 23B1 7y ago
    Oh don't get me wrong, I'd love to understand the algorithm that determines who gets how much credit. But we're assuming that it has to do with gender, and none of us actually know the difference between these two people historically. My wife has better credit than I do but I can borrow more because my income is much higher, for example.
ibero 7y ago
there are a lot of variables that go into determining a credit score. in this particular case, i could imagine that in his role as partner as Basecamp there is a longer "credit trail" attributed to him just from a business operational standpoint beyond his joint family tax filing. that alone could explain the discrepancy between him and his wife beyond sexism.
- felideon 7y ago
  Explaining the sexism away does not make it less sexist or defensible. The point is the algorithm has a sexist bias, as multiple anecdotes show.
  - buzzkillington 7y ago
    >The point is the algorithm has a sexist bias, as multiple anecdotes show. By definition anecdotes prove nothing. Until there is statistically significant evidence of sexism this is just hearsay and your post gets close to slander. In this case it will be absolutely trivial to check. Just feed in the same data for the same person only with the gender flipped.
    - bjtitus 7y ago
      The whole point of the original tweet was to point potential systemic gender bias in the algorithm. The argument here is that the black box algorithm may produce sexist results whether gender is included or not. While there may only be anecdotes at this point, there should be some investigation and discussion into whether bias may occur toward particular classes outside of explicit gender checks. Seems like that would be done by comparing the allowed limits with gender, race, etc. and considering why the algorithm may unnecessarily limit those classes on a broad scale.
  - FDSGSG 7y ago
    Could you share this compelling evidence that the algorithm has a sexist bias?
- zimpenfish 7y ago
richard_todd 7y ago
When I was looking through the twitter replies last night, multiple people claimed the opposite happened (the female got the higher limit). It’s sad that the journalist either didn’t look into it or preferred not to stray from the implication that the algorithm is sexist rather than just buggy.
- tom_mellior 7y ago
  When I was looking through the thread I didn't see this. Do you have links to tweets?
  - ryanlol 7y ago
    Here's what I could find in 5 minutes: https://twitter.com/Flkittie/status/1193725782788628480 https://twitter.com/Flkittie/status/1193725782788628480 https://twitter.com/HailMaryHumble/status/1193855011735908352 https://twitter.com/HailMaryHumble/status/119385501173590835... https://twitter.com/mindnotnull/status/1193794029189767168 https://twitter.com/mindnotnull/status/1193794029189767168 https://twitter.com/Chaubin123/status/1193729943701794823 https://twitter.com/Chaubin123/status/1193729943701794823 https://twitter.com/StrosniderDave/status/1193698022288510981 https://twitter.com/StrosniderDave/status/119369802228851098... https://twitter.com/ksgant/status/1193666460561879042 https://twitter.com/ksgant/status/1193666460561879042 https://twitter.com/fauxgressive/status/1193635810475683842 https://twitter.com/fauxgressive/status/1193635810475683842 https://twitter.com/snitstwits/status/1193626585821319168 https://twitter.com/snitstwits/status/1193626585821319168 https://twitter.com/chrishedlund/status/1193539000176455680 https://twitter.com/chrishedlund/status/1193539000176455680 https://twitter.com/Nukes4A/status/1193341783180283906 https://twitter.com/Nukes4A/status/1193341783180283906 https://twitter.com/Wayner83/status/1193332092291239936 https://twitter.com/Wayner83/status/1193332092291239936 https://twitter.com/JeffBessling/status/1193331529453363202 https://twitter.com/JeffBessling/status/1193331529453363202 https://twitter.com/bdruth/status/1193315200147054593 https://twitter.com/bdruth/status/1193315200147054593 https://twitter.com/Dcorbett818/status/1193282534622158848 https://twitter.com/Dcorbett818/status/1193282534622158848 https://twitter.com/NatalieEvansMD/status/1193269824169947136 https://twitter.com/NatalieEvansMD/status/119326982416994713... https://twitter.com/jetkatphoto/status/1193211571222142976 https://twitter.com/jetkatphoto/status/1193211571222142976 https://twitter.com/mbfuchik/status/1193143158504349697 https://twitter.com/mbfuchik/status/1193143158504349697 https://twitter.com/allenjs/status/1192959576477298688 https://twitter.com/allenjs/status/1192959576477298688 https://twitter.com/bstahlhood/status/1192816047117746178 https://twitter.com/bstahlhood/status/1192816047117746178 Of course, I'm just some "mansplaining" asshole. You should just ignore me.
fabian2k 7y ago
It's pretty much impossible to determine from the outside whether the algorithm discriminates against woman (without a reasonably large and actually random sample of customers). But that is also part of the complaint, as the algorithm is a black box that even the customer support employees can't look into. Humans just blindly following a black box algorithm without the option to manually handle cases where the algorithm produces weird results has quite a dystopian feel to me. Especially if it's used for more consequential cases than credit card limits.
- felideon 7y ago
  Hence the probe by DFS.
heavymark 7y ago
Confused. They are saying that nearly all married couples (male:female) with joint everything and the same credit history are showing consistently the husbands having more credit? Or just one single person out of millions? As one would assume there are probably many similar cases but where the wife got more credit no?
- FDSGSG 7y ago
  > Or just one single person out of millions This appears to be about a single loud person out of millions. I see mixed reports while searching on twitter, but nothing seems to suggest a bias in either direction.
- tom_mellior 7y ago
  When I read the Twitter thread yesterday there were plenty of people posting that they had the same observation: men getting more credit than their wives. I don't recall seeing anyone posting the opposite.
FDSGSG 7y ago
Honestly it just sounds like this guy hasn't applied for very many credit cards :) Sometimes you get a good limit, sometimes you don't.

29 points by shakes 7y ago

> David Heinemeier Hansson railed against the Apple Card for giving him 20 times the credit limit that his wife got Heh, railed. Good on DHH for using his platform and bringing light to an issue like this. Original thread on Twitter: https://twitter.com/dhh/status/1192540900393705474 https://twitter.com/dhh/status/1192540900393705474

ryandrake 7y ago
Went to the article to find all the other evidence, and no it pretty much boils down to your single quote. Some guy is not happy with his wife’s credit limit and went on a tweet-rant accusing sexism. No idea how credit limits are calculated but maybe his credit limit is higher not because he’s a man but because he’s an entrepreneur?
- felideon 7y ago
  Sure. Someone on Twitter pointed out that GS is accustomed to catering to business accounts. (The article here also says it is GS' first credit card.) They, and presumably Apple moreso, should have considered this before using the biased algorithm for their first consumer credit card. They should have analyzed -who- would be affected by it.
  - deleted 7y ago
    [deleted]
  - FDSGSG 7y ago
    > should have considered this before using the biased algorithm for their first consumer credit card Why would such bias be bad when assigning a credit limit? Are you suggesting that this just isn't related to creditworthiness?

23B1 7y ago

What's the sample size here?

ryanlol 7y ago
Non-existent apparently.
arcturus17 7y ago
Enough to know a 20-to-1 difference is abhorrent, given they file joint tax returns and she has a better credit score?
- derision 7y ago
  If you're married and you're filing jointly, it's really no different than the same individual applying twice in a row. They've already extended one credit line to a specific entity, so it makes perfect sense that the second line would either be denied or minimal. It baffles me that this has exploded so much
  - ryandrake 7y ago
    Surely tax filing status is not the only factor taken into account, nor is credit score. If I were a stay-at-home dad and my wife was a doctor, I’d expect her creditworthiness to be much more than mine, regardless of how we did our taxes. If both of us took out $100k in credit and then got divorced who is most likely to be able to pay it back?
    - derision 7y ago
      Theoretically both, because the lower earning spouse will be taking in alimony
- 23B1 7y ago
  Oh don't get me wrong, I'd love to understand the algorithm that determines who gets how much credit. But we're assuming that it has to do with gender, and none of us actually know the difference between these two people historically. My wife has better credit than I do but I can borrow more because my income is much higher, for example.

ibero 7y ago

there are a lot of variables that go into determining a credit score. in this particular case, i could imagine that in his role as partner as Basecamp there is a longer "credit trail" attributed to him just from a business operational standpoint beyond his joint family tax filing. that alone could explain the discrepancy between him and his wife beyond sexism.

felideon 7y ago
Explaining the sexism away does not make it less sexist or defensible. The point is the algorithm has a sexist bias, as multiple anecdotes show.
- buzzkillington 7y ago
  >The point is the algorithm has a sexist bias, as multiple anecdotes show. By definition anecdotes prove nothing. Until there is statistically significant evidence of sexism this is just hearsay and your post gets close to slander. In this case it will be absolutely trivial to check. Just feed in the same data for the same person only with the gender flipped.
  - bjtitus 7y ago
    The whole point of the original tweet was to point potential systemic gender bias in the algorithm. The argument here is that the black box algorithm may produce sexist results whether gender is included or not. While there may only be anecdotes at this point, there should be some investigation and discussion into whether bias may occur toward particular classes outside of explicit gender checks. Seems like that would be done by comparing the allowed limits with gender, race, etc. and considering why the algorithm may unnecessarily limit those classes on a broad scale.
- FDSGSG 7y ago
  Could you share this compelling evidence that the algorithm has a sexist bias?
zimpenfish 7y ago

richard_todd 7y ago

When I was looking through the twitter replies last night, multiple people claimed the opposite happened (the female got the higher limit). It’s sad that the journalist either didn’t look into it or preferred not to stray from the implication that the algorithm is sexist rather than just buggy.

tom_mellior 7y ago
When I was looking through the thread I didn't see this. Do you have links to tweets?
- ryanlol 7y ago
  Here's what I could find in 5 minutes: https://twitter.com/Flkittie/status/1193725782788628480 https://twitter.com/Flkittie/status/1193725782788628480 https://twitter.com/HailMaryHumble/status/1193855011735908352 https://twitter.com/HailMaryHumble/status/119385501173590835... https://twitter.com/mindnotnull/status/1193794029189767168 https://twitter.com/mindnotnull/status/1193794029189767168 https://twitter.com/Chaubin123/status/1193729943701794823 https://twitter.com/Chaubin123/status/1193729943701794823 https://twitter.com/StrosniderDave/status/1193698022288510981 https://twitter.com/StrosniderDave/status/119369802228851098... https://twitter.com/ksgant/status/1193666460561879042 https://twitter.com/ksgant/status/1193666460561879042 https://twitter.com/fauxgressive/status/1193635810475683842 https://twitter.com/fauxgressive/status/1193635810475683842 https://twitter.com/snitstwits/status/1193626585821319168 https://twitter.com/snitstwits/status/1193626585821319168 https://twitter.com/chrishedlund/status/1193539000176455680 https://twitter.com/chrishedlund/status/1193539000176455680 https://twitter.com/Nukes4A/status/1193341783180283906 https://twitter.com/Nukes4A/status/1193341783180283906 https://twitter.com/Wayner83/status/1193332092291239936 https://twitter.com/Wayner83/status/1193332092291239936 https://twitter.com/JeffBessling/status/1193331529453363202 https://twitter.com/JeffBessling/status/1193331529453363202 https://twitter.com/bdruth/status/1193315200147054593 https://twitter.com/bdruth/status/1193315200147054593 https://twitter.com/Dcorbett818/status/1193282534622158848 https://twitter.com/Dcorbett818/status/1193282534622158848 https://twitter.com/NatalieEvansMD/status/1193269824169947136 https://twitter.com/NatalieEvansMD/status/119326982416994713... https://twitter.com/jetkatphoto/status/1193211571222142976 https://twitter.com/jetkatphoto/status/1193211571222142976 https://twitter.com/mbfuchik/status/1193143158504349697 https://twitter.com/mbfuchik/status/1193143158504349697 https://twitter.com/allenjs/status/1192959576477298688 https://twitter.com/allenjs/status/1192959576477298688 https://twitter.com/bstahlhood/status/1192816047117746178 https://twitter.com/bstahlhood/status/1192816047117746178 Of course, I'm just some "mansplaining" asshole. You should just ignore me.

fabian2k 7y ago

It's pretty much impossible to determine from the outside whether the algorithm discriminates against woman (without a reasonably large and actually random sample of customers). But that is also part of the complaint, as the algorithm is a black box that even the customer support employees can't look into. Humans just blindly following a black box algorithm without the option to manually handle cases where the algorithm produces weird results has quite a dystopian feel to me. Especially if it's used for more consequential cases than credit card limits.

felideon 7y ago
Hence the probe by DFS.

heavymark 7y ago

Confused. They are saying that nearly all married couples (male:female) with joint everything and the same credit history are showing consistently the husbands having more credit? Or just one single person out of millions? As one would assume there are probably many similar cases but where the wife got more credit no?

FDSGSG 7y ago
> Or just one single person out of millions This appears to be about a single loud person out of millions. I see mixed reports while searching on twitter, but nothing seems to suggest a bias in either direction.
tom_mellior 7y ago
When I read the Twitter thread yesterday there were plenty of people posting that they had the same observation: men getting more credit than their wives. I don't recall seeing anyone posting the opposite.

FDSGSG 7y ago

Honestly it just sounds like this guy hasn't applied for very many credit cards :) Sometimes you get a good limit, sometimes you don't.