4 ms·

I guess it worked for him to throw a hissy fit, but there's absolutely no reason to actually believe they're conditioning on gender. It could be "it's a new pr

I guess it worked for him to throw a hissy fit, but there's absolutely no reason to actually believe they're conditioning on gender. It could be "it's a new product, we randomly assign credit limits to see how it affects behavior". It could be "it's a community property state and we're overexposed to this household if we give the second card the same limit as the first". It could, realistically, be almost anything except for evil bankers deciding to use illegal criteria to underwrite that has a side effect of limiting the amount that can be charged to the account each month (you know, how they actually make money). Oh, random CSRs don't get a pithy explanation of a multivariate nonlinear underwriting decision to poorly convey to customers? That's kind of precedented!

tenpies 7y ago
I mean you know the system is sexist when one gender out-lives the other by a considerable number of years yet the retirement age for both is the same.
elvinyung 7y ago
I mean, don't you think it's kind of a strawman to assume anyone thinks machines are explicitly sexist? It seems much more likely that whatever method they use to do feature selection on this obviously high-dimensional data just happens to end up picking something that was a proxy for gender. You can be implicitly biased without being aware of it. This is true for both humans and algorithms.
- kyrieeschaton 7y ago
  It is absolutely not a strawman. It is incredibly easy to find people granted authority by the state that will claim that, eg, mathematics per se is an instance of exclusionist masculine thinking. It's even easier to find instances where people granted authority by the state will claim that any discrepancy in outcomes is ipso facto intentional.
- gwd 7y ago
  > It seems much more likely that whatever method they use to do feature selection on this obviously high-dimensional data just happens to end up picking something that was a proxy for gender. Yes, this is exactly what everyone thinks happened. Nobody thinks people at Goldman-Sachs wrote if (applicant.gender == "F") limit /= 20 somewhere in their algorithm. But regardless of how the thing happened, if millions of people are treated significantly differently for no reason other than their plumbing, that's a major problem. People have been talking about this "accidental proxy for gender" for years now; there's absolutely no excuse for no doing a basic sanity check to make sure that this kind of thing isn't happening. edit: typo
  - kyrieeschaton 7y ago
    "for no other reason than their plumbing" is exactly what you just excluded in the prior sentence.
jyrkesh 7y ago
I've heard anecdotally a number of times that actuaries peg men with higher car insurance rates. I'm only just learning that there's anything like this in the credit system to prevent gender bias. Does something like that protection exist for auto insurance? As a side note, Idon't really understand the thread's appeal to credit scores here, considering that the TransUnion rating system is supposed to be the inferior one that Apple is looking to replace. But most of all, I'm shocked by the number of people here outraged by Apple's behavior who will not even be considering switching off their iPhone (Twitter OP included, hell, he even posted a screenshot of his recurring TransUnion payment still being served via Apple Pay). I actually happen to agree with OP, but this blaise attitude towards real customer complaints has kept me from using Apple products for years.
- dijit 7y ago
  For car insurance? This is incredibly well known and not at all anecdotal. There was even a popular car insurance firm in the UK that only gave women car insurance if they were women called “Sheilas Wheels” which touted lower fees than “other insurance companies that have to also cater to men”. https://youtu.be/GzNJh1o84-E https://youtu.be/GzNJh1o84-E
  - TAForObvReasons 7y ago
    Even though it wasn't a protected class, GEICO (Government Employees Insurance Company) originally started as an auto insurance company specifically for federal government employees, a pool perceived to be less risky than the driving population
  - tialaramex 7y ago
    It's also now illegal for insurers to do this in the EU since the ECJ reasoned that discriminating on gender, even if as a result of such a correlation, was prohibited. Anecdotally I've been told that (British car) insurers don't care very much about your real actuarial risk, they're focused more on whether you'll actually pay their premiums. Specifically I was told that work to integrate with credit checking services was a priority whereas an integration with the UK Government's service which gives them access to driving offences and other records related to a driving license was back-burnered. The reason I was given was that in practice they'd found if you require drivers to give their license details, it causes a big drop in purchases, if you make the license details _optional_ lots of people fill them in, and you can just give all those people a better price even though you don't use the details to actually check anything automatically.
burnte 7y ago
> I guess it worked for him to throw a hissy fit, but there's absolutely no reason to actually believe they're conditioning on gender. There certainly is. Ignoring the anecdotal data of people replying to him who saw the same outcome, credit scores for women skew lower than for men. One of many sources: https://www.federalreserve.gov/econres/notes/feds-notes/gender-related-differences-in-credit-use-and-credit-scores-20180622.htm https://www.federalreserve.gov/econres/notes/feds-notes/gend...
- zamalek 7y ago
  > credit scores for women skew lower than for men. Which is crazy, because for most of my guy friends (myself included), saying that their significant others/women make significantly better financial decisions is putting it far too lightly. I basically wasn't making decisions at all (besides savings) until my s/o set me right. This discrimination is tragic and disgusting.
  - DoreenMichele 7y ago
    Women tend to have lower incomes and lower net worths. Here is a recent article squeeing about women finally being willing to buy homes on their own: https://realestate.usnews.com/real-estate/articles/the-rise-of-the-single-female-homeowner https://realestate.usnews.com/real-estate/articles/the-rise-... One of the statistics in it: Single women can only afford about 39% of homes. Single men can afford more than half. Historically, couples with traditional marriages (primary breadwinner husband, wife whose primary responsibilities were women's work) often did not bother to put her name on a real estate transaction or car purchase. This substantially impairs a woman's ability to establish a credit record and reduces her legal claim to assets. If you get divorced, the person with a real career and resume to match will likely continue having a good income. Someone who was a full-time wife and mom will face serious barriers to establishing a real career at all and may well remain poor for years to come. I'm quite financially savvy. Financial savvy only goes so far. Ability to pay still matters and men are more likely to have that piece covered.
    - acjohnson55 7y ago
notadoc 7y ago
> I guess it worked for him to throw a hissy fit, but there's absolutely no reason to actually believe they're conditioning on gender. Of course not, but why spoil the outrage mobs victimhood narrative du jour?
dictum 7y ago
I would love a customer-company relationship in which the customer gets meaningful information by privately contacting the company with a polite message, but in my experience this is rare. When an answer comes, it tends to misdirect rather than tell the whole truth. Here you have a bank and a company known for secrecy. But you acknowledge this; > random CSRs don't get a pithy explanation What the throwers of hissy fits are pointing to is that, by building blackboxes, you can get whatever result you want (which doesn't mean all results are expected) with plausible deniability.
- r00fus 7y ago
  Perhaps because this exact approach of a candid private question can be abused to support litigation or corporate espionage.
reaperducer 7y ago
It could, realistically, be almost anything except for evil bankers deciding to use illegal criteria FTA: “My belief isn’t there was some nefarious person wanting to discriminate. But that doesn’t matter. How do you know there isn’t an issue with the machine-learning algo when no one can explain how this decision was made?”
- ndzig 7y ago
  They can explain, they most likely don't want to. Also how does this guy know this is a "machine-learning algo" and not just a series of ifs and elses?
  - sgerenser 7y ago
    It’s much more likely a series of ifs and else’s, not a fancy ML model. Companies have been doing automatic credit decisions for decades and there’s no evidence that GS bank is doing anything different than all the others.
_bxg1 7y ago
It could also very well be "We trained our neural net on historical allocations of credit limits by professional analysts, who had a subconscious gender bias". The point is there's no way to know.
thereare5lights 7y ago
> but there's absolutely no reason to actually believe they're conditioning on gender. The anecdotal evidence, that many women are claiming that they received much lower credit limits than their partners, despite similar or superior credit conditions, would seem to indicate otherwise. I don't believe anyone is saying that they're directly doing it. Just that their algorithms have that outcome.

tenpies 7y ago

I mean you know the system is sexist when one gender out-lives the other by a considerable number of years yet the retirement age for both is the same.

elvinyung 7y ago

I mean, don't you think it's kind of a strawman to assume anyone thinks machines are explicitly sexist? It seems much more likely that whatever method they use to do feature selection on this obviously high-dimensional data just happens to end up picking something that was a proxy for gender. You can be implicitly biased without being aware of it. This is true for both humans and algorithms.

kyrieeschaton 7y ago
It is absolutely not a strawman. It is incredibly easy to find people granted authority by the state that will claim that, eg, mathematics per se is an instance of exclusionist masculine thinking. It's even easier to find instances where people granted authority by the state will claim that any discrepancy in outcomes is ipso facto intentional.
gwd 7y ago
> It seems much more likely that whatever method they use to do feature selection on this obviously high-dimensional data just happens to end up picking something that was a proxy for gender. Yes, this is exactly what everyone thinks happened. Nobody thinks people at Goldman-Sachs wrote if (applicant.gender == "F") limit /= 20 somewhere in their algorithm. But regardless of how the thing happened, if millions of people are treated significantly differently for no reason other than their plumbing, that's a major problem. People have been talking about this "accidental proxy for gender" for years now; there's absolutely no excuse for no doing a basic sanity check to make sure that this kind of thing isn't happening. edit: typo
- kyrieeschaton 7y ago
  "for no other reason than their plumbing" is exactly what you just excluded in the prior sentence.

jyrkesh 7y ago

I've heard anecdotally a number of times that actuaries peg men with higher car insurance rates. I'm only just learning that there's anything like this in the credit system to prevent gender bias. Does something like that protection exist for auto insurance? As a side note, Idon't really understand the thread's appeal to credit scores here, considering that the TransUnion rating system is supposed to be the inferior one that Apple is looking to replace. But most of all, I'm shocked by the number of people here outraged by Apple's behavior who will not even be considering switching off their iPhone (Twitter OP included, hell, he even posted a screenshot of his recurring TransUnion payment still being served via Apple Pay). I actually happen to agree with OP, but this blaise attitude towards real customer complaints has kept me from using Apple products for years.

dijit 7y ago
For car insurance? This is incredibly well known and not at all anecdotal. There was even a popular car insurance firm in the UK that only gave women car insurance if they were women called “Sheilas Wheels” which touted lower fees than “other insurance companies that have to also cater to men”. https://youtu.be/GzNJh1o84-E https://youtu.be/GzNJh1o84-E
- TAForObvReasons 7y ago
  Even though it wasn't a protected class, GEICO (Government Employees Insurance Company) originally started as an auto insurance company specifically for federal government employees, a pool perceived to be less risky than the driving population
- tialaramex 7y ago
  It's also now illegal for insurers to do this in the EU since the ECJ reasoned that discriminating on gender, even if as a result of such a correlation, was prohibited. Anecdotally I've been told that (British car) insurers don't care very much about your real actuarial risk, they're focused more on whether you'll actually pay their premiums. Specifically I was told that work to integrate with credit checking services was a priority whereas an integration with the UK Government's service which gives them access to driving offences and other records related to a driving license was back-burnered. The reason I was given was that in practice they'd found if you require drivers to give their license details, it causes a big drop in purchases, if you make the license details _optional_ lots of people fill them in, and you can just give all those people a better price even though you don't use the details to actually check anything automatically.

burnte 7y ago

> I guess it worked for him to throw a hissy fit, but there's absolutely no reason to actually believe they're conditioning on gender. There certainly is. Ignoring the anecdotal data of people replying to him who saw the same outcome, credit scores for women skew lower than for men. One of many sources: https://www.federalreserve.gov/econres/notes/feds-notes/gender-related-differences-in-credit-use-and-credit-scores-20180622.htm https://www.federalreserve.gov/econres/notes/feds-notes/gend...

zamalek 7y ago
> credit scores for women skew lower than for men. Which is crazy, because for most of my guy friends (myself included), saying that their significant others/women make significantly better financial decisions is putting it far too lightly. I basically wasn't making decisions at all (besides savings) until my s/o set me right. This discrimination is tragic and disgusting.
- DoreenMichele 7y ago
  Women tend to have lower incomes and lower net worths. Here is a recent article squeeing about women finally being willing to buy homes on their own: https://realestate.usnews.com/real-estate/articles/the-rise-of-the-single-female-homeowner https://realestate.usnews.com/real-estate/articles/the-rise-... One of the statistics in it: Single women can only afford about 39% of homes. Single men can afford more than half. Historically, couples with traditional marriages (primary breadwinner husband, wife whose primary responsibilities were women's work) often did not bother to put her name on a real estate transaction or car purchase. This substantially impairs a woman's ability to establish a credit record and reduces her legal claim to assets. If you get divorced, the person with a real career and resume to match will likely continue having a good income. Someone who was a full-time wife and mom will face serious barriers to establishing a real career at all and may well remain poor for years to come. I'm quite financially savvy. Financial savvy only goes so far. Ability to pay still matters and men are more likely to have that piece covered.
  - acjohnson55 7y ago

notadoc 7y ago

> I guess it worked for him to throw a hissy fit, but there's absolutely no reason to actually believe they're conditioning on gender. Of course not, but why spoil the outrage mobs victimhood narrative du jour?

dictum 7y ago

I would love a customer-company relationship in which the customer gets meaningful information by privately contacting the company with a polite message, but in my experience this is rare. When an answer comes, it tends to misdirect rather than tell the whole truth. Here you have a bank and a company known for secrecy. But you acknowledge this; > random CSRs don't get a pithy explanation What the throwers of hissy fits are pointing to is that, by building blackboxes, you can get whatever result you want (which doesn't mean all results are expected) with plausible deniability.

r00fus 7y ago
Perhaps because this exact approach of a candid private question can be abused to support litigation or corporate espionage.

reaperducer 7y ago

It could, realistically, be almost anything except for evil bankers deciding to use illegal criteria FTA: “My belief isn’t there was some nefarious person wanting to discriminate. But that doesn’t matter. How do you know there isn’t an issue with the machine-learning algo when no one can explain how this decision was made?”

ndzig 7y ago
They can explain, they most likely don't want to. Also how does this guy know this is a "machine-learning algo" and not just a series of ifs and elses?
- sgerenser 7y ago
  It’s much more likely a series of ifs and else’s, not a fancy ML model. Companies have been doing automatic credit decisions for decades and there’s no evidence that GS bank is doing anything different than all the others.

_bxg1 7y ago

It could also very well be "We trained our neural net on historical allocations of credit limits by professional analysts, who had a subconscious gender bias". The point is there's no way to know.

thereare5lights 7y ago

> but there's absolutely no reason to actually believe they're conditioning on gender. The anecdotal evidence, that many women are claiming that they received much lower credit limits than their partners, despite similar or superior credit conditions, would seem to indicate otherwise. I don't believe anyone is saying that they're directly doing it. Just that their algorithms have that outcome.