4 ms·

I would take DHH's interpretation of this with a large grain of salt. He seems to assume that living in a "community property state" will be taken into account

by themgt 7y ago

I would take DHH's interpretation of this with a large grain of salt. He seems to assume that living in a "community property state" will be taken into account (presumably then considering his wife to have parity with his income/score) by the credit limit algorithm, but we have no way to know that that's the case. He also adds "It gets even worse. Even when she pays off her ridiculously low limit in full, the card won’t approve any spending until the next billing period. Women apparently aren’t good credit risks even when they pay off the fucking balance in advance and in full." Are we really to imagine the Apple/Goldman algorithm has some "if(gender.female){ cc_payment_terms = :discrimination }" sort of code in it? FWIW I'm a male and have a mid-700s credit score and was denied Apple card approval. I am fairly certain there was no sex discrimination involved in the denial.

rriepe 7y ago
If they're using machine learning, then yes it may literally have code like that in it.
minimaxir 7y ago
> Are we really to imagine the Apple/Goldman algorithm has some "if(gender.female){ cc_payment_terms = :discrimination }" sort of code in it? Not intentionally, but it's possible for black box AI algorithms to engineer such a model feature.
jacquesm 7y ago
It won't have that line in their code. What it very likely will have is some Bayesian algo or maybe if they're really high tech some ML black box that will have gender as one of its inputs. And it probably shouldn't have that input. The whole idea behind decisions like these is that they have to be explainable and ML especially does not lend itself well for that.
- matthewmacleod 7y ago
  Note that an ML box doesn’t even need gender as an input. Name alone probably gets you 90% of the way there.
  - jacquesm 7y ago
    True, you could have an internal table that maps name to gender with a very high probability of being right.
  - xtacy 7y ago
    That points to the core of the issue. "Fairness" in ML algorithms can be hard to define and assess. It's easy to say "omit gender from the model", but the real issue here has to do with the _causal_ pathways between your input variables and the output variable. Since ML mostly works by exploiting correlations between the input and output variables, omitting gender doesn't mean gender's influence is removed. You'll have to omit all the causal pathways from gender -> the output, effectively "d-separating" [1] gender from the output. Whether that's practical or not depends on how well we understand the data generating process. [1] http://bayes.cs.ucla.edu/BOOK-2K/d-sep.html http://bayes.cs.ucla.edu/BOOK-2K/d-sep.html
    - rightbyte 7y ago
      You can simulate it with artificial querries with the same data except gender and name and see if women get less or more.
- themgt 7y ago
  Here's a thread on Apple support[1], with many reporting similar experience and an anecdote going the opposite way: I’ve been having the same issue since Monday 10/14. I paid my balance in full, money was taken from my Chase account right away and cleared next day, but my Apple Card available balance hasn’t updated. I’ve also gotten the same response from support. It could take several days. Funny thing I paid my wife’s card off a few weeks ago and it updated within seconds, same bank account and everything...... [1] https://discussions.apple.com/thread/250676909?page=2 https://discussions.apple.com/thread/250676909?page=2
zrail 7y ago
We don’t know, and that’s the problem. Machine learning models are entirely dependent on their training data. If the data is biased the model will be too.

by themgt 7y ago

rriepe 7y ago

If they're using machine learning, then yes it may literally have code like that in it.

minimaxir 7y ago

> Are we really to imagine the Apple/Goldman algorithm has some "if(gender.female){ cc_payment_terms = :discrimination }" sort of code in it? Not intentionally, but it's possible for black box AI algorithms to engineer such a model feature.

jacquesm 7y ago

It won't have that line in their code. What it very likely will have is some Bayesian algo or maybe if they're really high tech some ML black box that will have gender as one of its inputs. And it probably shouldn't have that input. The whole idea behind decisions like these is that they have to be explainable and ML especially does not lend itself well for that.

matthewmacleod 7y ago
Note that an ML box doesn’t even need gender as an input. Name alone probably gets you 90% of the way there.
- jacquesm 7y ago
  True, you could have an internal table that maps name to gender with a very high probability of being right.
- xtacy 7y ago
  That points to the core of the issue. "Fairness" in ML algorithms can be hard to define and assess. It's easy to say "omit gender from the model", but the real issue here has to do with the _causal_ pathways between your input variables and the output variable. Since ML mostly works by exploiting correlations between the input and output variables, omitting gender doesn't mean gender's influence is removed. You'll have to omit all the causal pathways from gender -> the output, effectively "d-separating" [1] gender from the output. Whether that's practical or not depends on how well we understand the data generating process. [1] http://bayes.cs.ucla.edu/BOOK-2K/d-sep.html http://bayes.cs.ucla.edu/BOOK-2K/d-sep.html
  - rightbyte 7y ago
    You can simulate it with artificial querries with the same data except gender and name and see if women get less or more.
themgt 7y ago
Here's a thread on Apple support[1], with many reporting similar experience and an anecdote going the opposite way: I’ve been having the same issue since Monday 10/14. I paid my balance in full, money was taken from my Chase account right away and cleared next day, but my Apple Card available balance hasn’t updated. I’ve also gotten the same response from support. It could take several days. Funny thing I paid my wife’s card off a few weeks ago and it updated within seconds, same bank account and everything...... [1] https://discussions.apple.com/thread/250676909?page=2 https://discussions.apple.com/thread/250676909?page=2

zrail 7y ago

We don’t know, and that’s the problem. Machine learning models are entirely dependent on their training data. If the data is biased the model will be too.