3 ms·

I work for a big bank... As long as people freak out when banks are found to be discriminating they will do their best to not discriminate. Banks are built on

I work for a big bank... As long as people freak out when banks are found to be discriminating they will do their best to not discriminate. Banks are built on trustworthiness. Having your bank's name in the headlines for discriminatory practices can have a severe negative impact on trustworthiness. They have a whole teams of people devoted to this topic and every year at most banks every employee has to learn about, "reputational risk." Having worked at big banks for over a decade now I am 90% certain that discrimination by banks at this point is primarily due to carelessness.

tptacek 10y ago
That is a story that banks like to believe about themselves, but it's worth considering that emergent discrimination (and the broader bucket of emergent malfeasance) is a property of all complex systems, not just machine learning. So, for instance, Wells Fargo needed to do more than just hope that its trustworthiness would survive its incentive systems.
- riskable 10y ago
  Not taking steps to prevent emergent discrimination or malfeasance is careless. Do you think the board of directors wants to find out that the CEO intentionally ignored calls to the ethics hotline? That he didn't take steps to ensure whistleblowers weren't punished? The consequences of these things are very bad for stock prices! This is doubly true for banks which take reputational risk far more seriously than most businesses.
  - GFK_of_xmaspast 10y ago
    Wells Fargo stock appears to be back up to where it was a few months ago
BickNowstrom 10y ago
Like the line in Fight Club [1] about recalls it is a simple formula: Only if the cost * probability of a PR disaster outweighs the cost of a poor credit risk model, is a bank economically incentivized to not discriminate. Then again, even if you take care to not directly discriminate, you will probably indirectly discriminate. For instance, in your credit risk model, remove the `gender` column from the features, and use the remaining features to try to predict it. If performance is better than random guessing, you are using proxy features (like `income`) to discriminate on `gender`. You will find that nearly every feature you use is correlated with `race`. Now what? Throw away all these features and let your competition eat your lunch? Along the lines of what Pedro Domingos said [2], you can not solve the problem of discrimination by making poorer performing machine learning models that adhere to your view of what is ideal. Discrimination won't disappear because you made a model that makes you feel good. Want no discrimination of women? Work on closing the wage gap. Don't cripple your statistically correct ML models or sweep the discrimination under the rug, covered by correlated variables. It is not so much carelessness, as it is the nature of the beast. And banks remain in business by how much they can trust their customers first and foremost, trustworthiness by customers is a second (and customer trust is very much malleable: It is the perception of trust, not objective trust like "can we trust this customer to pay back their loan"). It also depends on how you (mathematically) define "fairness" [3]. You can define fairness in ways that still allow you to discriminate. [1] A new car built by my company leaves somewhere traveling at 60 mph. The rear differential locks up. The car crashes and burns with everyone trapped inside. Now, should we initiate a recall? Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one. [2] https://www.youtube.com/watch?v=furfdqtdAvc https://www.youtube.com/watch?v=furfdqtdAvc [3] https://algorithmicfairness.wordpress.com/2016/09/26/on-the-impossibility-of-fairness/ https://algorithmicfairness.wordpress.com/2016/09/26/on-the-... https://arxiv.org/abs/1609.07236 https://arxiv.org/abs/1609.07236
yummyfajitas 10y ago
The problem here is that non-discrimination costs you money (in practice: lots!), because it requires you to issue bad loans to non-Asian minorities. Secondly, it isn't mathematically possible to be "nondiscriminatory" - there are multiple definitions of that term and they are mutually conflicting. For example, as this article shows, "equal outcomes", "equal opportunity" and "equal treatment" (group unaware) don't make the same decisions. So no matter which definition of "fair" you choose, some intrepid reporter can choose a different definition and then write a clickbait article calling you racist.

tptacek 10y ago

That is a story that banks like to believe about themselves, but it's worth considering that emergent discrimination (and the broader bucket of emergent malfeasance) is a property of all complex systems, not just machine learning. So, for instance, Wells Fargo needed to do more than just hope that its trustworthiness would survive its incentive systems.

riskable 10y ago
Not taking steps to prevent emergent discrimination or malfeasance is careless. Do you think the board of directors wants to find out that the CEO intentionally ignored calls to the ethics hotline? That he didn't take steps to ensure whistleblowers weren't punished? The consequences of these things are very bad for stock prices! This is doubly true for banks which take reputational risk far more seriously than most businesses.
- GFK_of_xmaspast 10y ago
  Wells Fargo stock appears to be back up to where it was a few months ago

BickNowstrom 10y ago

Like the line in Fight Club [1] about recalls it is a simple formula: Only if the cost * probability of a PR disaster outweighs the cost of a poor credit risk model, is a bank economically incentivized to not discriminate. Then again, even if you take care to not directly discriminate, you will probably indirectly discriminate. For instance, in your credit risk model, remove the `gender` column from the features, and use the remaining features to try to predict it. If performance is better than random guessing, you are using proxy features (like `income`) to discriminate on `gender`. You will find that nearly every feature you use is correlated with `race`. Now what? Throw away all these features and let your competition eat your lunch? Along the lines of what Pedro Domingos said [2], you can not solve the problem of discrimination by making poorer performing machine learning models that adhere to your view of what is ideal. Discrimination won't disappear because you made a model that makes you feel good. Want no discrimination of women? Work on closing the wage gap. Don't cripple your statistically correct ML models or sweep the discrimination under the rug, covered by correlated variables. It is not so much carelessness, as it is the nature of the beast. And banks remain in business by how much they can trust their customers first and foremost, trustworthiness by customers is a second (and customer trust is very much malleable: It is the perception of trust, not objective trust like "can we trust this customer to pay back their loan"). It also depends on how you (mathematically) define "fairness" [3]. You can define fairness in ways that still allow you to discriminate. [1] A new car built by my company leaves somewhere traveling at 60 mph. The rear differential locks up. The car crashes and burns with everyone trapped inside. Now, should we initiate a recall? Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one. [2] https://www.youtube.com/watch?v=furfdqtdAvc https://www.youtube.com/watch?v=furfdqtdAvc [3] https://algorithmicfairness.wordpress.com/2016/09/26/on-the-impossibility-of-fairness/ https://algorithmicfairness.wordpress.com/2016/09/26/on-the-... https://arxiv.org/abs/1609.07236 https://arxiv.org/abs/1609.07236

yummyfajitas 10y ago

The problem here is that non-discrimination costs you money (in practice: lots!), because it requires you to issue bad loans to non-Asian minorities. Secondly, it isn't mathematically possible to be "nondiscriminatory" - there are multiple definitions of that term and they are mutually conflicting. For example, as this article shows, "equal outcomes", "equal opportunity" and "equal treatment" (group unaware) don't make the same decisions. So no matter which definition of "fair" you choose, some intrepid reporter can choose a different definition and then write a clickbait article calling you racist.