Like the line in Fight Club [1] about recalls it is a simple formula: Only if the cost * probability of a PR disaster outweighs the cost of a poor credit risk

Like the line in Fight Club [1] about recalls it is a simple formula: Only if the cost * probability of a PR disaster outweighs the cost of a poor credit risk model, is a bank economically incentivized to not discriminate. Then again, even if you take care to not directly discriminate, you will probably indirectly discriminate. For instance, in your credit risk model, remove the `gender` column from the features, and use the remaining features to try to predict it. If performance is better than random guessing, you are using proxy features (like `income`) to discriminate on `gender`. You will find that nearly every feature you use is correlated with `race`. Now what? Throw away all these features and let your competition eat your lunch? Along the lines of what Pedro Domingos said [2], you can not solve the problem of discrimination by making poorer performing machine learning models that adhere to your view of what is ideal. Discrimination won't disappear because you made a model that makes you feel good. Want no discrimination of women? Work on closing the wage gap. Don't cripple your statistically correct ML models or sweep the discrimination under the rug, covered by correlated variables. It is not so much carelessness, as it is the nature of the beast. And banks remain in business by how much they can trust their customers first and foremost, trustworthiness by customers is a second (and customer trust is very much malleable: It is the perception of trust, not objective trust like "can we trust this customer to pay back their loan"). It also depends on how you (mathematically) define "fairness" [3]. You can define fairness in ways that still allow you to discriminate. [1] A new car built by my company leaves somewhere traveling at 60 mph. The rear differential locks up. The car crashes and burns with everyone trapped inside. Now, should we initiate a recall? Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one. [2] https://www.youtube.com/watch?v=furfdqtdAvc https://www.youtube.com/watch?v=furfdqtdAvc [3] https://algorithmicfairness.wordpress.com/2016/09/26/on-the-impossibility-of-fairness/ https://algorithmicfairness.wordpress.com/2016/09/26/on-the-... https://arxiv.org/abs/1609.07236 https://arxiv.org/abs/1609.07236