I saw a comment on HN a while ago that this reminded me of. It made the argument that the primary value of a lot of machine learning models is essentially to "

I saw a comment on HN a while ago that this reminded me of. It made the argument that the primary value of a lot of machine learning models is essentially to "launder" information about protected classes so that it becomes harder to discover when you have been discriminated against, and harder to assign blame to a specific entity when that's the case. For instance, using a complicated nonlinear combination of address (e.g. predominantly black neighborhoods), educational history (e.g. historically black colleges), work history, credit rating, and who knows what else, might simply allow a model to recover a feature (ethnicity) which is correlated with likelihood to repay a loan in historical data. Like you pointed out, that would lock us into a positive feedback loop where people are being denied loans based on "statistical, race-blind models based only on data", but the data only looks that way because of existing racism, which is now being further enshrined and legitimized. But the thorniest issue is that the model reflects an actual, true, relationship in the world, so asking banks to ignore it would be asking them to leave money on the table until such a time as society has eliminated the structural inequalities that cause those correlations. How do we get them to do that, instead of just engaging in more and more sophisticated data laundering like above? Would we have to create an approved list of features that can be used for loan approval, which provably cannot be used to discover race? How would we even know that such a rule was working?