I'm aware that political processes surrounding loans were racially biased. That's not evidence that a profit maximizing algorithm will be. That's really just a

I'm aware that political processes surrounding loans were racially biased. That's not evidence that a profit maximizing algorithm will be. That's really just a claim that "this was once politicized, therefore it must remain political forever". The whole benefit of using algorithms is that you make your goals explicit and non-opaque; you just look at the objective function. Opacity of the models isn't really a big deal - machine learning is fundamentally the mathematics of studying opaque models. It's the objective function that should be transparent. If the objective function says "minimize defaults", that's your explicit goal. If the objective function says "maximize # of blacks getting loans", that's what you'll do. But it will do one or the other, unless you explicitly set your objective function to a x "minimize defaults" + b x "maximize blacks". The tricky bit is that by doing this, we are explicitly and verifiably lowering the bar. Just witness how much flak people get for discussing this: https://techcrunch.com/2015/11/03/twitter-engineering-manager-leslie-miley-leaves-company-because-of-diversity-issues/ https://techcrunch.com/2015/11/03/twitter-engineering-manage... But unfortunately, an optimization system won't give you what you want unless you explicitly put what you want in the objective function, something our modern activists (probably including Cathy O'Neil) refuse to do.