This explanation is solid and speaks loudly to the benefits of right to explanation. Because if the explanation is "We've found signal in the data that correlat

This explanation is solid and speaks loudly to the benefits of right to explanation. Because if the explanation is "We've found signal in the data that correlates strongly with social biases (example: surname analysis---of course there's increased risk, if the lender is operating in a country where people don't trust anyone whose name starts with "Mc", for example), and we can use that signal to basically ethnically profile without having to admit that we're ethnically profiling..." that's bad, and the law may address it. "Redlining" practices aren't bad because they don't economically work. They do economically work; if one demographic is societally disadvantaged, you don't do your firm any short-run favors by catering to them, because they aren't where the money is. They're bad because they work by unfairly shifting the pie away from people who don't actually deserve to have the pie shifted away from them---it's a variant on "sins of the fathers" reasoning to bias on categories like surname, or race, or demographics that correlate strongly with those demographics (like geography, in cities in the United States). There are moral reasons to not allow those practices to get codified into algorithms so the people using them can excuse bad behavior as "just following the computer's orders."