Suppose the training data is a bank that has historically given higher-interest loans to minorities. Those loans will default more often, and so the algorithm c

Suppose the training data is a bank that has historically given higher-interest loans to minorities. Those loans will default more often, and so the algorithm could "correctly" conclude that minorities default more often. But if the relationship is actually `defaultProb = a x interest_rate + b`, then the algorithm will reflect that. I discuss this case explicitly in the blog post I linked to in the section "What if black people don't perform as well?", and explicitly give an example of a linear model NOT doing what you claim it will do. The fact is, both statements in this example would be true: "higher interest is correlated with more defaults" and "minority borrowers are correlated with defaults." The algorithm doesn't know one from the other. No, it wouldn't because you'd also have cases of white people with high interest who defaulted. The model using race as a predictive variable would fail to fit those data points. I explicitly discuss this case in my blog post. You are correct that in some cases you don't have enough training data. But in these cases, there is no particular reason for bias to have a particular sign. Again, as I discuss in the blog post I linked, why would Captain Kirk be biased against "black on left guy" vs "black on right guy"? Insufficient training data causes variance, not bias, and could just as easily result in bias in favor of blacks. (In fact, I link to a number of examples where it does.)