The problem is not so much about whether predictions are biased. It's a problem of whether the algorithm is reinforcing pre-existing biases. In other words, it'

The problem is not so much about whether predictions are biased. It's a problem of whether the algorithm is reinforcing pre-existing biases. In other words, it's about taking short-term losses in return for long-term gains. Suppose a certain group of people have experienced some kind of historical discrimination, and as a result people in that group are starting from a disadvantaged position (wealth, etc) compared to the rest of the population. It may well be an unbiased prediction that such people will be a greater credit risk, etc. However, by making decisions solely on the basis of this, although the bank saves some money in the short-term, it reinforces the cycle of discrimination. If your goal is to improve society in the long-term, making decisions based on these short-term predictors is not a desirable thing to do.