This is totally ridiculous. All it does is remove the useful data from the dataset and makes it's predictions worse. Imagine are using an algorithm to predict

This is totally ridiculous. All it does is remove the useful data from the dataset and makes it's predictions worse. Imagine are using an algorithm to predict how likely someone is to pay back a loan, or make an insurance claim, or perform as an employee, etc. Things like their grades or education would correlate with that. but it might also correlate with race or gender, so the algorithm would be labelled "racist" and "sexist". Even though it has nothing to do with racism whatsoever. A white person and a black person with the same education would get exactly the same result. The algorithm is totally fair and meritorious. The true solution would be to condition on the race or sex. So the algorithm predicts the probability they will be paid back given that they are male or female. Then in production you average the results of each possible sex and race together. So if women were really better drivers than men, they would still pay the same rate. The algorithm would know that they are better drivers, but it would treat everyone gender neutral. But it would still take into account education or other variables. Although it's not clear to me why you would want to do this. If men really are more risky drivers than women, they should pay more. And they do. If they really are less likely to pay back a loan, then they should have a higher interest rate. In any case, humans make decisions about stuff like this all the time and they are way, way, worse and more biased. In most cases very simple statistics vastly outperform the best humans: http://lesswrong.com/lw/3gv/statistical_prediction_rules_outperform_expert/ http://lesswrong.com/lw/3gv/statistical_prediction_rules_out...