However, we avoid discriminating between protected groups because we believe it's wrong to subject an individual to that kind of discrimination, even if such di

However, we avoid discriminating between protected groups because we believe it's wrong to subject an individual to that kind of discrimination, even if such discrimination would be rational for the discriminator. I don't think most of us would have a problem with discriminating on an inherent property of a group that is relevant to the decision being made. Indeed, often neither do anti-discrimination laws. Such discrimination can be objectively justified. For example, consider a job where a certain level of fitness is a functional requirement, say a firefighter whose role includes being able to carry someone out of a burning building. Setting a relatively high bar for physical strength is going to discriminate against female job applicants and those with various physical disabilities as their respective groups, but it's not discriminating against someone because they're female or in a wheelchair, it's discriminating against them because they can't carry someone out of a burning building and so that person is going to die. I don't think most of us would have a problem with this kind of rule. The only thing unfair here is life not making us all equal, and there isn't much we can do about that. The problems usually start when either you discriminate against a whole group based on some property that is somewhat correlated with membership of the group but not inherent, or you discriminate against a whole group based on a property that is inherent but isn't actually relevant to the decision being made. For example, if a job involves sitting at a desk and using a computer all day, a woman or someone in a wheelchair can presumably do that just as well as our hypothetical strong male. In this case, discrimination on the basis of gender or physical strength simply isn't relevant to the decision being made but sadly does sometimes happen because of individual prejudice, and thus we have laws to protect those in more vulnerable situations. Unfortunately, the kinds of tools and mathematical analyses we're talking about here don't necessarily see those situations any differently. They may learn the wrong lesson if there are coincidental, misleading correlations in whatever data is used to train them, which brings us back to where we came in.