Possibly because race is one of the protected classes by anti-discrimination law, which makes the characteristic a legal hazard. Perhaps you've heard of stories

Possibly because race is one of the protected classes by anti-discrimination law, which makes the characteristic a legal hazard. Perhaps you've heard of stories where people are hesitant to hire women, because sex and pregnancy are both protected. In fact, most of the groups you mentioned are covered by anti-discrimination law, so they would naturally fit this model. The USA legislates most major workplace interactions I can think of. I don't find it too surprising that people want to avoid the costly and painful legal system through whatever means they can think of, regardless of their effectiveness or ethicalness.