Forcing people to work with someone against their will or preference also causes harm, and if society wants to do that it has to justify that the harm prevented

Forcing people to work with someone against their will or preference also causes harm, and if society wants to do that it has to justify that the harm prevented is significantly greater than that - which is true for some types of discrimination and not true for others. If nobody in the village wants to work with Bob because he's an asshole, they should be able to discriminate against Bob (and other assholes) as the social harm of Bob being unemployed and unemployable does not justify the social harm caused by the state coercing people to work with Bob, violating their freedom to choose. It would be harmful (and IMHO immoral) to prohibit discrimination against assholes. Freedom of association - and the freedom of not-associating it implies - is not inherently hateful, it is one of fundamental human rights that should be curtailed only if other alternatives aren't feasible - merely having some reason or some harm prevented is not necessarily enough.