I don't think it's obvious that those men were chosen for thousands of years. In those long-gone societies, did the women really get to pick men based on looks?

I don't think it's obvious that those men were chosen for thousands of years. In those long-gone societies, did the women really get to pick men based on looks? Maybe they didn't get to pick at all, or maybe they picked based on survival resources. In most cases, it isn't even possible for male looks and female looks to be separately selected. That can only happen if a sex-specific gene promoter is in front of the gene. Most traits are selected without regard to sex. So if selection by looks is mostly a matter of men choosing women, traits that create attractive women will spread through the population even if those traits are also changing male appearance in ways that women dislike.