The misinterpretation comes from conflating difference in population averages with differences in population. He said that on average men and women are differen

The misinterpretation comes from conflating difference in population averages with differences in population. He said that on average men and women are different, but not that all women and men are different. The statement "men and women are different" implies the latter.