If you consider the difference between your two examples, the French man and the black man, I think you may see how one would be more put off than the other by

If you consider the difference between your two examples, the French man and the black man, I think you may see how one would be more put off than the other by the uncomfortable compliment. Specifically, within the United States (and other countries as well) the black man has been historically disenfrachised and therefore any comment/compliment that they do not belong there will not be well received. The 'too xyz to be here' comment is typically made by people that aren't in a position to judge, possibly for a number of reasons.