I think it is considered problematic because ability in the modern world (at the very least) is largely correlated with "mental attributes", and, in a meritocra

I think it is considered problematic because ability in the modern world (at the very least) is largely correlated with "mental attributes", and, in a meritocratic society, a group which was less able would be afforded less power. That unequal distribution of power would be in direct conflict with the democratic ideal of equal representation, leading to the tension. If it was generally believed that the democratic will represents everyone's interests roughly equally, then this wouldn't of itself present a problem. Unfortunately, the failure of nominally inclusive establishments to address the political concerns of large sections of their societies strains the credibility of this belief, to the point of breaking, in some cases. Beyond the political consequences, meritocratic societies have a tendency to draw a relationship between economic value and social value. Telling a group of people that they are less socially valuable just isn't very nice. Of course, these relationships aren't facts of the world - they are simply choices made be societies. So, it would be possible to have a society where such things as you mention could be discussed openly without drawing criticism, but it would be necessary for the labelling that you were engaging in to not make implicit claims about people's social value.