I am not sure exactly what your argument is, but it doesn’t sound correct. The differences will be larger in the smaller group, but the expected value for the a

I am not sure exactly what your argument is, but it doesn’t sound correct. The differences will be larger in the smaller group, but the expected value for the average pay in a group is independent of its size. That’s why you can use a sample to find out something about a population.