Actually we can measure the frequency of light in both directions, and the wavelength of light in both directions, and calculate that it's going the same speed

Actually we can measure the frequency of light in both directions, and the wavelength of light in both directions, and calculate that it's going the same speed in both directions. It's just that if you try to measure the elapsed time in each direction you run into a chicken and egg problem trying to define what 'elapsed time' really means in a relativistic environment.