Yes, but the spectrum of the light hitting the eye is a physical thing. The physical problem is that multiple environments can make the same spectrum, e.g. is

Yes, but the spectrum of the light hitting the eye is a physical thing. The physical problem is that multiple environments can make the same spectrum, e.g. is it a bluish light source & white dress or white light and bluish dress. White balancing can fix this if you know the light source, e.g. the sun, but that's either an assumption or something calibrated separately from the image. That said, the perceptual issues are certainly a problem as well :)