That's really valid only in the case when the frequency response and the impulse response completely characterize a system--that is, only for linear and time-in

That's really valid only in the case when the frequency response and the impulse response completely characterize a system--that is, only for linear and time-invariant systems. I know you covered this base by mentioning nonlinear distortions, but I think that's the big issue. Otherwise you could just correct the frequency response with equalization (assuming the frequency response isn't literally 0 ever). For example, consider a system that just squares or cubes its input. The unit impulse response will look the same in both cases. The frequency response isn't even defined, since the steady state response to a sinusoid is not itself a sinusoid.