It's only a sign flip when the detector and emitter are perfectly in phase. Adding delay causes the required phase shift will drift slightly. Also note that the

It's only a sign flip when the detector and emitter are perfectly in phase. Adding delay causes the required phase shift will drift slightly. Also note that the phase offset of a delay is frequency dependent, so this drift will be as well. Another reason is that devices and materials have different frequency responses, so an external signal needs to be filtered to match the levels in the device.