The only way I've found to get around this is to either temporarily disable the setting if I am expecting a particular call or make sure that I have contact inf

The only way I've found to get around this is to either temporarily disable the setting if I am expecting a particular call or make sure that I have contact info for every single person I interact with during these exchanges. Even if I know where the call is coming from, your experience shows it is still possible to mess up because you can't possibly account for all the kinds of weird phone systems people might be using.