It sounds like part of this issue is that it loses tracking if it can't see both of your eyes, which of course could be defeated by using a couple of cameras sp

It sounds like part of this issue is that it loses tracking if it can't see both of your eyes, which of course could be defeated by using a couple of cameras spaced at 45° to one another and calibrated to work together in some way. Instead of a "deep fake" face swap an attacker could send virtual video from a fully-virtual environment using something like an nvidia Metahuman controlled by the camera array. I think that would be pretty easily detectable today but maybe less so with an emulated bad webcam and low-res video link. The models/rigging are only going to improve in the future. The classic "Put a shoe on your head" verification route would still defeat that, at least until someone invents a very good tool to allow those types of models to spawn and manipulate props.