Solution: capture the light field of the scene using a renderer with a thin-lens camera model and then reconstruct the correct rays to display in the headset in

Solution: capture the light field of the scene using a renderer with a thin-lens camera model and then reconstruct the correct rays to display in the headset in real time. Oh, wait, we already have the software to do it, while I'm again convinced the military units and projects discussed in OP don't.