I've logged over 200 hours in Beat Saber, mostly playing with friends. What you're talking about is definitely a real problem, but the solution is to just have

I've logged over 200 hours in Beat Saber, mostly playing with friends. What you're talking about is definitely a real problem, but the solution is to just have the game rendered to the TV as well as the headset. Everyone sees and hears what the person is doing, even though they're the one in VR. My wife watched me play the entirety of Half Life Alyx that way, and she enjoyed it a lot. And were were able to talk and share the experience the entire time.