When you're wearing the headset, you're not truly seeing what's in front of you in realtime; it's not a window. Instead, you're seeing a video using the cameras

When you're wearing the headset, you're not truly seeing what's in front of you in realtime; it's not a window. Instead, you're seeing a video using the cameras on the exterior of the device which is then streamed to the displays on the interior. This introduces latency. The published stats are something absurd, like 12ms which is allegedly faster than the human eye can even perceive, but what if there's a CPU spike and lag? On top of that, the FOV is much more restricted in the headset than the average human's, and there is noticeable distortion introduced at the edges of the displays as documented by several reviews. So you're seeing less than you normally would, and it's warped. Those things aside, if you're wearing the VR headset, it's likely you're doing something that is pulling your focus away from the act of driving.