It's actually worse than that, for a near-eye display: unless you know where the person is focusing, you actually don't know where the redundancy is, so you hav

It's actually worse than that, for a near-eye display: unless you know where the person is focusing, you actually don't know where the redundancy is, so you have to draw the entire 4D raster. (If you know their focus distance, you can probably just draw a 2D image on the retina and be done -- you get that 4D->2D projection.) Otherwise, the existence of those additional rays is what allows for focus accommodation: as you focus in different planes, it 'shuffles' the light around, to create sharp edges (where similar light rays line up) or blurry foreground/background (similar light rays strike different portions of the retina -- and if any are missing, there is a 'hole' in the blur disk). Pedantic, perhaps... but these are the stakes.