You can get a lot done with re-projection and temporal anti-aliasing. At the most abstract level you can consider it an encoding problem: given an image, what's

You can get a lot done with re-projection and temporal anti-aliasing. At the most abstract level you can consider it an encoding problem: given an image, what's the marginal entropy of a slightly deviating image? There's very little that actually needs to be uniquely computed. Edit: If you're willing to give up strict coherence between views you might be able to do even better.