Exactly, while doing background research for this I read a really neat paper that describes what you're saying, and offers a solution based on trying to predict

Exactly, while doing background research for this I read a really neat paper that describes what you're saying, and offers a solution based on trying to predict the viewer's eye motion: Temporal Video Filtering and Exposure Control for Perceptual Motion Blur (Stengel et al., 2015)