To some extent it does fill in based on what it has seen before. Although there is a fair number of parameters in that model, it cannot memorise the entire clip

To some extent it does fill in based on what it has seen before. Although there is a fair number of parameters in that model, it cannot memorise the entire clip. This is apparent, since the "dream" sequences cannot reproduce the entire sequence and compress/stretch/repeat certain subsequences or collapse into a fixed point. So the fill in does have some temporal context of what was there recently. I'm running a bigger model now (great thing about PVM, it scales seamlessly) which should provide a better quality image.