It would have to remember the data that is off-screen. It's quite possible to train a recurrent network to do this. Some Atari games require this to play well,

It would have to remember the data that is off-screen. It's quite possible to train a recurrent network to do this. Some Atari games require this to play well, like Pacman requires remembering where the ghosts are when they're blinking. More complicated is the logic to know what to look at when, by keeping track of which parts of the state might be important and out of date, and controlling the viewpoint to update it. I haven't seen a good example of this.