Recurrent neural networks with e.g. Long Short Term Memory (https://en.wikipedia.org/wiki/Long_short_term_memory https://en.wikipedia.org/wiki/Long_short_term_m

Recurrent neural networks with e.g. Long Short Term Memory (https://en.wikipedia.org/wiki/Long_short_term_memory https://en.wikipedia.org/wiki/Long_short_term_memory) can model very long term dependencies effectively, and there has been some work lately that get good results with simpler models. They don't use it because it's computationally expensive and totally unnecessary for Atari games, but it's certainly possible.