The system as shown has a state of a few frames. Suppose it had 1 instead for illustration. Suppose there's a dangerous ball right next to you, but it's actuall

The system as shown has a state of a few frames. Suppose it had 1 instead for illustration. Suppose there's a dangerous ball right next to you, but it's actually moving away from you, so you don't need to run; in this model, you will probably learn to always run since you cannot know the velocity of the ball (whether it's coming at you). So this model could only work if you have a state-full representation of your world, which seems to be the case with simple games and a few frames, but is not the case in the real world (in the real world your naive state would need to arbitrarily back in time -- e.g. say you see a ticking time bomb, and then the clock is hidden from you). This flaw is understandable since it seems Q-Learning was conceived to deal with Finite Markov Processes, which are finitely state-full, by definition (if you don't know them, they're essentially a non-deterministic state machine).