One way to solve this problem is to give your neural network a stack of successive frames, instead of given them one by one. If the framerate is too high, you m

One way to solve this problem is to give your neural network a stack of successive frames, instead of given them one by one. If the framerate is too high, you may need to skip some of the frames to get idea of speed/acceleration. See eg the concepts of "frame stack" and "frame skip" in reinforcement learning: https://danieltakeshi.github.io/2016/11/25/frame-skipping-and-preprocessing-for-deep-q-networks-on-atari-2600-games/ https://danieltakeshi.github.io/2016/11/25/frame-skipping-an...