For historical background on this part of the algorithm, called "experience replay", see this paper from Long-Ji Lin in 1992: Self-Improving Reactive Agents Ba

For historical background on this part of the algorithm, called "experience replay", see this paper from Long-Ji Lin in 1992: Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching http://link.springer.com/article/10.1023/A:1022628806385 http://link.springer.com/article/10.1023/A:1022628806385 as well as his excellent 1993 PhD thesis: Reinforcement Learning for Robots Using Neural Networks http://www.dtic.mil/dtic/tr/fulltext/u2/a261434.pdf http://www.dtic.mil/dtic/tr/fulltext/u2/a261434.pdf