4 ms·
Does “massively parallel simulation” help advance Reinforcement Learning?
- yanglet 4y agoNVIDIA's Isaac Gym project revealed GPU's capability of performing massively parallel simulation for gym-style environments. Detailed information can be found in the following paper: [1] Makoviychuk, Viktor, et al. "Isaac Gym: High-Performance GPU Based Physics Simulation For Robot Learning." Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021. At its release, people commented on Twitter that "it is the MNIST moment for reinforcement learning." And over the past year, I saw several follow-up works and tested NVIDIA's implementations. For example, a demo by this blog https://towardsdatascience.com/a-new-era-of-massively-parallel-simulation-a-practical-tutorial-using-elegantrl-5ebc483c3385 https://towardsdatascience.com/a-new-era-of-massively-parall... The question is, does that technique help advance Reinforcement Learning, as expected?
- hcrisp 4y agoIt's not new. A paper in 2021 [0] showed you can train a quadruped robot to walk in minutes using parallel simulation in GPU, and then deploy it on the physical robot. Being parallel it is faster, and more so if on GPU. But sim-to-reality transfer is still a concern, and the architecture doesn't help if there are sparse rewards. [0] https://arxiv.org/abs/2109.11978?context=cs.LG https://arxiv.org/abs/2109.11978?context=cs.LG
- yanglet 4y agoThanks for the information and sharing the paper. I also read it. Agree with you that the "simulation-to-reality gap" would be more critical to the reinforcement learning community.