3 ms·

> https://www.alexirpan.com/public/rl-hard/upsidedown_half_cheetah.mp4 https://www.alexirpan.com/public/rl-hard/upsidedown_half_che... You can argue, but it wo

> https://www.alexirpan.com/public/rl-hard/upsidedown_half_cheetah.mp4 https://www.alexirpan.com/public/rl-hard/upsidedown_half_che... You can argue, but it works. Goal accomplished. Nature did things that are stranger than that. Now you just have to add efficiency aspects into your reward functions and observe how it slowly finds a local minimum. Also something nature did a lot of times. Now hope that the remaining inefficiency is ok for you. Done. - You can start with a random neural net. It's not exactly empty, but it's ok. The randomness defines which local minimum you'll find this time around while you burn through your VC money desperately hoping that the AWS bills for all the GPU time will arrive after you've found the holy grail that makes your startup worth <insert bullshit valuation>. > "picking the right reward function" in RL is shockingly hard. Agree. x1000. And computational power and data quality. - on another post I've added some links (AI playing Mario): https://news.ycombinator.com/item?id=17489459 https://news.ycombinator.com/item?id=17489459

gwern 8y ago
> You can start with a random neural net. It's not exactly empty, but it's ok. One interesting thing about random NNs is how much they can already do. For example, you can do single-shot image inpainting or superresolution with an untrained randomly-wired CNN: https://dmitryulyanov.github.io/deep_image_prior https://dmitryulyanov.github.io/deep_image_prior In RL, you can use randomly sampled NNs to create various artificial arbitrary 'reward functions' to force an RL agent to explore an environment & learn the dynamics, and then when you give it the real reward function, it learns much faster how to optimize it. Similarly, you can sample random NNs to execute for entire trajectories for 'deep exploration', providing demonstrations of potentially long-range strategies much more efficiently than simple random-action strategies. In 'reservoir sampling', as I understand it, you don't even bother training the NN, you just randomly initialize it and train a simple model on the outputs, assuming that some of the random highly nonlinear relationships encoded into the NN will turn out to be useful, which sounds crazy but apparently works. Makes one think about Tegmark's interpretations.

gwern 8y ago

> You can start with a random neural net. It's not exactly empty, but it's ok. One interesting thing about random NNs is how much they can already do. For example, you can do single-shot image inpainting or superresolution with an untrained randomly-wired CNN: https://dmitryulyanov.github.io/deep_image_prior https://dmitryulyanov.github.io/deep_image_prior In RL, you can use randomly sampled NNs to create various artificial arbitrary 'reward functions' to force an RL agent to explore an environment & learn the dynamics, and then when you give it the real reward function, it learns much faster how to optimize it. Similarly, you can sample random NNs to execute for entire trajectories for 'deep exploration', providing demonstrations of potentially long-range strategies much more efficiently than simple random-action strategies. In 'reservoir sampling', as I understand it, you don't even bother training the NN, you just randomly initialize it and train a simple model on the outputs, assuming that some of the random highly nonlinear relationships encoded into the NN will turn out to be useful, which sounds crazy but apparently works. Makes one think about Tegmark's interpretations.