4 ms·

In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. “What are you doing?”, asked Minsky. “I am training a randomly

In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. “What are you doing?”, asked Minsky. “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied. “Why is the net wired randomly?”, asked Minsky. “I do not want it to have any preconceptions of how to play”, Sussman said. Minsky then shut his eyes. “Why do you close your eyes?”, Sussman asked his teacher. “So that the room will be empty.” At that moment, Sussman was enlightened. "picking the right reward function" in RL is shockingly hard. It actually works OK-ish when the problem space is strictly bounded, like with a game whose rules are known. After that, you start getting into sky-humping cheetah problems:https://www.alexirpan.com/public/rl-hard/upsidedown_half_cheetah.mp4 https://www.alexirpan.com/public/rl-hard/upsidedown_half_che... https://www.alexirpan.com/2018/02/14/rl-hard.html https://www.alexirpan.com/2018/02/14/rl-hard.html is a better article, perhaps, than this one.

mockingbirdy 8y ago
> https://www.alexirpan.com/public/rl-hard/upsidedown_half_cheetah.mp4 https://www.alexirpan.com/public/rl-hard/upsidedown_half_che... You can argue, but it works. Goal accomplished. Nature did things that are stranger than that. Now you just have to add efficiency aspects into your reward functions and observe how it slowly finds a local minimum. Also something nature did a lot of times. Now hope that the remaining inefficiency is ok for you. Done. - You can start with a random neural net. It's not exactly empty, but it's ok. The randomness defines which local minimum you'll find this time around while you burn through your VC money desperately hoping that the AWS bills for all the GPU time will arrive after you've found the holy grail that makes your startup worth <insert bullshit valuation>. > "picking the right reward function" in RL is shockingly hard. Agree. x1000. And computational power and data quality. - on another post I've added some links (AI playing Mario): https://news.ycombinator.com/item?id=17489459 https://news.ycombinator.com/item?id=17489459
- gwern 8y ago
  > You can start with a random neural net. It's not exactly empty, but it's ok. One interesting thing about random NNs is how much they can already do. For example, you can do single-shot image inpainting or superresolution with an untrained randomly-wired CNN: https://dmitryulyanov.github.io/deep_image_prior https://dmitryulyanov.github.io/deep_image_prior In RL, you can use randomly sampled NNs to create various artificial arbitrary 'reward functions' to force an RL agent to explore an environment & learn the dynamics, and then when you give it the real reward function, it learns much faster how to optimize it. Similarly, you can sample random NNs to execute for entire trajectories for 'deep exploration', providing demonstrations of potentially long-range strategies much more efficiently than simple random-action strategies. In 'reservoir sampling', as I understand it, you don't even bother training the NN, you just randomly initialize it and train a simple model on the outputs, assuming that some of the random highly nonlinear relationships encoded into the NN will turn out to be useful, which sounds crazy but apparently works. Makes one think about Tegmark's interpretations.
denzil_correa 8y ago
I have been searching for this Marvin Minsky quote for a long time. Thank you for posting it again!
- aktenlage 8y ago
  I don't get it, could you please explain it?
  - AnimalMuppet 8y ago
    If you wire the neural net randomly, it still has preconceptions about how to play - you just don't know what they are. You're just closing your eyes to the initial wiring, but that doesn't make the initial wiring non-existent (in the same way that the room did not become empty when Minsky closed his eyes).
    - aktenlage 8y ago
      Thanks!
gowld 8y ago
Re sky cheetah: yes, if you ask a stupid question, you get a stupid answer. That cheetah model is oversimplified and bad -- it has a heard but doesn't have any requirements or objectives for the head,
- hprotagonist 8y ago
  my point re: sky cheetah is that there are large domains where the only questions you can ask are stupid —- which is to say, ill-posed. in those situations you cannot expect RL to come up with “normal” solutions that do what you actually want. The sky-humping is a totally valid answer to the question asked; it’s merely that the reward function for “walk forward” doesn’t (and in many situations, may not ever) fully constrain the search space such that you get sane solutions.
jaytaylor 8y ago
The rl-hard.html article was discussed on HN only four months ago: https://news.ycombinator.com/item?id=16383264 https://news.ycombinator.com/item?id=16383264 Makes for an interesting trail of additional breadcrumbs to wind into the stack :)

mockingbirdy 8y ago

> https://www.alexirpan.com/public/rl-hard/upsidedown_half_cheetah.mp4 https://www.alexirpan.com/public/rl-hard/upsidedown_half_che... You can argue, but it works. Goal accomplished. Nature did things that are stranger than that. Now you just have to add efficiency aspects into your reward functions and observe how it slowly finds a local minimum. Also something nature did a lot of times. Now hope that the remaining inefficiency is ok for you. Done. - You can start with a random neural net. It's not exactly empty, but it's ok. The randomness defines which local minimum you'll find this time around while you burn through your VC money desperately hoping that the AWS bills for all the GPU time will arrive after you've found the holy grail that makes your startup worth <insert bullshit valuation>. > "picking the right reward function" in RL is shockingly hard. Agree. x1000. And computational power and data quality. - on another post I've added some links (AI playing Mario): https://news.ycombinator.com/item?id=17489459 https://news.ycombinator.com/item?id=17489459

gwern 8y ago
> You can start with a random neural net. It's not exactly empty, but it's ok. One interesting thing about random NNs is how much they can already do. For example, you can do single-shot image inpainting or superresolution with an untrained randomly-wired CNN: https://dmitryulyanov.github.io/deep_image_prior https://dmitryulyanov.github.io/deep_image_prior In RL, you can use randomly sampled NNs to create various artificial arbitrary 'reward functions' to force an RL agent to explore an environment & learn the dynamics, and then when you give it the real reward function, it learns much faster how to optimize it. Similarly, you can sample random NNs to execute for entire trajectories for 'deep exploration', providing demonstrations of potentially long-range strategies much more efficiently than simple random-action strategies. In 'reservoir sampling', as I understand it, you don't even bother training the NN, you just randomly initialize it and train a simple model on the outputs, assuming that some of the random highly nonlinear relationships encoded into the NN will turn out to be useful, which sounds crazy but apparently works. Makes one think about Tegmark's interpretations.

denzil_correa 8y ago

I have been searching for this Marvin Minsky quote for a long time. Thank you for posting it again!

aktenlage 8y ago
I don't get it, could you please explain it?
- AnimalMuppet 8y ago
  If you wire the neural net randomly, it still has preconceptions about how to play - you just don't know what they are. You're just closing your eyes to the initial wiring, but that doesn't make the initial wiring non-existent (in the same way that the room did not become empty when Minsky closed his eyes).
  - aktenlage 8y ago
    Thanks!

gowld 8y ago

Re sky cheetah: yes, if you ask a stupid question, you get a stupid answer. That cheetah model is oversimplified and bad -- it has a heard but doesn't have any requirements or objectives for the head,

hprotagonist 8y ago
my point re: sky cheetah is that there are large domains where the only questions you can ask are stupid —- which is to say, ill-posed. in those situations you cannot expect RL to come up with “normal” solutions that do what you actually want. The sky-humping is a totally valid answer to the question asked; it’s merely that the reward function for “walk forward” doesn’t (and in many situations, may not ever) fully constrain the search space such that you get sane solutions.

jaytaylor 8y ago

The rl-hard.html article was discussed on HN only four months ago: https://news.ycombinator.com/item?id=16383264 https://news.ycombinator.com/item?id=16383264 Makes for an interesting trail of additional breadcrumbs to wind into the stack :)