5 ms·

As requested, I put up a demo of neural slime volleyball without any pre-trained agents so you can see them improve gradually as they evolve: http://otoro.net/

As requested, I put up a demo of neural slime volleyball without any pre-trained agents so you can see them improve gradually as they evolve: http://otoro.net/slimevolley/training.html http://otoro.net/slimevolley/training.html Basically, I left the computer running for a day or so, and came back realising that they have become quite good. But a day of training on a macbook pro is probably equivalent of a _few real decades_ of simulation at 30 frames per second, so to answer someone's question is that evolutionary approach will probably not work well when training with actual human players in human-time. Other approaches such as deep-q-learning may be more suitable. But you gotta wonder about the power and possibilities of simulating time at super human speeds to artificially evolve cool stuff!

sahilwasan 11y ago
Hi hardmaru. Thats great stuff. I always want to learn AI and ML. But do not know where to start. Can you please guide where to start and how can you done this. Thanks
- hardmaru 11y ago
  Hi Sahilwasan If you want to learn I found it is best to get your hands dirty and code rather than just watch videos from MOOCs. I didn't know how to do any of this stuff a few months ago. I recommend Karpathy's Hacker's Guide to Neural Networks if you like JS http://karpathy.github.io/neuralnets/ http://karpathy.github.io/neuralnets/ Most of my intro to neural nets is just by going thru programming examples in this stanford tutorial on deep learning: http://deeplearning.stanford.edu/wiki/index.php/UFLDL_Tutorial http://deeplearning.stanford.edu/wiki/index.php/UFLDL_Tutori...
deepnet 11y ago
Mindblowing that the trained agents are 10s of years old - this puts their skill in perspective. Very interesting that we humans may be too 'realtime' to train evolving agents. Do your plank avoiding creatures evolve faster ? http://otoro.net/ml/planks/index.html http://otoro.net/ml/planks/index.html Watching the creatures avoiding planks it is hard to reconcile the deterministic mathematics with the seeming agency and 'beingness' of the simulated life.
- hardmaru 11y ago
  The creatures evolved much faster! If you run the training demo at http://otoro.net/planks/training.html http://otoro.net/planks/training.html I noticed that after 50-100 generations they develop insect like characteristics. But for the slime agents, they needed much evolution. I think when I did it I actually had them evolve on the easy version first, and then increased the area size to evolve them on the larger play area. This sort of step by step evolution strategy was covered in some neuroevolution papers I read. I also applied it to the pendulum example.
hardmaru 11y ago
Posted a small write up here with regards to the evolution setup used: http://blog.otoro.net/2015/05/13/neural-slime-volleyball-evolution-demo/ http://blog.otoro.net/2015/05/13/neural-slime-volleyball-evo...
deepnet 11y ago
While Deep-Q-Learning is powerful, it uses backprop - your use of CNE genetic evolution rather than backprop may provide a more global search than Mnih's Deep-Q. The methods seem complementary , I have an inkling that the combination may be greater than the sum of the parts. Another innovation in the Atari paper, Mnih's deep-Q-agent remembers successful episodes and trains more from memory than experience. Of course they are not training against a human player but perhaps this memory training could be used with human games to exploit expert knowledge. The recent Neural Net Go players by Edinburgh's Clarke & Storkey group and Deepmind's Maddison & Huang both train on corpii of expert human play. But the Go players are passive backprop learners. Tesauro's TD-Gammon learned to expert level through self play. I am speculating that a Reinforcement Learner could learn to utilise the benefits of human play with memory, expert corpii and self-play. This is a fascinating parallel area - two equal players rather than Mnih's single player Atari - perhaps there are good 2-player Atari games that could be added to the ALE benchmark - Joust ?
- deepnet 11y ago
  *references Teaching Deep Convolutional Neural Networks to Play Go by Clarke & Storkey http://arxiv.org/abs/1412.3409 http://arxiv.org/abs/1412.3409 Move Evaluation in Go Using Deep Convolutional Neural Networks by Maddison , Huang , Silver & Sutskever http://arxiv.org/abs/1412.6564 http://arxiv.org/abs/1412.6564 Tesauro's TD-Gammon http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node108.html http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node108.htm... Playing Atari with Deep Reinforcement Learning , Mnih et al. https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf Otoro's CNE Bibliography http://blog.otoro.net/2015/01/27/neuroevolution-algorithms/ http://blog.otoro.net/2015/01/27/neuroevolution-algorithms/ Neuro Evolution - http://en.wikipedia.org/wiki/Neuroevolution http://en.wikipedia.org/wiki/Neuroevolution
  - hardmaru 11y ago
    Thanks for the references So you think basically recording human actions (possibly in some feature extracted way) in part of the 'replay memory' for DQN would work well? I have also been reading about deepmind's recent survey of combining deep learning methods with actor critic models What I also want to explore is the possibility to use evolution to evolve q-functions, which shouldn't be so hard to do, rather than evolve policies directly (like in this game). The possibility to evolve self learning machines excite me, rather than just evolving machines in a fixed state. I can also explore whether Darwinian evolution (weights are randomized at birth) is better or worst than Lamarckism evolution where weights are passed to offsprings for. I put some further thoughts and references in my post here http://blog.otoro.net/2015/04/08/evolutionary-function-approximation-for-reinforcement-learning/ http://blog.otoro.net/2015/04/08/evolutionary-function-appro... The surviving agents will be born with a better capacity to learn rather than the capacity to do a predefined job. Stay tuned!

sahilwasan 11y ago

Hi hardmaru. Thats great stuff. I always want to learn AI and ML. But do not know where to start. Can you please guide where to start and how can you done this. Thanks

hardmaru 11y ago
Hi Sahilwasan If you want to learn I found it is best to get your hands dirty and code rather than just watch videos from MOOCs. I didn't know how to do any of this stuff a few months ago. I recommend Karpathy's Hacker's Guide to Neural Networks if you like JS http://karpathy.github.io/neuralnets/ http://karpathy.github.io/neuralnets/ Most of my intro to neural nets is just by going thru programming examples in this stanford tutorial on deep learning: http://deeplearning.stanford.edu/wiki/index.php/UFLDL_Tutorial http://deeplearning.stanford.edu/wiki/index.php/UFLDL_Tutori...

deepnet 11y ago

Mindblowing that the trained agents are 10s of years old - this puts their skill in perspective. Very interesting that we humans may be too 'realtime' to train evolving agents. Do your plank avoiding creatures evolve faster ? http://otoro.net/ml/planks/index.html http://otoro.net/ml/planks/index.html Watching the creatures avoiding planks it is hard to reconcile the deterministic mathematics with the seeming agency and 'beingness' of the simulated life.

hardmaru 11y ago
The creatures evolved much faster! If you run the training demo at http://otoro.net/planks/training.html http://otoro.net/planks/training.html I noticed that after 50-100 generations they develop insect like characteristics. But for the slime agents, they needed much evolution. I think when I did it I actually had them evolve on the easy version first, and then increased the area size to evolve them on the larger play area. This sort of step by step evolution strategy was covered in some neuroevolution papers I read. I also applied it to the pendulum example.

hardmaru 11y ago

Posted a small write up here with regards to the evolution setup used: http://blog.otoro.net/2015/05/13/neural-slime-volleyball-evolution-demo/ http://blog.otoro.net/2015/05/13/neural-slime-volleyball-evo...

deepnet 11y ago

While Deep-Q-Learning is powerful, it uses backprop - your use of CNE genetic evolution rather than backprop may provide a more global search than Mnih's Deep-Q. The methods seem complementary , I have an inkling that the combination may be greater than the sum of the parts. Another innovation in the Atari paper, Mnih's deep-Q-agent remembers successful episodes and trains more from memory than experience. Of course they are not training against a human player but perhaps this memory training could be used with human games to exploit expert knowledge. The recent Neural Net Go players by Edinburgh's Clarke & Storkey group and Deepmind's Maddison & Huang both train on corpii of expert human play. But the Go players are passive backprop learners. Tesauro's TD-Gammon learned to expert level through self play. I am speculating that a Reinforcement Learner could learn to utilise the benefits of human play with memory, expert corpii and self-play. This is a fascinating parallel area - two equal players rather than Mnih's single player Atari - perhaps there are good 2-player Atari games that could be added to the ALE benchmark - Joust ?

deepnet 11y ago
*references Teaching Deep Convolutional Neural Networks to Play Go by Clarke & Storkey http://arxiv.org/abs/1412.3409 http://arxiv.org/abs/1412.3409 Move Evaluation in Go Using Deep Convolutional Neural Networks by Maddison , Huang , Silver & Sutskever http://arxiv.org/abs/1412.6564 http://arxiv.org/abs/1412.6564 Tesauro's TD-Gammon http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node108.html http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node108.htm... Playing Atari with Deep Reinforcement Learning , Mnih et al. https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf Otoro's CNE Bibliography http://blog.otoro.net/2015/01/27/neuroevolution-algorithms/ http://blog.otoro.net/2015/01/27/neuroevolution-algorithms/ Neuro Evolution - http://en.wikipedia.org/wiki/Neuroevolution http://en.wikipedia.org/wiki/Neuroevolution
- hardmaru 11y ago
  Thanks for the references So you think basically recording human actions (possibly in some feature extracted way) in part of the 'replay memory' for DQN would work well? I have also been reading about deepmind's recent survey of combining deep learning methods with actor critic models What I also want to explore is the possibility to use evolution to evolve q-functions, which shouldn't be so hard to do, rather than evolve policies directly (like in this game). The possibility to evolve self learning machines excite me, rather than just evolving machines in a fixed state. I can also explore whether Darwinian evolution (weights are randomized at birth) is better or worst than Lamarckism evolution where weights are passed to offsprings for. I put some further thoughts and references in my post here http://blog.otoro.net/2015/04/08/evolutionary-function-approximation-for-reinforcement-learning/ http://blog.otoro.net/2015/04/08/evolutionary-function-appro... The surviving agents will be born with a better capacity to learn rather than the capacity to do a predefined job. Stay tuned!