4 ms·

Show HN: AI QLearning Robot Simulation in JavaScript and WebGL

42 points by javierluraschi 12y ago

hmate9 12y ago
Looks very nice, I have one question: How does R2D2 "see" the blocks? Is it by pixels or is the game hard-coded in this sense?
- belltyler 12y ago
  It seems like board is kept as a grid and merely represented with three.js; look into the page's source and you'll see a GameBoard object as well as its logic.
- javierluraschi 12y ago
  Currently, it sees a board of 3x3 states, as in: _ _ G G B B _ _ G A next step would be to use real computer vision over a 3D camera in the environment aligned to R2D@ and a deep neural network to interpret pixels instead of states. DeepMind has a paper describing this approach: http://arxiv.org/pdf/1312.5602.pdf http://arxiv.org/pdf/1312.5602.pdf
dlsym 12y ago
Food / Poison break even was at 88. A chart would be interesting. However - since R2D2 seems like a slow learner it would be nice to speed up the time a little bit.
- hmate9 12y ago
  I am currently at 175 vs 75
- balazsdavid987 12y ago
  After ~30mins I got ratio of 0.77 (375 greens, 111 blacks), it got pretty good at avoiding blacks after reaching 0.7.
- doorhammer 12y ago
  Warning: I only glanced through the code real quickly, so I haven't checked to see if this has unintended side effects to the game logic, or to make sure it's a complete solution. It looks like there's a variable that controls how quickly the board itself is updated: var boardGameUpdateInterval = 10; And a function deciding how quickly the visual representation gets updated: var render = function () { setTimeout( function() { requestAnimationFrame(render); }, 1000 / 15); if (gameTicks % boardGameUpdateInterval == 0) { updateBoard(); } animateBoard(); gameTicks++; renderer.render(scene, camera); }; They're global variables so it's pretty easy to just drop into the developer console and overwrite them. I just changed the denominator of the setTimeout to 30 instead of 15, and the boardGameUpdateInterval to 5 instead of 10. Of course, you don't have to change the render speed to get the logic speedup, but I figured I'd do both. This appears to have sped the simulation speed up by a factor of two. ymmv
- javierluraschi 12y ago
  I've gone as far as ~1000 food ~100 poisson blocks to make sure it's on the right track. Speeding this up is a good idea, I'll make some time to add this and post an update. You can ping me on twitter @javierluraschi.
akyu 12y ago
Very Cool. I did a similar project using a Neural Network instead of a look up table, similar to how deepmind did their Atari game system. The neural network adds some overhead to the training time because of backprop, so for small state spaces which the table can handle easily it is a bit overkill. But the neural network can scale up to much more complex state spaces that would be impractical to use a table for.
- javierluraschi 12y ago
  Do you have a library for your project? It sounds really interesting. I think this space is interesting enough that would benefit from a GutHub project to collaborate.
deleted 12y ago
[deleted]
nrox 12y ago
It looks based on this repo: https://github.com/nrox/q-learning.js https://github.com/nrox/q-learning.js and this example: http://nrox.github.io/q-learning.js/test2.html http://nrox.github.io/q-learning.js/test2.html
- nrox 12y ago
  that makes me proud :) do you have suggestions to improve this library ?

Looks very nice, I have one question: How does R2D2 "see" the blocks? Is it by pixels or is the game hard-coded in this sense?

belltyler 12y ago
It seems like board is kept as a grid and merely represented with three.js; look into the page's source and you'll see a GameBoard object as well as its logic.
javierluraschi 12y ago
Currently, it sees a board of 3x3 states, as in: _ _ G G B B _ _ G A next step would be to use real computer vision over a 3D camera in the environment aligned to R2D@ and a deep neural network to interpret pixels instead of states. DeepMind has a paper describing this approach: http://arxiv.org/pdf/1312.5602.pdf http://arxiv.org/pdf/1312.5602.pdf

dlsym 12y ago

Food / Poison break even was at 88. A chart would be interesting. However - since R2D2 seems like a slow learner it would be nice to speed up the time a little bit.

hmate9 12y ago
I am currently at 175 vs 75
balazsdavid987 12y ago
After ~30mins I got ratio of 0.77 (375 greens, 111 blacks), it got pretty good at avoiding blacks after reaching 0.7.
doorhammer 12y ago
Warning: I only glanced through the code real quickly, so I haven't checked to see if this has unintended side effects to the game logic, or to make sure it's a complete solution. It looks like there's a variable that controls how quickly the board itself is updated: var boardGameUpdateInterval = 10; And a function deciding how quickly the visual representation gets updated: var render = function () { setTimeout( function() { requestAnimationFrame(render); }, 1000 / 15); if (gameTicks % boardGameUpdateInterval == 0) { updateBoard(); } animateBoard(); gameTicks++; renderer.render(scene, camera); }; They're global variables so it's pretty easy to just drop into the developer console and overwrite them. I just changed the denominator of the setTimeout to 30 instead of 15, and the boardGameUpdateInterval to 5 instead of 10. Of course, you don't have to change the render speed to get the logic speedup, but I figured I'd do both. This appears to have sped the simulation speed up by a factor of two. ymmv
javierluraschi 12y ago
I've gone as far as ~1000 food ~100 poisson blocks to make sure it's on the right track. Speeding this up is a good idea, I'll make some time to add this and post an update. You can ping me on twitter @javierluraschi.

akyu 12y ago

Very Cool. I did a similar project using a Neural Network instead of a look up table, similar to how deepmind did their Atari game system. The neural network adds some overhead to the training time because of backprop, so for small state spaces which the table can handle easily it is a bit overkill. But the neural network can scale up to much more complex state spaces that would be impractical to use a table for.

javierluraschi 12y ago
Do you have a library for your project? It sounds really interesting. I think this space is interesting enough that would benefit from a GutHub project to collaborate.

deleted 12y ago

[deleted]

nrox 12y ago

It looks based on this repo: https://github.com/nrox/q-learning.js https://github.com/nrox/q-learning.js and this example: http://nrox.github.io/q-learning.js/test2.html http://nrox.github.io/q-learning.js/test2.html

nrox 12y ago
that makes me proud :) do you have suggestions to improve this library ?