4 ms·

ConvNetJS Deep Q Learning Demo (2013)

64 points by eigenvalue 3y ago

eigenvalue 3y ago
This is related to the Q* stuff that is making the rounds now. I was quite impressed with this when I first saw it years ago. The little 2D robot is able to learn sophisticated looking behaviors with just reward based reinforcement. If you find this interesting, you should also check out Karpathy’s ScriptBots project from back when he was a college student: https://youtu.be/RjweUYtpNq4?si=IGUwl7cZ47V2DSGB https://youtu.be/RjweUYtpNq4?si=IGUwl7cZ47V2DSGB
- quickthrower2 3y ago
  Interesting that back then he still had that mix of (a) is genius (or at least very smart) and (b) can and is willing to explain things in a way that is entertaining, simple, fun.
  - eigenvalue 3y ago
    He has been a legit superstar since college. No one was surprised that he went on to do great things.
    - quickthrower2 3y ago
      Absolutely. I wasn't saying it that way round. More that the presentation skills were as impeccable back then as they are now. Which may not be surprising but it made me happy to watch. He is one of the few "wormhole" people who understands the hard stuff, but can teach some of it to the rest of us! P.S. username checks out.
- nielsinho 3y ago
  I stumbled across ScriptBots when I was a kid, and became obsessed with implementing my own variants, and subsequently with neural networks. This one [1] is close to Karpathy's original, and this one [2] turns the world 3D and the beings are slightly more advanced. Both are implemented in MoonScript/Lua and run with Love2D. In part due to this this YouTube video, I work as a Machine Learning Engineer today. [1] https://github.com/nilq/genesis https://github.com/nilq/genesis [2] https://github.com/nilq/plane https://github.com/nilq/plane
  - eigenvalue 3y ago
    Yes, I also became somewhat obsessed with ScriptBots when it first came out. I remember re-compiling it from source using the Intel C++ compiler and getting a pretty massive speedup on my machine. It was the first time I really dug into a fairly complex program. Even now I think it's probably too heavy to work well in JS.
aliljet 3y ago
This is incredibly interesting. Why wasn't this seen as fruitful over the last decade?
- smuser 3y ago
  My understanding (not an expert) is a lot of problem domains have very sparse / infrequent rewards - imagine if the only reward you gave a minecraft agent was when it mined a diamond, it would take a lot of gameplay for it to randomly do that and get a reward. So researchers spend time tuning the reward space (oh you mined some dirt, here's a tiny reward. Oh you mined rock, a greater reward, etc) but it's kind of akin to hand crafted feature detection from the pre-neural network days. The Q* mystery is did OpenAI 'solve' reward modelling the same way neural networks solved feature detection.
  - pyinstallwoes 3y ago
    Reward for a successful prediction against a goal, then the nuance is defining a goal?
  - throwaway4aday 3y ago
    Sounds like the process of tuning the reward space is a type of labelling and ranking problem. If I'm not mistaken, those are two things that GPT-4 is pretty good at. You wouldn't even necessarily pre-label every possible action since GPT-4 could do it in real time.
- eigenvalue 3y ago
  It was. Around the time this came out, something like half of the new ML papers were about reinforcement learning. The problem is that it’s incredibly slow and inefficient compared to any learning where you have access to a gradient and can use that to choose more targeted weight updates. But there are certain applications where it’s the only good way of doing it (for example in games, where you don’t have access to a gradient over the space of how good a certain move is given the current game state, and it’s relatively quick and and efficient to simulate the evolution of the game).
mortallywounded 3y ago
When I was in grad school and implemented Q-learning for the first time, was the moment when machine learning "clicked" for me. It went from being a "oh this algorithm is pretty cool, but it doesn't learn anything" to "oh my god, the program figured things out on its own." I've always had a soft spot for q-learning ever since :) edit: In grad school, I implemented q-learning to learn strategies for playing games (like tic-tac-toe, towers of hanoi, etc).
jwilber 3y ago
For a very high-level, visual introduction, see: https://mlu-explain.github.io/reinforcement-learning/ https://mlu-explain.github.io/reinforcement-learning/
- bloopernova 3y ago
  I found that helpful, thank you for sharing it.
ffwd 3y ago
I don't know much about RL but I was wondering has anyone tried the opposite? Like have a fixed set of actions, and a fixed/ranged movement speed, then a punishment every time it doesn't reach the goal? Or does this not work?
- eigenvalue 3y ago
  That’s just a negative reward, so basically the same thing. You train most efficiently with a mix of positive and negative reinforcement, just like with children.
  - ffwd 3y ago
    Thanks, I can't write a full reply now (need to think) but for some reason my intuition was, let's say you have a constant punishment signal, and a timer, and if it doesn't solve whatever problem by the time the timer goes down, then it has to find the optimal action set, and if it reaches a goal earlier than the timer then it has to weigh that solution stronger? Like at least how I see with organisms it's about finding the optimal use of the limbs in order to solve an ongoing problem/goal state, and if you accumulate specific actions (instead of one network that optimizes one "space"), and different types of goal states, then it has to find the optimal set of actions to reach the different goals, it just seemed more efficient. But this is off the cuff a bit right now. Edit: i think what i'm thinking of are two "global" numbers. A "closer to the goal" number (higher when closer), and a countdown timer, and then it has to maximize those within the above setting
    - wegfawefgawefg 3y ago
      if you only have negative rewards the bot will usually stop picking any actions at all. bc they all have reward values too low. sometimes a big negstive reward can scare it away from ever pressing the scary button ever again
- quickthrower2 3y ago
  I don't think these nets are sophisticated enough to understand complexities of what humans call punishment / reward. Therefore punishment = -reward, but this is not the case for humans. If punishment for example is prison, is reward making people "even more free" for example giving them a flying car or something :-). If money is a reward (or punishment a fine), why are people not optimizing every last cent.
PodgieTar 3y ago
I implemented Q Learning in University. Only very basic, sort of like the Mario style Q-Learning. It's quite intuitive, as a concept. You do need to know the expected outputs - positive rewards and negative.
born-jre 3y ago
if openapi is doing q* stuff, then Video by AI Explained sounds most plausible to me. https://www.youtube.com/watch?v=ARf0WyFau0A https://www.youtube.com/watch?v=ARf0WyFau0A

64 points by eigenvalue 3y ago

This is related to the Q* stuff that is making the rounds now. I was quite impressed with this when I first saw it years ago. The little 2D robot is able to learn sophisticated looking behaviors with just reward based reinforcement. If you find this interesting, you should also check out Karpathy’s ScriptBots project from back when he was a college student: https://youtu.be/RjweUYtpNq4?si=IGUwl7cZ47V2DSGB https://youtu.be/RjweUYtpNq4?si=IGUwl7cZ47V2DSGB

quickthrower2 3y ago
Interesting that back then he still had that mix of (a) is genius (or at least very smart) and (b) can and is willing to explain things in a way that is entertaining, simple, fun.
- eigenvalue 3y ago
  He has been a legit superstar since college. No one was surprised that he went on to do great things.
  - quickthrower2 3y ago
    Absolutely. I wasn't saying it that way round. More that the presentation skills were as impeccable back then as they are now. Which may not be surprising but it made me happy to watch. He is one of the few "wormhole" people who understands the hard stuff, but can teach some of it to the rest of us! P.S. username checks out.
nielsinho 3y ago
I stumbled across ScriptBots when I was a kid, and became obsessed with implementing my own variants, and subsequently with neural networks. This one [1] is close to Karpathy's original, and this one [2] turns the world 3D and the beings are slightly more advanced. Both are implemented in MoonScript/Lua and run with Love2D. In part due to this this YouTube video, I work as a Machine Learning Engineer today. [1] https://github.com/nilq/genesis https://github.com/nilq/genesis [2] https://github.com/nilq/plane https://github.com/nilq/plane
- eigenvalue 3y ago
  Yes, I also became somewhat obsessed with ScriptBots when it first came out. I remember re-compiling it from source using the Intel C++ compiler and getting a pretty massive speedup on my machine. It was the first time I really dug into a fairly complex program. Even now I think it's probably too heavy to work well in JS.

aliljet 3y ago

This is incredibly interesting. Why wasn't this seen as fruitful over the last decade?

smuser 3y ago
My understanding (not an expert) is a lot of problem domains have very sparse / infrequent rewards - imagine if the only reward you gave a minecraft agent was when it mined a diamond, it would take a lot of gameplay for it to randomly do that and get a reward. So researchers spend time tuning the reward space (oh you mined some dirt, here's a tiny reward. Oh you mined rock, a greater reward, etc) but it's kind of akin to hand crafted feature detection from the pre-neural network days. The Q* mystery is did OpenAI 'solve' reward modelling the same way neural networks solved feature detection.
- pyinstallwoes 3y ago
  Reward for a successful prediction against a goal, then the nuance is defining a goal?
- throwaway4aday 3y ago
  Sounds like the process of tuning the reward space is a type of labelling and ranking problem. If I'm not mistaken, those are two things that GPT-4 is pretty good at. You wouldn't even necessarily pre-label every possible action since GPT-4 could do it in real time.
eigenvalue 3y ago
It was. Around the time this came out, something like half of the new ML papers were about reinforcement learning. The problem is that it’s incredibly slow and inefficient compared to any learning where you have access to a gradient and can use that to choose more targeted weight updates. But there are certain applications where it’s the only good way of doing it (for example in games, where you don’t have access to a gradient over the space of how good a certain move is given the current game state, and it’s relatively quick and and efficient to simulate the evolution of the game).

mortallywounded 3y ago

When I was in grad school and implemented Q-learning for the first time, was the moment when machine learning "clicked" for me. It went from being a "oh this algorithm is pretty cool, but it doesn't learn anything" to "oh my god, the program figured things out on its own." I've always had a soft spot for q-learning ever since :) edit: In grad school, I implemented q-learning to learn strategies for playing games (like tic-tac-toe, towers of hanoi, etc).

jwilber 3y ago

For a very high-level, visual introduction, see: https://mlu-explain.github.io/reinforcement-learning/ https://mlu-explain.github.io/reinforcement-learning/

bloopernova 3y ago
I found that helpful, thank you for sharing it.

ffwd 3y ago

I don't know much about RL but I was wondering has anyone tried the opposite? Like have a fixed set of actions, and a fixed/ranged movement speed, then a punishment every time it doesn't reach the goal? Or does this not work?

eigenvalue 3y ago
That’s just a negative reward, so basically the same thing. You train most efficiently with a mix of positive and negative reinforcement, just like with children.
- ffwd 3y ago
  Thanks, I can't write a full reply now (need to think) but for some reason my intuition was, let's say you have a constant punishment signal, and a timer, and if it doesn't solve whatever problem by the time the timer goes down, then it has to find the optimal action set, and if it reaches a goal earlier than the timer then it has to weigh that solution stronger? Like at least how I see with organisms it's about finding the optimal use of the limbs in order to solve an ongoing problem/goal state, and if you accumulate specific actions (instead of one network that optimizes one "space"), and different types of goal states, then it has to find the optimal set of actions to reach the different goals, it just seemed more efficient. But this is off the cuff a bit right now. Edit: i think what i'm thinking of are two "global" numbers. A "closer to the goal" number (higher when closer), and a countdown timer, and then it has to maximize those within the above setting
  - wegfawefgawefg 3y ago
    if you only have negative rewards the bot will usually stop picking any actions at all. bc they all have reward values too low. sometimes a big negstive reward can scare it away from ever pressing the scary button ever again
quickthrower2 3y ago
I don't think these nets are sophisticated enough to understand complexities of what humans call punishment / reward. Therefore punishment = -reward, but this is not the case for humans. If punishment for example is prison, is reward making people "even more free" for example giving them a flying car or something :-). If money is a reward (or punishment a fine), why are people not optimizing every last cent.

PodgieTar 3y ago

I implemented Q Learning in University. Only very basic, sort of like the Mario style Q-Learning. It's quite intuitive, as a concept. You do need to know the expected outputs - positive rewards and negative.

born-jre 3y ago

if openapi is doing q* stuff, then Video by AI Explained sounds most plausible to me. https://www.youtube.com/watch?v=ARf0WyFau0A https://www.youtube.com/watch?v=ARf0WyFau0A