4 ms·

I’m by no means an expert, but I’m fascinated by the idea that a neural net playing against itself can substantially outperform a supervised learning approach w

I’m by no means an expert, but I’m fascinated by the idea that a neural net playing against itself can substantially outperform a supervised learning approach with a large training data set. I mean, gathering training data and making sure it’s labeled correctly and all that is a huge hassle so if you could eliminate that step or even reduce the amount or quality of training data required that should be a big win for AI, right? Especially if doing this not only makes things easier but also improves the performance of the model.

NamTaf 9y ago
Reinforcement learning isn't a new idea - I did a Berkeley-based edX course on it a few years ago now and it was not state-of-the-art to my knowledge. That had no deep aspect to it, we just generated a reinfrocement algorithm that utilised a good measure of performance (specifically, it was pacman and the score value is pretty good at that) and changed a few algorithm weighting variables at each iteration. My understanding, from talking to a ML friend this morning, is that the latest progress is taking reinforcement learning and applying deep learning approaches (nets, etc.) to it. The key becomes finding the right scoring algorithms to tweak the neural net correctly towards the desired outcome. The self-play really is the reinforcement side of things at work. How you take that 'score' and use it to correctly modify the input weightings - be them in a neural net, traditional algorithm, etc. - is the key.
- noway421 9y ago
  Why wouldn't algorithm reach a local maxima when playing with itself, or even degrade over time by opening up to unknown attacks?
  - gcp 9y ago
    One typically keeps pools of trained networks to combat this.
  - Epokhe 9y ago
    it does. the bot can be defeated easily with tactics it didn't encounter while playing with itself.
- mannykannot 9y ago
  > The key becomes finding the right scoring algorithms to tweak the neural net correctly towards the desired outcome. Does this not become something similar to supervised learning if you are scoring internal states of the game? (i.e. scoring on more than just the outcome and things that violate the rules?)
  - NamTaf 9y ago
    I don't know what they're scoring on; they've not elaborated on that. However, when I first read into OpenAI it looked like many of their game links were to simply view the screen. To that end, it wouldn't surprise me if they're identifying things you'd otherwise visually see in the game and simply working off them. To use internal states not immediately visible to the player would be sort of disingenuous IMO.
tictacttoe 9y ago
It's very cool, but I think it also requires a very specific "adversarial" problem with a well defined notion of success (winning the game). If your machine learning task is something more nuanced and harder to define, e.g. identify word synonyms, I don't see how you can get around having a training data set.
- kpil 9y ago
  I played with evolving algorithms playing some kind of robot war (in java) almost 20 years ago, and beside the fact that you had to be veeery patient since evolving a generation took 10 minutes at best, I realized that it's very hard to create a survival criteria that matches your intention. I e, most robots fled and hid in the corners. I added additional critiera for hitting enemies, and the robots fled while randomly shooting bullets and hid in the corners... No epic fights. Hmm. Maybe I have the code somewhere...
pycal 9y ago
Just a guess, but wouldn't this approach used in isolation result in a high potential for getting stuck at local optima? It seems to me that the success of this experiment is the result of interaction with players in different MMR brackets.
- Bartweiss 9y ago
  Yes, it can. It's fairly common to keep a pool of learners which aren't identical for exactly this reason - either playing against prior snapshots, forking training, or lightly randomizing some components of the learner.

NamTaf 9y ago

Reinforcement learning isn't a new idea - I did a Berkeley-based edX course on it a few years ago now and it was not state-of-the-art to my knowledge. That had no deep aspect to it, we just generated a reinfrocement algorithm that utilised a good measure of performance (specifically, it was pacman and the score value is pretty good at that) and changed a few algorithm weighting variables at each iteration. My understanding, from talking to a ML friend this morning, is that the latest progress is taking reinforcement learning and applying deep learning approaches (nets, etc.) to it. The key becomes finding the right scoring algorithms to tweak the neural net correctly towards the desired outcome. The self-play really is the reinforcement side of things at work. How you take that 'score' and use it to correctly modify the input weightings - be them in a neural net, traditional algorithm, etc. - is the key.

noway421 9y ago
Why wouldn't algorithm reach a local maxima when playing with itself, or even degrade over time by opening up to unknown attacks?
- gcp 9y ago
  One typically keeps pools of trained networks to combat this.
- Epokhe 9y ago
  it does. the bot can be defeated easily with tactics it didn't encounter while playing with itself.
mannykannot 9y ago
> The key becomes finding the right scoring algorithms to tweak the neural net correctly towards the desired outcome. Does this not become something similar to supervised learning if you are scoring internal states of the game? (i.e. scoring on more than just the outcome and things that violate the rules?)
- NamTaf 9y ago
  I don't know what they're scoring on; they've not elaborated on that. However, when I first read into OpenAI it looked like many of their game links were to simply view the screen. To that end, it wouldn't surprise me if they're identifying things you'd otherwise visually see in the game and simply working off them. To use internal states not immediately visible to the player would be sort of disingenuous IMO.

tictacttoe 9y ago

It's very cool, but I think it also requires a very specific "adversarial" problem with a well defined notion of success (winning the game). If your machine learning task is something more nuanced and harder to define, e.g. identify word synonyms, I don't see how you can get around having a training data set.

kpil 9y ago
I played with evolving algorithms playing some kind of robot war (in java) almost 20 years ago, and beside the fact that you had to be veeery patient since evolving a generation took 10 minutes at best, I realized that it's very hard to create a survival criteria that matches your intention. I e, most robots fled and hid in the corners. I added additional critiera for hitting enemies, and the robots fled while randomly shooting bullets and hid in the corners... No epic fights. Hmm. Maybe I have the code somewhere...

pycal 9y ago

Just a guess, but wouldn't this approach used in isolation result in a high potential for getting stuck at local optima? It seems to me that the success of this experiment is the result of interaction with players in different MMR brackets.

Bartweiss 9y ago
Yes, it can. It's fairly common to keep a pool of learners which aren't identical for exactly this reason - either playing against prior snapshots, forking training, or lightly randomizing some components of the learner.