3 ms·

> ...the authors used the same algorithm, network architecture, and hyperparameters on each game... This is huge. It shows that the algorithm was able to gener

by bmh100 12y ago

> ...the authors used the same algorithm, network architecture, and hyperparameters on each game... This is huge. It shows that the algorithm was able to generalize across multiple problem sets within the same domain of "playing Atari 2600 games", and not simply a "lucky" choice of algorithm, network architecture, or hyperparameters that a random search for each game might choose. This is also not a violation of the No Free Lunch (NFL) Theorem [1], because the domain is limited to playing Atari 2600 games, which share many characteristics. [1]: https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_op...

bsdetector 12y ago
> the algorithm was able to generalize across multiple problem sets Did it really? I think they reset it and retrained it for each game. I'd like to know how much more is needed to make one instance of the AI that can successfully play any of the games. To play all 49 games that it could learn, does it need to be an extra level deep? Or 49 times larger? Or 2^49 times more?
- ZeroFries 12y ago
  I think you could establish an upperbound at 50 times larger, because each set of "neurons" could be completely independent for each game, with 49 neurons to determine which set to use. If they have similar features (which they undoubtedly would), the size could drop a lot. Of course, I have no training in neural nets, and so my conclusions are reached from general understanding/reasoning.
- DavidSJ 12y ago
  What they mean by "the algorithm" is the learning algorithm, not the actual learned behavior. Historically in machine learning a great deal of hand tuning of learning algorithms is required for them to even be effective on a single problem, so it's impressive that a single algorithm was able to learn to play many different games effectively. Having a single instance of an algorithm learn to perform well on many different tasks is really a separate problem than what they're dealing with here.
- meric 12y ago
  Can one human successfully play all of the games without prior practice in each game? As far as I know a human has to practice almost every game to be able to play all of them without losses. I think for an AI, achieving this standard is a good result - first practicing in each game and then play through all of them without losses.
  - bsdetector 12y ago
    A person that learns 1 game will learn the next game much faster, because they have learned a concept such as a bullet or switch or reflection or wrapping. We take this for granted, but there was a time when Breakout was actually marginally fun because it was new. A person that's played all the other games in the list can win Montezuma's Revenge on the first try; this AI can't play Montezuma's Revenge at all.
chriswarbo 12y ago
The No Free Lunch Theorems don't apply in reality anyway, as explained at https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization#NFL_and_Kolmogorov_randomness https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_op... Essentially: NFL says the performance (1 / number of trials are required to find a solution), when averaged over all fitness functions, will be the same for all optimisation algorithms. However, most fitness functions require more computational power than the Universe to evaluate; making even a single trial impossible to perform. Since those problems are already impossible, it doesn't matter how many trials our optimisation algorithm would require when tackling them. We can use this unobservable bad performance to offset some observable good performance, to remain within the average dictated by NFL whilst still becoming objectively better. In fact, one of this paper's authors (Shane Legg) has used this fact to define a general intelligence test which favours performance on simple functions http://arxiv.org/abs/0712.3329 http://arxiv.org/abs/0712.3329

by bmh100 12y ago

bsdetector 12y ago

> the algorithm was able to generalize across multiple problem sets Did it really? I think they reset it and retrained it for each game. I'd like to know how much more is needed to make one instance of the AI that can successfully play any of the games. To play all 49 games that it could learn, does it need to be an extra level deep? Or 49 times larger? Or 2^49 times more?

ZeroFries 12y ago
I think you could establish an upperbound at 50 times larger, because each set of "neurons" could be completely independent for each game, with 49 neurons to determine which set to use. If they have similar features (which they undoubtedly would), the size could drop a lot. Of course, I have no training in neural nets, and so my conclusions are reached from general understanding/reasoning.
DavidSJ 12y ago
What they mean by "the algorithm" is the learning algorithm, not the actual learned behavior. Historically in machine learning a great deal of hand tuning of learning algorithms is required for them to even be effective on a single problem, so it's impressive that a single algorithm was able to learn to play many different games effectively. Having a single instance of an algorithm learn to perform well on many different tasks is really a separate problem than what they're dealing with here.
meric 12y ago
Can one human successfully play all of the games without prior practice in each game? As far as I know a human has to practice almost every game to be able to play all of them without losses. I think for an AI, achieving this standard is a good result - first practicing in each game and then play through all of them without losses.
- bsdetector 12y ago
  A person that learns 1 game will learn the next game much faster, because they have learned a concept such as a bullet or switch or reflection or wrapping. We take this for granted, but there was a time when Breakout was actually marginally fun because it was new. A person that's played all the other games in the list can win Montezuma's Revenge on the first try; this AI can't play Montezuma's Revenge at all.

chriswarbo 12y ago

The No Free Lunch Theorems don't apply in reality anyway, as explained at https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization#NFL_and_Kolmogorov_randomness https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_op... Essentially: NFL says the performance (1 / number of trials are required to find a solution), when averaged over all fitness functions, will be the same for all optimisation algorithms. However, most fitness functions require more computational power than the Universe to evaluate; making even a single trial impossible to perform. Since those problems are already impossible, it doesn't matter how many trials our optimisation algorithm would require when tackling them. We can use this unobservable bad performance to offset some observable good performance, to remain within the average dictated by NFL whilst still becoming objectively better. In fact, one of this paper's authors (Shane Legg) has used this fact to define a general intelligence test which favours performance on simple functions http://arxiv.org/abs/0712.3329 http://arxiv.org/abs/0712.3329