3 ms·

Human-level performance in 3D multiplayer games with population-based RL

58 points by ajay-d 7y ago

isseu 7y ago
I feel problem definition is underappreciated in current ml world and the effort is put into the ml methods. People has been able to tackle almost any problem well defined but these are usually simple in comparison to true ai so we still get "stupid" agents. What problem is the test to true ai?
- yodon 7y ago
  This paper would seem to be a fairly broad based and complex set of capabilities to learn and accomplish, including both collaboration and exceeding the performance of high skill humans at a highly skill based activity. Is there something you feel is missing from the task in this paper?
  - avsteele 7y ago
    How much would the agent's skill level decline, and how much training would be required to recover if, for example, the game were reskinned? How does the skill in this one game mode translate into others? Stating the critique more directly: in what way does the expertise demonstrated mirror the kind of expertise possessed by a similar level human ELO player? If would seem a really shallow kind of expertise, if it didn't translate into some competence in very similar games.
    - yodon 7y ago
      The article discusses a wide range of what constitute significant conceptual mappings discovered autonomously. This work was trained on a single game with a single art direction. The AI can generalize to new maps it has not seen before within that style of art direction, including reliably beating expert human players on maps that are new to both human and AI. That is a significant generalization. The technique should be generalizable to other styles of skinning or different art direction in other games, but there is no way for the the AI to have learned that yet in this experiment because the AI was only ever exposed to a single game here. Over time this technique undoubtedly will be extended to multiple games with different styles of art direction, the challenge there being less on the AI/machine learning side and more on the effort required to hook up multiple games for use in this type of training environment.
dontreact 7y ago
What is deepmind going to do once they solve all video games? It seems like they are leveraging the massive economic demand for entertaining simulations of the world. But once they hit the limit of that they will have to make their own simulations.
- jimmy_dean 7y ago
  Deepmind can always transfer learning to the robotics industry, if they haven't already. There's bound to be a space they can fill with the progress they've made in these video game reinforcement learning tasks. Maybe aiding in medical discoveries?
  - dontreact 7y ago
    The sim to real gap is very large and it honestly seems to me like maybe most of the work is not in the RL part of it but rather instrumenting the robot properly, setting up the right simulation, figuring out how to scale up the real world learning. Haven't seen much robotics stuff come out of Deepmind yet, maybe for these reasons.
    - sjg007 7y ago
      You need algorithms that can learn complex tasks and also reason under uncertainty. Not just uncertainty of sensor inputs but also uncertainty of state. The goal of all of these video games is to develop the algorithms over a wide range of difficulty. So algorithms that have memory, learn / plan and are reasonably robust etc...
- Creationer 7y ago
  AI in an RTS, like Deepmind in Starcraft, can simply issue more commands and react faster than a human - since game information can be gleaned directly from the frame buffer, and instructions issued without going through the analogue of a mouse and keyboard.
  - Mirioron 7y ago
d--b 7y ago
Do they handicap the AI to have sub-perfect aiming? 3D shooters and Quake in particular are aiming games above. If you aim perfectly, you will always win against humans. Adding randomness to aiming is the way to provide fair comparison to give any conclusion regarding true AI features.
- HanayamaTriplet 7y ago
  The ML-based AI (or "agent" as the authors referred to it) in the article never achieved perfect aiming, only up to 80% at close range and 0.5% at long range. After artificially delaying reaction time, a team of one strong human plus one delayed agent was only able to win against a team of two delayed agents only 21% of the time. While this only controlled for reaction time and not accuracy, the authors note that the human and delayed agents had similar numbers of hits per game. Meanwhile, the agents had 4-5x more flag captures than humans, leading them to believe that it had developed superior strategy which carried the games even without god-like aim.
  - de_watcher 7y ago
    > it had developed superior strategy The problem is that when it starts to play against humans the humans develop an anti-strategy after just a several matches. Feeding that back into AI doesn't immediately give enough data points for training.
- ctoth 7y ago
  But did they ... but did they ... But did you ever consider that an AI can just aim better than you period? I'm totally blind. I could enter a marksmanship competition and demand that everybody be made to play by rules that would work for me, but I then wouldn't be able to say that I'm the best marksman if I win.

58 points by ajay-d 7y ago

I feel problem definition is underappreciated in current ml world and the effort is put into the ml methods. People has been able to tackle almost any problem well defined but these are usually simple in comparison to true ai so we still get "stupid" agents. What problem is the test to true ai?

yodon 7y ago
This paper would seem to be a fairly broad based and complex set of capabilities to learn and accomplish, including both collaboration and exceeding the performance of high skill humans at a highly skill based activity. Is there something you feel is missing from the task in this paper?
- avsteele 7y ago
  How much would the agent's skill level decline, and how much training would be required to recover if, for example, the game were reskinned? How does the skill in this one game mode translate into others? Stating the critique more directly: in what way does the expertise demonstrated mirror the kind of expertise possessed by a similar level human ELO player? If would seem a really shallow kind of expertise, if it didn't translate into some competence in very similar games.
  - yodon 7y ago
    The article discusses a wide range of what constitute significant conceptual mappings discovered autonomously. This work was trained on a single game with a single art direction. The AI can generalize to new maps it has not seen before within that style of art direction, including reliably beating expert human players on maps that are new to both human and AI. That is a significant generalization. The technique should be generalizable to other styles of skinning or different art direction in other games, but there is no way for the the AI to have learned that yet in this experiment because the AI was only ever exposed to a single game here. Over time this technique undoubtedly will be extended to multiple games with different styles of art direction, the challenge there being less on the AI/machine learning side and more on the effort required to hook up multiple games for use in this type of training environment.

dontreact 7y ago

What is deepmind going to do once they solve all video games? It seems like they are leveraging the massive economic demand for entertaining simulations of the world. But once they hit the limit of that they will have to make their own simulations.

jimmy_dean 7y ago
Deepmind can always transfer learning to the robotics industry, if they haven't already. There's bound to be a space they can fill with the progress they've made in these video game reinforcement learning tasks. Maybe aiding in medical discoveries?
- dontreact 7y ago
  The sim to real gap is very large and it honestly seems to me like maybe most of the work is not in the RL part of it but rather instrumenting the robot properly, setting up the right simulation, figuring out how to scale up the real world learning. Haven't seen much robotics stuff come out of Deepmind yet, maybe for these reasons.
  - sjg007 7y ago
    You need algorithms that can learn complex tasks and also reason under uncertainty. Not just uncertainty of sensor inputs but also uncertainty of state. The goal of all of these video games is to develop the algorithms over a wide range of difficulty. So algorithms that have memory, learn / plan and are reasonably robust etc...
Creationer 7y ago
AI in an RTS, like Deepmind in Starcraft, can simply issue more commands and react faster than a human - since game information can be gleaned directly from the frame buffer, and instructions issued without going through the analogue of a mouse and keyboard.
- Mirioron 7y ago

d--b 7y ago

Do they handicap the AI to have sub-perfect aiming? 3D shooters and Quake in particular are aiming games above. If you aim perfectly, you will always win against humans. Adding randomness to aiming is the way to provide fair comparison to give any conclusion regarding true AI features.

HanayamaTriplet 7y ago
The ML-based AI (or "agent" as the authors referred to it) in the article never achieved perfect aiming, only up to 80% at close range and 0.5% at long range. After artificially delaying reaction time, a team of one strong human plus one delayed agent was only able to win against a team of two delayed agents only 21% of the time. While this only controlled for reaction time and not accuracy, the authors note that the human and delayed agents had similar numbers of hits per game. Meanwhile, the agents had 4-5x more flag captures than humans, leading them to believe that it had developed superior strategy which carried the games even without god-like aim.
- de_watcher 7y ago
  > it had developed superior strategy The problem is that when it starts to play against humans the humans develop an anti-strategy after just a several matches. Feeding that back into AI doesn't immediately give enough data points for training.
ctoth 7y ago
But did they ... but did they ... But did you ever consider that an AI can just aim better than you period? I'm totally blind. I could enter a marksmanship competition and demand that everybody be made to play by rules that would work for me, but I then wouldn't be able to say that I'm the best marksman if I win.