3 ms·

Isn't it the case that many behaviors had to be specifically encouraged or coded into the bots to get them to work at all? They learned an impressive amount on

Isn't it the case that many behaviors had to be specifically encouraged or coded into the bots to get them to work at all? They learned an impressive amount on their own after being pushed in the right direction, but this is far from a pure random reinforcement learning approach as this article suggests. This article also really understates how clowny the bots were playing. They appeared confused and aimless for much of the game, until they found a clear objective like an enemy hero that moved too close to them, or they had a numbers advantage that allowed them to pressure an enemy building. They were brutal at closing gaps and pursuing enemies to kill without hesitation, but didn't perform well at all beyond that. This isn't to take away from the accomplishments of OpenAI - to get to this stage after 18 months of work is an impressive engineering feat. It's just telling that the bots have only learned one single strategy, don't even execute that very well, and still have to play with hero restrictions that rule out counters to that specific strategy of aggressive skirmishing and pushing.

tlb 8y ago
Looking "clowny" isn't something that RL agents learn to avoid. You see it even in Go agents. The MCTS algorithm looks at probability of winning, and if it's ~100% regardless of the next move, it'll often waste moves rather than trying to win as quickly as possible.
SOLAR_FIELDS 8y ago
My understanding is that there are behaviors that are coded into the bots at first to bootstrap their learning capabilities, which are then removed after a certain amount of learning has taken place. In another article that was posted here a few days ago one of the OpenAI team talks about removing the last of these "scripted" capabilities as part of the improvements they plan on making to the bots after this round of games against human players. I kind of liken it to a human who learns math - you start off by learning the rules (addition, subtraction, algebra, etc.) and work within those confines before you start to use these basic rules to push the boundaries and explore (proofs).
dx87 8y ago
Some behaviors did have to be encouraged, but from what I understand, the behaviors were encouraged through small indirect nudges instead of specifically adding code to the bots to tell them a certain behavior is good. For example, the devs said that the bots never tried to fight Roshan on their own because he does a lot of damage, has a lot of health, and the bots don't know what the reward for killing him is. To encourage the bots to try it, Roshan was given artificially low health so the bots would kill it to see what happens. Once the bots learned that there are rewards for killing him, the devs started to randomize his health between very high and very low values so the bots could get a feel for when they are strong enough to try and kill him during a normal match.
- glup 8y ago
  Very amusing that this is the same as the common practice in carnivorous mammals of maiming prey so that their offspring can practice hunting.
  - drusepth 8y ago
    It's also pretty similar to how I used to check if I were strong enough to kill Roshan (yet) when I first started playing Dota 2. :)
mywittyname 8y ago
> It's just telling that the bots have only learned one single strategy, don't even execute that very well, and still have to play with hero restrictions that rule out counters to that specific strategy of aggressive skirmishing and pushing. I suspect this has to with the fact that the bots were designed to be agents with no leadership. As you say, the bots performed very well in skirmishes, they lost because they didn't really know what to do beyond that. They might fare better if one of the bots could act as a team captain and provide high level instructions to the team, i.e., assign people to harvest, defend, or push. It seems like that's what the human teams have over the bots -- somebody telling the individual team members what to do.
- DuskStar 8y ago
  The bots might be running 5 instances of the software without communication, but they all have the same vision. It's probably better to think of it as one instance controlling 5 bots, since each instance can predict the others perfectly.
  - kenhwang 8y ago
    I think they should be training 5 different bots for the 1-5 positions because their priorities and roles are quite different. It seemed like all 5 bots had the same, "kill enemy, hit building" strategy, which is locally optimal, but globally harmful (sort of a tragedy of the commons scenario).
- Notorious_BLT 8y ago
  Generally, captains in Dota teams don't give specific instructions on what to do when not making specific plays. Carry players will default to returning to farming as efficiently as possible given the current state of the map. Support players will default to placing wards for map vision, stacking camps for the carry to farm, and searching for potential kills. But there definitely is a need for a shot-caller, and OpenAI has no idea how to play that role. OpenAI 5 had a number of glaring flaws in what it learned. It seemed to have never come to the conclusion that certain characters would have better outcomes if they focused on certain activities. Every character farmed, any character would grab the Aegis (reward item) after killing Roshan (usually this is reserved for a character likely to be in the middle of the fray, dealing a lot of damage). Another problem was that, until the last few weeks before its matches against pro players, they had allowed both teams to have 5 couriers. This allowed OpenAI 5 to keep up relentless pressure by constantly bringing themselves consumable health regeneration items. This was so unlike a regular game of Dota that the community (including their opponents) complained because normally the courier's time is a valuable resource, and the courier can normally be killed if it is used too close to enemies. With an endless supply of healing items with no risk involved, it didn't even resemble a game of Dota. They did away with this for TI, and it revealed a serious weakness in their relentless-aggression strategy. I get the impression most of OpenAI's games vs itself ended very quickly, with one team making a mistake during the relentless-aggression early game pushes. As a result, it seemed to have no idea what to do differently as the game went on.

tlb 8y ago

Looking "clowny" isn't something that RL agents learn to avoid. You see it even in Go agents. The MCTS algorithm looks at probability of winning, and if it's ~100% regardless of the next move, it'll often waste moves rather than trying to win as quickly as possible.

SOLAR_FIELDS 8y ago

My understanding is that there are behaviors that are coded into the bots at first to bootstrap their learning capabilities, which are then removed after a certain amount of learning has taken place. In another article that was posted here a few days ago one of the OpenAI team talks about removing the last of these "scripted" capabilities as part of the improvements they plan on making to the bots after this round of games against human players. I kind of liken it to a human who learns math - you start off by learning the rules (addition, subtraction, algebra, etc.) and work within those confines before you start to use these basic rules to push the boundaries and explore (proofs).

dx87 8y ago

Some behaviors did have to be encouraged, but from what I understand, the behaviors were encouraged through small indirect nudges instead of specifically adding code to the bots to tell them a certain behavior is good. For example, the devs said that the bots never tried to fight Roshan on their own because he does a lot of damage, has a lot of health, and the bots don't know what the reward for killing him is. To encourage the bots to try it, Roshan was given artificially low health so the bots would kill it to see what happens. Once the bots learned that there are rewards for killing him, the devs started to randomize his health between very high and very low values so the bots could get a feel for when they are strong enough to try and kill him during a normal match.

glup 8y ago
Very amusing that this is the same as the common practice in carnivorous mammals of maiming prey so that their offspring can practice hunting.
- drusepth 8y ago
  It's also pretty similar to how I used to check if I were strong enough to kill Roshan (yet) when I first started playing Dota 2. :)

mywittyname 8y ago

> It's just telling that the bots have only learned one single strategy, don't even execute that very well, and still have to play with hero restrictions that rule out counters to that specific strategy of aggressive skirmishing and pushing. I suspect this has to with the fact that the bots were designed to be agents with no leadership. As you say, the bots performed very well in skirmishes, they lost because they didn't really know what to do beyond that. They might fare better if one of the bots could act as a team captain and provide high level instructions to the team, i.e., assign people to harvest, defend, or push. It seems like that's what the human teams have over the bots -- somebody telling the individual team members what to do.

DuskStar 8y ago
The bots might be running 5 instances of the software without communication, but they all have the same vision. It's probably better to think of it as one instance controlling 5 bots, since each instance can predict the others perfectly.
- kenhwang 8y ago
  I think they should be training 5 different bots for the 1-5 positions because their priorities and roles are quite different. It seemed like all 5 bots had the same, "kill enemy, hit building" strategy, which is locally optimal, but globally harmful (sort of a tragedy of the commons scenario).
Notorious_BLT 8y ago
Generally, captains in Dota teams don't give specific instructions on what to do when not making specific plays. Carry players will default to returning to farming as efficiently as possible given the current state of the map. Support players will default to placing wards for map vision, stacking camps for the carry to farm, and searching for potential kills. But there definitely is a need for a shot-caller, and OpenAI has no idea how to play that role. OpenAI 5 had a number of glaring flaws in what it learned. It seemed to have never come to the conclusion that certain characters would have better outcomes if they focused on certain activities. Every character farmed, any character would grab the Aegis (reward item) after killing Roshan (usually this is reserved for a character likely to be in the middle of the fray, dealing a lot of damage). Another problem was that, until the last few weeks before its matches against pro players, they had allowed both teams to have 5 couriers. This allowed OpenAI 5 to keep up relentless pressure by constantly bringing themselves consumable health regeneration items. This was so unlike a regular game of Dota that the community (including their opponents) complained because normally the courier's time is a valuable resource, and the courier can normally be killed if it is used too close to enemies. With an endless supply of healing items with no risk involved, it didn't even resemble a game of Dota. They did away with this for TI, and it revealed a serious weakness in their relentless-aggression strategy. I get the impression most of OpenAI's games vs itself ended very quickly, with one team making a mistake during the relentless-aggression early game pushes. As a result, it seemed to have no idea what to do differently as the game went on.