6 ms·

In the last game of the series the bots were forced to play an intentionally terrible hero lineup. The humans finally got to win a game but more interesting for

by ufo 8y ago

In the last game of the series the bots were forced to play an intentionally terrible hero lineup. The humans finally got to win a game but more interesting for me was that the times when the AI did seemingly crazy things were much more common. I wonder if this is an artifact of the training methodology: maybe if your team is very weak then your choices are also weaker, and reinforcement learning doesn't work as well?

tux3 8y ago
It reminds me of the Go AIs going on tilt when they're far behind. When the win percentage for Go AIs gets to around 5%, every action it can take results in a losing game so it can't make the difference between normal play and super strange moves anymore. When every choice is really bad, humans tend to still go with their normal strategy and wait for their chance to turn things around, but bots assume the opponent is playing perfectly, so they act like their winrate is going to stay near zero no matter what they do.
- deleted 8y ago
  [deleted]
- nopinsight 8y ago
  If this is true (which it might be, I haven’t studied these systems in detail), then an obvious fix is to get the AI to randomly train with ‘weaker’ versions of itself (perhaps while the strong instance is handicapped) in addition to the latest generation. Several levels of weak opponents should be used, with varying probabilities, to tune the AI’s robustness against real-world, imperfect competitors.
dooglius 8y ago
That's probably part of it, but another explanation would be that the AI wouldn't choose such a lineup on its own, and so it doesn't train on the relevant gameplay.
hackandtrip 8y ago
I'd say that those were crazy things, but not too much. AI understood that teamfight were impossibile to win, and so the really weird choices (sven dying for the tower lol) . I'd say that matches with such lineup are not too much and so they didn't really know, but they did an awesome job and kept things balanced for so long
- Rychard 8y ago
  > really weird choices (sven dying for the tower lol) Taking a tier 2 tower nets everyone on the team 120 gold (a further 150+ gold goes to the hero who gets the last hit), and losing Sven probably gave the opposing team less than was gained. Perhaps the AI simply placed more value on increasing the total net worth of the team than it valued saving the life of one of its core heroes. Additionally, there was no guarantee that he would have been able to escape, as Sven was deep on the enemy's side of the map, and there's a very real possibility that he could have been ganked from someone in the jungle had he attempted to retreat.
  - drusepth 8y ago
    Dota player here: other metrics that could have been involved in the trading decision are: - Potential chance the enemy team would deny the tower before another friendly hero could take it (netting Sven's team 0 gold for the time spent whacking away at it) - Map vision (removing a T2 often cuts a significant section of map awareness away, since the tower is no longer providing vision or protection) - XP gains (Sven won't gain any XP while dead, nor from killing the tower) - Creep equilibrium (this is less important, or at least thought about less often, later on in the game and past T1 towers, but might've been a factor in drawing the creep clash point to a particular location) - Dictating team net worth averages (to some extent, if they predicted a loss in opponents forcing a teamfight or predicted a likely pickoff, gold lost could be minimized now by taking a death early, lowering the average net worth on the team). Obviously, there are others and these can also be mixed and matched in various ways (e.g. cutting off map vision so they can more safely farm additional jungle creeps). Not saying any of these aspects _were_ a part of the decision to trade Sven for a tower, but.. just wanted to include a few more subsurface aspects that _could_ be used in such a decision.
make3 8y ago
maybe the team never picked lineup it knew were bad, and as such never trained with them
zaarn 8y ago
I think this might be a result of "the only winning move is not to play", so to speak. If the game is, in the mind of the AI agents, unwinnable, not playing is not an option, therefore it begins to pick random actions instead. I'm not sure if the AI can surrender (I only managed to watch the first two games as it was rather late at night) but it might be a path to explore; having the AI give up if the game cannot be won anymore.
- wwhitlow 8y ago
  At what percentage would you allow an AI to consider a game unwinnable? While an AI that behaves erratically when the odds are low might be worth allowing it to be considered forfeit worthy, but the thing about humans is we make mistakes. Therefore an ideal AI that can continue to execute reasonable moves should have a lower percentage threshold where it decides to forfeit. See this match[0] for an example of a spectacular comeback that I feel an AI might have considered forfeit worthy if not well defined. [0]: https://youtu.be/LwSQv_sNZBI https://youtu.be/LwSQv_sNZBI
  - ionforce 8y ago
    Maybe this is a limitation of self-play. If the opponent an AI faces during training is always optimal, then there's no surface area of mistakes. The losing AI, in its model/mind, knows that the game is over after a specific threshold. So it hasn't learned how to optimize for capitalizing on mistakes. I wonder if this situation can be fixed by adding more randomness. For example, force AI'1 to be in a losing position to AI'2, but then suddenly switch the power level of AI'2 to be much weaker (where mistakes happen) so that AI'1 learns how to fight its way out of tough situations.
    - ufo 8y ago
      One of the most interesting takeaways from the post game interview for me was that the AI can be very stupid if you just blindly throw it in a self-play setting but with clever use of randomization (modifying power levels) and action restrictions (for example, only allowing the agent to spend an anti-invis item when a nearby enemy goes out of sight) it is possible to provide better learning opportunities for the AI.

by ufo 8y ago

tux3 8y ago

It reminds me of the Go AIs going on tilt when they're far behind. When the win percentage for Go AIs gets to around 5%, every action it can take results in a losing game so it can't make the difference between normal play and super strange moves anymore. When every choice is really bad, humans tend to still go with their normal strategy and wait for their chance to turn things around, but bots assume the opponent is playing perfectly, so they act like their winrate is going to stay near zero no matter what they do.

deleted 8y ago
[deleted]
nopinsight 8y ago
If this is true (which it might be, I haven’t studied these systems in detail), then an obvious fix is to get the AI to randomly train with ‘weaker’ versions of itself (perhaps while the strong instance is handicapped) in addition to the latest generation. Several levels of weak opponents should be used, with varying probabilities, to tune the AI’s robustness against real-world, imperfect competitors.

dooglius 8y ago

That's probably part of it, but another explanation would be that the AI wouldn't choose such a lineup on its own, and so it doesn't train on the relevant gameplay.

hackandtrip 8y ago

I'd say that those were crazy things, but not too much. AI understood that teamfight were impossibile to win, and so the really weird choices (sven dying for the tower lol) . I'd say that matches with such lineup are not too much and so they didn't really know, but they did an awesome job and kept things balanced for so long

Rychard 8y ago
> really weird choices (sven dying for the tower lol) Taking a tier 2 tower nets everyone on the team 120 gold (a further 150+ gold goes to the hero who gets the last hit), and losing Sven probably gave the opposing team less than was gained. Perhaps the AI simply placed more value on increasing the total net worth of the team than it valued saving the life of one of its core heroes. Additionally, there was no guarantee that he would have been able to escape, as Sven was deep on the enemy's side of the map, and there's a very real possibility that he could have been ganked from someone in the jungle had he attempted to retreat.
- drusepth 8y ago
  Dota player here: other metrics that could have been involved in the trading decision are: - Potential chance the enemy team would deny the tower before another friendly hero could take it (netting Sven's team 0 gold for the time spent whacking away at it) - Map vision (removing a T2 often cuts a significant section of map awareness away, since the tower is no longer providing vision or protection) - XP gains (Sven won't gain any XP while dead, nor from killing the tower) - Creep equilibrium (this is less important, or at least thought about less often, later on in the game and past T1 towers, but might've been a factor in drawing the creep clash point to a particular location) - Dictating team net worth averages (to some extent, if they predicted a loss in opponents forcing a teamfight or predicted a likely pickoff, gold lost could be minimized now by taking a death early, lowering the average net worth on the team). Obviously, there are others and these can also be mixed and matched in various ways (e.g. cutting off map vision so they can more safely farm additional jungle creeps). Not saying any of these aspects _were_ a part of the decision to trade Sven for a tower, but.. just wanted to include a few more subsurface aspects that _could_ be used in such a decision.

make3 8y ago

maybe the team never picked lineup it knew were bad, and as such never trained with them

zaarn 8y ago

I think this might be a result of "the only winning move is not to play", so to speak. If the game is, in the mind of the AI agents, unwinnable, not playing is not an option, therefore it begins to pick random actions instead. I'm not sure if the AI can surrender (I only managed to watch the first two games as it was rather late at night) but it might be a path to explore; having the AI give up if the game cannot be won anymore.

wwhitlow 8y ago
At what percentage would you allow an AI to consider a game unwinnable? While an AI that behaves erratically when the odds are low might be worth allowing it to be considered forfeit worthy, but the thing about humans is we make mistakes. Therefore an ideal AI that can continue to execute reasonable moves should have a lower percentage threshold where it decides to forfeit. See this match[0] for an example of a spectacular comeback that I feel an AI might have considered forfeit worthy if not well defined. [0]: https://youtu.be/LwSQv_sNZBI https://youtu.be/LwSQv_sNZBI
- ionforce 8y ago
  Maybe this is a limitation of self-play. If the opponent an AI faces during training is always optimal, then there's no surface area of mistakes. The losing AI, in its model/mind, knows that the game is over after a specific threshold. So it hasn't learned how to optimize for capitalizing on mistakes. I wonder if this situation can be fixed by adding more randomness. For example, force AI'1 to be in a losing position to AI'2, but then suddenly switch the power level of AI'2 to be much weaker (where mistakes happen) so that AI'1 learns how to fight its way out of tough situations.
  - ufo 8y ago
    One of the most interesting takeaways from the post game interview for me was that the AI can be very stupid if you just blindly throw it in a self-play setting but with clever use of randomization (modifying power levels) and action restrictions (for example, only allowing the agent to spend an anti-invis item when a nearby enemy goes out of sight) it is possible to provide better learning opportunities for the AI.