6 ms·

The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games

67 points by jonbaer 5y ago

ddoran 5y ago
PPO = Proximal Policy Optimization [https://openai.com/blog/openai-baselines-ppo/ https://openai.com/blog/openai-baselines-ppo/]
- jdlyga 5y ago
  Thank you!
  - Robotbeat 5y ago
    Indeed. I looked for the definition in the whole webpage but couldn’t find it. Even Googling initially failed. https://arxiv.org/abs/1707.06347 https://arxiv.org/abs/1707.06347
    - throwaway81523 5y ago
      Yeah, did the same, then looked at the linked article. Its abstract: Proximal Policy Optimization (PPO) is a popular on-policy reinforcement learning algorithm but is significantly less utilized than off-policy learning algorithms in multi-agent settings. This is often due the belief that on-policy methods are significantly less sample efficient than their off-policy counterparts in multi-agent problems. In this work, we investigate Multi-Agent PPO (MAPPO), a variant of PPO which is specialized for multi-agent settings. Using a 1-GPU desktop, we show that MAPPO achieves surprisingly strong performance in three popular multi-agent testbeds: the particle-world environments, the Starcraft multi-agent challenge, and the Hanabi challenge, with minimal hyperparameter tuning and without any domain-specific algorithmic modifications or architectures. In the majority of environments, we find that compared to off-policy baselines, MAPPO achieves strong results while exhibiting comparable sample efficiency. Finally, through ablation studies, we present the implementation and algorithmic factors which are most influential to MAPPO's practical performance.
- cratermoon 5y ago
  https://jonathan-hui.medium.com/rl-proximal-policy-optimization-ppo-explained-77f014ec3f12 https://jonathan-hui.medium.com/rl-proximal-policy-optimizat...
- politician 5y ago
  The hero we need.
isaacimagine 5y ago
PPO is awesome, but so is GPT-style reward-trajectory prediction! http://arxiv.org/pdf/2106.01345v1 http://arxiv.org/pdf/2106.01345v1. As a RL hobbyist, I'd love to see some sort of hybrid approach. Thoughts?
bsder 5y ago
Are these things amazingly effective or are they simply demonstrating that Starcraft/DOTA aren't as difficult as we thought?
- seabird 5y ago
  Neither. Once knowledgeable people get a read on these type of things, they can usually handle it. The OpenAI Dota 2 "team" was open for the public to play -- it was certainly very good, but multiple teams beat it, sometimes even multiple times in a row. It was great at cheesy stuff like superhuman Force Staff plays that humans could never reliably pull off, but could be beat through macro pressure.
  - Buttons840 5y ago
    Facebook is hosting a NetHack AI competition I'm watching with interest. I want to see how AIs perform in a challenging environment that's pure decision making with no chance for superhuman performance sneaking in like we've seen with StarCraft and Dota.
  - sillysaurusx 5y ago
    I’m not sure this is true. Humans were unable to beat the bot when playing by the rules the bot trained on. The bot was only trained with a very specific hero pool, and special circumstances such as having a courier for each hero (highly unusual in competitive dota; basically unheard of) which it used to ferry health potions to itself constantly. The humans couldn't deal with this constant source of health regeneration. It was only after the hero pool was expanded to additional heroes that the humans were able to win. But this is like losing because 86 new heroes were just released that day; it would be unfair, to say the least, if you had to face heroes you knew nothing about, especially when your opponents were experts. OpenAI gave up working on it after they had proven that "if you spend $X amount of money, you can win at whatever rules you decide." Unfortunately the cost of $X is very high for this method, because it requires thousands (tens of thousands?) of simulations, each of which need to happen in real time on an actual dota client (i.e. it's running the game). The game rules also change continuously with new patch releases, so the training quickly becomes more or less obsolete, especially if the bot overfits on a particular ruleset. That doesn't change that the bot can overfit on a specific ruleset, which I think is a positive statement: it's quite remarkable, really, that it's possible at all.
tedunangst 5y ago
Research titled "surprising effectiveness" should quantify the surprise, not just the effect.
- carrolldunham 5y ago
  No, it should be retitled entirely because it's a terrible verbal-tic tier meme. Sick to death of every third post on hackernews being "The ___ ___ness of " especially "The un__ __ness of ". Language is so rich but we have this cesspit
  - gradschoolfail 5y ago
    I upvoted you, but the bottom of this cesspit unfortunately lie the outputs of some otherwise illuminated people like… ugh... feel free to find out for yourself if you have the stomach for such a thing..: https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences#Related_quotations https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...

67 points by jonbaer 5y ago

PPO = Proximal Policy Optimization [https://openai.com/blog/openai-baselines-ppo/ https://openai.com/blog/openai-baselines-ppo/]

jdlyga 5y ago
Thank you!
- Robotbeat 5y ago
  Indeed. I looked for the definition in the whole webpage but couldn’t find it. Even Googling initially failed. https://arxiv.org/abs/1707.06347 https://arxiv.org/abs/1707.06347
  - throwaway81523 5y ago
    Yeah, did the same, then looked at the linked article. Its abstract: Proximal Policy Optimization (PPO) is a popular on-policy reinforcement learning algorithm but is significantly less utilized than off-policy learning algorithms in multi-agent settings. This is often due the belief that on-policy methods are significantly less sample efficient than their off-policy counterparts in multi-agent problems. In this work, we investigate Multi-Agent PPO (MAPPO), a variant of PPO which is specialized for multi-agent settings. Using a 1-GPU desktop, we show that MAPPO achieves surprisingly strong performance in three popular multi-agent testbeds: the particle-world environments, the Starcraft multi-agent challenge, and the Hanabi challenge, with minimal hyperparameter tuning and without any domain-specific algorithmic modifications or architectures. In the majority of environments, we find that compared to off-policy baselines, MAPPO achieves strong results while exhibiting comparable sample efficiency. Finally, through ablation studies, we present the implementation and algorithmic factors which are most influential to MAPPO's practical performance.
cratermoon 5y ago
https://jonathan-hui.medium.com/rl-proximal-policy-optimization-ppo-explained-77f014ec3f12 https://jonathan-hui.medium.com/rl-proximal-policy-optimizat...
politician 5y ago
The hero we need.

isaacimagine 5y ago

PPO is awesome, but so is GPT-style reward-trajectory prediction! http://arxiv.org/pdf/2106.01345v1 http://arxiv.org/pdf/2106.01345v1. As a RL hobbyist, I'd love to see some sort of hybrid approach. Thoughts?

bsder 5y ago

Are these things amazingly effective or are they simply demonstrating that Starcraft/DOTA aren't as difficult as we thought?

seabird 5y ago
Neither. Once knowledgeable people get a read on these type of things, they can usually handle it. The OpenAI Dota 2 "team" was open for the public to play -- it was certainly very good, but multiple teams beat it, sometimes even multiple times in a row. It was great at cheesy stuff like superhuman Force Staff plays that humans could never reliably pull off, but could be beat through macro pressure.
- Buttons840 5y ago
  Facebook is hosting a NetHack AI competition I'm watching with interest. I want to see how AIs perform in a challenging environment that's pure decision making with no chance for superhuman performance sneaking in like we've seen with StarCraft and Dota.
- sillysaurusx 5y ago
  I’m not sure this is true. Humans were unable to beat the bot when playing by the rules the bot trained on. The bot was only trained with a very specific hero pool, and special circumstances such as having a courier for each hero (highly unusual in competitive dota; basically unheard of) which it used to ferry health potions to itself constantly. The humans couldn't deal with this constant source of health regeneration. It was only after the hero pool was expanded to additional heroes that the humans were able to win. But this is like losing because 86 new heroes were just released that day; it would be unfair, to say the least, if you had to face heroes you knew nothing about, especially when your opponents were experts. OpenAI gave up working on it after they had proven that "if you spend $X amount of money, you can win at whatever rules you decide." Unfortunately the cost of $X is very high for this method, because it requires thousands (tens of thousands?) of simulations, each of which need to happen in real time on an actual dota client (i.e. it's running the game). The game rules also change continuously with new patch releases, so the training quickly becomes more or less obsolete, especially if the bot overfits on a particular ruleset. That doesn't change that the bot can overfit on a specific ruleset, which I think is a positive statement: it's quite remarkable, really, that it's possible at all.

tedunangst 5y ago

Research titled "surprising effectiveness" should quantify the surprise, not just the effect.

carrolldunham 5y ago
No, it should be retitled entirely because it's a terrible verbal-tic tier meme. Sick to death of every third post on hackernews being "The ___ ___ness of " especially "The un__ __ness of ". Language is so rich but we have this cesspit
- gradschoolfail 5y ago
  I upvoted you, but the bottom of this cesspit unfortunately lie the outputs of some otherwise illuminated people like… ugh... feel free to find out for yourself if you have the stomach for such a thing..: https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences#Related_quotations https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...