5 ms·

For context: OpenAI is playing against a professional team this time, unlike the event a few weeks ago where the humans were skilled but not active pros or used

For context: OpenAI is playing against a professional team this time, unlike the event a few weeks ago where the humans were skilled but not active pros or used to playing together. This team isn't the strongest (they placed 17-18th during this TI) but OpenAI will play again tomorrow and the day after, presumably against progressively better teams.

roenxi 8y ago
> his team isn't the strongest (they placed 17-18th during this TI) Oh, is that all :P. Assuming a 2x buffer for players who are good but not competitive, there are maybe 200 humans in the world who are better at DotA. If mastery is 1 in 10,000, these people are masters.
- Q6T46nT668w6i3m 8y ago
  Many believe that it’s an exponential curve so there’s a substantial difference between the first and twentieth place teams.
  - whymauri 8y ago
    Looking historically at team ELO and how squads have gone on 27-series win streaks, this is probably right. Though I think the gap between "low-tier" TI teams and TI winners is getting much tighter. Group B at TI8 was so close the difference between upper-bracket and lower-bracket was 1 game for most of the teams.
    - noxvilleza 8y ago
      *Elo, not ELO :)
      - whymauri 8y ago
        lol, good catch. Gotta give Arpad his credit :)
  - PeCaN 8y ago
    Even if this is true, pain is one of the unpredictable teams that can take a game off anyone. They went 3-1 against Liquid at the Birmingham Major. Actually, "weaker" (top 20) teams regularly do take games off better teams so I don't think there's much evidence for an exponential skill increase between teams.
- nindalf 8y ago
  No, there are way more than 200 players better than Pain, all due respect to them. That’s clear by looking at leaderboards. But team dynamics and synergy in this game matter so much that they are definitely in the top 20 of teams regardless of their individual skill. Seeing OpenAI do so well against this team impressed the hell out of me.
  - Strom 8y ago
    A really big factor making up your ranking on the leaderboard is your ability to play in a chaotic game where your team and yourself aren't on the same page. Just taking the top players from the leaderboard (who aren't already on a top team) and putting them together makes up a team that is good enough to win against other similar random teams (hence their high rank) but can't win against actual teams, which is evidenced all the time in the open qualifers where actual teams that consist of lower ranked players beat them. TI open qualifer / regional qualifer / actual event placements are way better indicators on how good a team is than pub leaderboards.
  - throw_away_777 8y ago
    The way TI is structured they invite one team from South America. So Pain is likely not in top 20, but still a very good team.
    - nindalf 8y ago
      I know how TI is structured, but I’m confident they’re top 20 anyway. Which team not at TI is better, do you think? Complexity? Navi? I don’t think so.
sambroner 8y ago
Seems like playing one of the best 200 players in the world is a reasonable proxy for a top tier player. I'm unfamiliar with DOTA rules, but does anyone know if there are any limitations on the openAI team? e.g. Things like keystrokes per minute, scroll speed, etc
- jsheard 8y ago
  During the last OpenAI showmatch the bots had an artificial 200ms reaction time imposed on them, I assume it's the same here but I'm not sure.
  - sambroner 8y ago
    Thanks.
  - larkeith 8y ago
    I don't believe this is the case, as OpenAI was able to hex (an immediate disable) Earthshaker before he got off Echoslam (a spell with no cast delay except the time to click the key).
    - def_true_false 8y ago
      The ES player didn't queue his abilities so there was delay between blinking and casting echo. Someone counted the frames and it was well over the 200ms minimum.
      - el_devo 8y ago
        Here is a post from the Dota 2 subreddit discussing the timing with proof that OpenAI's reaction was over the 200ms minimum: https://www.reddit.com/r/DotA2/comments/94vdpm/openai_hex_was_within_the_200ms_response_time/ https://www.reddit.com/r/DotA2/comments/94vdpm/openai_hex_wa...
        Q6T46nT668w6i3m 8y ago
nopinsight 8y ago
In general, the differences among a top echelon of human expertise are vastly smaller than those between humans and other kinds of agents, animals or AIs. We all share similar cognitive architecture for handling complex domains after all. Once OpenAI Five defeat 3-4 pro teams convincingly, it could be presumed from the observation above that they will defeat any other top teams within a year at most, if sufficient resources are used for improving the AI. An exception is when a fundamental hidden weakness is found in the AI. Another confounding factor is major rule changes where the game essentially becomes a different one.
- chibg10 8y ago
  > Once OpenAI Five defeat 3-4 pro teams convincingly, it could be presumed from the observation above that they will defeat any other top teams within a year at most, if sufficient resources are used for improving the AI. An exception is when a fundamental hidden weakness is found in the AI. Not sure how you intended this, but the hero pool constraint is actually an enormous advantage for the AI. The current hero pool isn't, by any means, random. I don't know OpenAI selected it, but it's highly conducive to OpenAI Five's strengths--deathballing, mechanics, teamfight coordination, etc. It lacks essentially all of the heroes that humans would normally pick to counter that strategy. Playing on an unrestricted hero set would test OpenAI Five in entirely different ways than playing on the current hero pool would. I would not be comfortable betting on your statement at all.
  - noxvilleza 8y ago
    > The current hero pool isn't, by any means, random. Some of the restrictions they've discussed - like initially they couldn't handle with illusions, summons or invisibility at all. They obviously dropped the invis restriction. My feeling is that the AI just learned that of the given pool, 'deathball' was the best (and specific heroes like Gyrocopter/Sniper were the most effective carries).
  - ajuc 8y ago
    > I don't know OpenAI selected it, but it's highly conducive to OpenAI Five's strengths--deathballing, mechanics, teamfight coordination, etc. It lacks essentially all of the heroes that humans would normally pick to counter that strategy. Isn't it the other way around - openai adjusted to the hero pool to come up with its' strategy, because it was trained on this hero pool. If hero pool had only late-game melee carries - I assume openai would come up with a strategy that works with that.

roenxi 8y ago

> his team isn't the strongest (they placed 17-18th during this TI) Oh, is that all :P. Assuming a 2x buffer for players who are good but not competitive, there are maybe 200 humans in the world who are better at DotA. If mastery is 1 in 10,000, these people are masters.

Q6T46nT668w6i3m 8y ago
Many believe that it’s an exponential curve so there’s a substantial difference between the first and twentieth place teams.
- whymauri 8y ago
  Looking historically at team ELO and how squads have gone on 27-series win streaks, this is probably right. Though I think the gap between "low-tier" TI teams and TI winners is getting much tighter. Group B at TI8 was so close the difference between upper-bracket and lower-bracket was 1 game for most of the teams.
  - noxvilleza 8y ago
    *Elo, not ELO :)
    - whymauri 8y ago
      lol, good catch. Gotta give Arpad his credit :)
- PeCaN 8y ago
  Even if this is true, pain is one of the unpredictable teams that can take a game off anyone. They went 3-1 against Liquid at the Birmingham Major. Actually, "weaker" (top 20) teams regularly do take games off better teams so I don't think there's much evidence for an exponential skill increase between teams.
nindalf 8y ago
No, there are way more than 200 players better than Pain, all due respect to them. That’s clear by looking at leaderboards. But team dynamics and synergy in this game matter so much that they are definitely in the top 20 of teams regardless of their individual skill. Seeing OpenAI do so well against this team impressed the hell out of me.
- Strom 8y ago
  A really big factor making up your ranking on the leaderboard is your ability to play in a chaotic game where your team and yourself aren't on the same page. Just taking the top players from the leaderboard (who aren't already on a top team) and putting them together makes up a team that is good enough to win against other similar random teams (hence their high rank) but can't win against actual teams, which is evidenced all the time in the open qualifers where actual teams that consist of lower ranked players beat them. TI open qualifer / regional qualifer / actual event placements are way better indicators on how good a team is than pub leaderboards.
- throw_away_777 8y ago
  The way TI is structured they invite one team from South America. So Pain is likely not in top 20, but still a very good team.
  - nindalf 8y ago
    I know how TI is structured, but I’m confident they’re top 20 anyway. Which team not at TI is better, do you think? Complexity? Navi? I don’t think so.

sambroner 8y ago

Seems like playing one of the best 200 players in the world is a reasonable proxy for a top tier player. I'm unfamiliar with DOTA rules, but does anyone know if there are any limitations on the openAI team? e.g. Things like keystrokes per minute, scroll speed, etc

jsheard 8y ago
During the last OpenAI showmatch the bots had an artificial 200ms reaction time imposed on them, I assume it's the same here but I'm not sure.
- sambroner 8y ago
  Thanks.
- larkeith 8y ago
  I don't believe this is the case, as OpenAI was able to hex (an immediate disable) Earthshaker before he got off Echoslam (a spell with no cast delay except the time to click the key).
  - def_true_false 8y ago
    The ES player didn't queue his abilities so there was delay between blinking and casting echo. Someone counted the frames and it was well over the 200ms minimum.
    - el_devo 8y ago
      Here is a post from the Dota 2 subreddit discussing the timing with proof that OpenAI's reaction was over the 200ms minimum: https://www.reddit.com/r/DotA2/comments/94vdpm/openai_hex_was_within_the_200ms_response_time/ https://www.reddit.com/r/DotA2/comments/94vdpm/openai_hex_wa...
      - Q6T46nT668w6i3m 8y ago

nopinsight 8y ago

In general, the differences among a top echelon of human expertise are vastly smaller than those between humans and other kinds of agents, animals or AIs. We all share similar cognitive architecture for handling complex domains after all. Once OpenAI Five defeat 3-4 pro teams convincingly, it could be presumed from the observation above that they will defeat any other top teams within a year at most, if sufficient resources are used for improving the AI. An exception is when a fundamental hidden weakness is found in the AI. Another confounding factor is major rule changes where the game essentially becomes a different one.

chibg10 8y ago
> Once OpenAI Five defeat 3-4 pro teams convincingly, it could be presumed from the observation above that they will defeat any other top teams within a year at most, if sufficient resources are used for improving the AI. An exception is when a fundamental hidden weakness is found in the AI. Not sure how you intended this, but the hero pool constraint is actually an enormous advantage for the AI. The current hero pool isn't, by any means, random. I don't know OpenAI selected it, but it's highly conducive to OpenAI Five's strengths--deathballing, mechanics, teamfight coordination, etc. It lacks essentially all of the heroes that humans would normally pick to counter that strategy. Playing on an unrestricted hero set would test OpenAI Five in entirely different ways than playing on the current hero pool would. I would not be comfortable betting on your statement at all.
- noxvilleza 8y ago
  > The current hero pool isn't, by any means, random. Some of the restrictions they've discussed - like initially they couldn't handle with illusions, summons or invisibility at all. They obviously dropped the invis restriction. My feeling is that the AI just learned that of the given pool, 'deathball' was the best (and specific heroes like Gyrocopter/Sniper were the most effective carries).
- ajuc 8y ago
  > I don't know OpenAI selected it, but it's highly conducive to OpenAI Five's strengths--deathballing, mechanics, teamfight coordination, etc. It lacks essentially all of the heroes that humans would normally pick to counter that strategy. Isn't it the other way around - openai adjusted to the hero pool to come up with its' strategy, because it was trained on this hero pool. If hero pool had only late-game melee carries - I assume openai would come up with a strategy that works with that.