12 ms·

OpenAI at the Dota 2 World Championships

510 points by frisco 9y ago

dvt 9y ago
Saw this live and as someone that's played Dota for many years, it's not as impressive as people make it out to be. A Shadowfiend 1v1 is primarily a test of mechanical skill and not judgment. A similar comparison is if a Counter-Strike or Quake bot would instantly snap to your head getting a headshot. Like, yeah, cool I guess, and it would beat 100% of human players 100% of the time, but it's not even remotely as impressive as Chess or Go. The razes in particular are "skill shots" that are sometimes hard for humans to estimate whereas for a robot it's just simple math. I will say that the one impressive aspect of the bot was raze faking, but I think that it was one of those cases where it was "coached" (same as the creep blocking).
- jlebar 9y ago
  > but I think that it was one of those cases where it was "coached" (same as the creep blocking). What makes you think that? In the interview, they explicitly disclaim this.
  - dvt 9y ago
    In the interview, they literally said they coached it to have certain behaviors.
    - cjbprime 9y ago
      I didn't get that from the interview. It implied the bot learned the game itself from a blank slate.
    - keerthiko 9y ago
      They said "yes, it does that"/"it's been known to do that", not "it was coached to do that". Just as a "We've seen it do that before."
      - dvt 9y ago
        Yeah, just re-watched the clip, I think you're right. My mistake (the word "coached" confused me).
      - nnnnnande 9y ago
        4:33 into the video: "[...] and coached it on what we thought was good or bad".
  - nnnnnande 9y ago
    No, actually they explicitly state that they also coached it on what they though was good or bad (4:33 into the video). Having said that, I guess we still don't know which game mechanics were coached.
- keerthiko 9y ago
  It pulled some interesting tactical moves that I was keeping an eye out for. - In game 1, when the AI triggers his flask of health regen (which gets dispelled if you take hero damage) it zones Dendi out knowing he's going to come in for an attempt to dispel his regen. It does this with lots of non-committal, pre-emptive shadowrazes. It doesn't do this any other time. It's aggressively predicting what Dendi may favor based on its own cirucmstances. - In game 2, Dendi said he was going to try letting one creep ahead and see if it would give him a lane advantage. [very technical DOTA mechanics] What this does is if the opponent creep blocks better than you, their creep wave strikes your overextending creep without taking any damage, under their tower, thus significantly weakening your creep wave, making the tug of war swing very heavily in your favor. You counter this by also letting one of your creeps out ahead as soon as you realize it. The AI that was perfectly blocking all the way till the creeps met up in G1, as soon as it saw the leading creep let one of its creeps forward. A perfect response that is highly unlikely to have been coached but determined as the best choice to, what I'm guessing is to return the game to a familiar state. - The bot cornered Dendi between some trees and his tower, and then with 0 fakeouts/hesitation shadowrazed him. It was not just a matter of knowing the target was in range, but also accounting for the probability that he could dodge it. Not 100% certain about the mechanics here, but I believe: Shadowraze has a ~.2s cast animation. It can be canceled during the first .1s (fake out), but after that it is committed to, you will complete the animation, spend the mana, and cast the spell. If the cast crosses this window and the opponent is capable of moving the short distance needed to dodge in this time, it is theoretically not something the AI can guarantee on pure mechanical skill. So cornering is an approach advanced players use to reduce the probability and room for dodging a skillshot, and I found it taking instant advantage of that opportunity very impressive. Besides the obvious mechanical precision of a bot once it's made a decision, the other inhuman factor here is what Dendi mentions, that it has zero hesitation to take advantage of even the smallest opening it has available. A pro human who is even instantly aware of an opportunity, no matter how theoretically certain they are that they should take it, hesitates at least a little bit to think "is there a trap? Am I not considering something?" Hesitation is an incredibly human trait.
  - Retric 9y ago
    Bots hesitate via thinking time. You see this in chess matches most obviously when a high level bot spends more time on some moves because it realizes the situation is unusually important or tricky.
- pinouchon 9y ago
  In this video listing "learned bot behaviors", they show creep blocking as one such behavior: https://youtu.be/wpa5wyutpGc?t=25s https://youtu.be/wpa5wyutpGc?t=25s. So it looks like the bot figured it on its own. (blog post: https://blog.openai.com/dota-2 https://blog.openai.com/dota-2)
brandonhsiao 9y ago
Is it taking in raw pixels or reading from game memory? How did they gather data to train for this? (Did Valve give them an API?)
- nacs 9y ago
  I'm sure they got some kind of official API since this seems to be a partnership between openAI and Valve. Blizzard's Starcraft did the same thing for their recently announced AI project where Starcraft 2 got an API for the AI developers to work with: https://techcrunch.com/2017/08/09/blizzard-and-deepmind-turn-starcraft-ii-into-an-ai-research-lab/ https://techcrunch.com/2017/08/09/blizzard-and-deepmind-turn...
- mIREdeRMedeFLaO 9y ago
  https://developer.valvesoftware.com/wiki/Dota_Bot_Scripting https://developer.valvesoftware.com/wiki/Dota_Bot_Scripting > Bot scripting in Dota is done via lua scripting. This is done at the server level, so there's no need to do things like examine screen pixels or simulate mouse clicks; instead scripts can query the game state and issue orders directly to units. Scripts have full access to all the entity locations, cooldowns, mana values, etc that a player on that team would expect to. The API is restricted such that scripts can't cheat -- units in FoW can't be queried, commands can't be issued to units the script doesn't control, etc.
  - hhmc 9y ago
    I wouldn't be surprised if they had access to a less restricted AI. Until we see more concrete information I remain sceptical that they are only using the official API.
    - mAritz 9y ago
      Why? Everything it needed for that 1v1 was available. There was nothing in the FoW that needed to be known, it can keep timers about enemy cooldowns, gold and xp (approximations at least since it's somewhat random).
nacs 9y ago
Looks like its also being hosted on Twitch.tv in addition to Youtube: https://www.twitch.tv/dota2ti https://www.twitch.tv/dota2ti
- justicezyx 9y ago
  Twitch is the official streaming partner for 7 years.
justicezyx 9y ago
When will the match happen or did it already happen? I did not find any vods.
- justicezyx 9y ago
  OK it's happening right now, I saw machine the host's announcement
  - BLanen 9y ago
    Weird website... No indication that a match is happening/happened/ or going to happen or with times or not. I guess I missed it and the stream link is now just a general link to the TI stream, which confused me for a bit.
    - justicezyx 9y ago
      They probably dont expect the dota tournament to have such a large influence scope...
thatsadude 9y ago
I love both Dota and ML, this is awesome. I would love to know whether they release the source code for us to play around.
- popcorncolonel 9y ago
  It is OpenAI, after all.
deleted 9y ago
[deleted]
sondr3 9y ago
Wow, that was not even close either. And it only takes it two weeks to reach that level of play? OpenAI should create a team of five (as they said they would) and let it play in the qualifiers next year.
- hmate9 9y ago
  5v5 is much more complex than 1v1 gameplay, it's not as simple as taking the current AI and making 5 of it.
  - criloz2 9y ago
    also, it only will be fair if the bots are completely independents instead of a unique bot controlling 5 heroes
  - sondr3 9y ago
    I'm not arguing that, just that it would be cool. The creators mentioned on stage that next year it would be a team of five bots. They also said it took the bot two weeks to reach the level it was at right now, so I'm curious to see how/if it'll evolve in the future.
  - ronald_raygun 9y ago
    I don't want to be "that guy", but I used to play a lot of LOL, and I feel like 5v5 is in a different ballpark in terms of difficulty than a 1v1. Even champion select is strategically intensive. TSM (one of the best US teams), lost to an amateur team because they got out done in team selection. https://www.youtube.com/watch?v=g7l30-4A6WQ https://www.youtube.com/watch?v=g7l30-4A6WQ
    - hmate9 9y ago
      It is extremely difficult. 1v1 is like beating humans at chess. 5v5 is like beating humans at Go. That's the kind of jump it is.
      - unrealhoang 9y ago
eduren 9y ago
This is really cool outreach on OpenAI's part. So many young people watch The International and I'm guessing that at least a few of them are more interested in CS and STEM from seeing this.
savethefuture 9y ago
Well the presentation was a little meh but very impressive tech they built, excited to see the 5v5.
taion 9y ago
Is the "two weeks" of time in the Dota environment, or two weeks of training time running in parallel as with A3C or something?
- hhmc 9y ago
  From what the devs indicated on stream it sounded like processing hours (with a ratio of around 300 in-game hours to 1 processing hour).
  - nopinsight 9y ago
    Interesting. That means it accumulates the equivalent of 24 * 14 * 300 = 100,800 hours of experience. That's about double the amount of practice one gets for playing 10 hours a day for 14 years.
cissou 9y ago
It looks like it's over…? Someone has a replay of the relevant part(s)?
- cissou 9y ago
  https://www.youtube.com/watch?v=ac1getNs2P8 https://www.youtube.com/watch?v=ac1getNs2P8
Analemma_ 9y ago
Not that this isn’t very cool, but 1v1 Dota isn’t anything like the full game, it’s mostly a competition of who has better micro. If it can beat a team of pros at 5v5– which is where the imperfect information, short-vs-long term strategy and inter-agent communication challenges come into play— then I’ll be impressed.
- popinman322 9y ago
  Honestly, you don't even have to make all the AI heroes separate. We automatically assume that each would be controlled by a separate virtual player, but one virtual player could very well control all controllable units on a team. That'd be interesting to watch.
justicezyx 9y ago
To be clear 1v1 mid lane is harder than 1 unit v 1 unit in sc. But it is at least 3 levels behind 5v5 human play. So first it's laining, which is showing in the game. Then is the ganking basically 2-3 player working together. Above that that's 5 men team fight. Then you have the strategy planning in the band pick phase, and in game movement coordination. There are other things like item choices, in game communication etc. I guess bots never tilt... This 1v1 match is definitely a proof of the strength and maturity of modern AI. It will be extremely interesting to see if they can train a bot to achieve the above intelligence. If so, I guess the game is pretty much losing a lot of its appeal.
- hellbanner 9y ago
  How does having a bot succeed cause appeal to be lost? First you're suggesting that the strategy will be "found" and never beaten (no room for improvement, not even in hero selection). Secondly.. Chess & Go have been around for thousands of years and they still are played en masse daily.
  - justicezyx 9y ago
    > How does having a bot succeed cause appeal to be lost? You might think not beating bots is OK. Not for me and many people have been involved in competitive scene. The moment machine beats human, a large part of the competition is gone. The essence is that you want to be the best. But if all you can do is to beat some other inferior opponents, what's the point? > Secondly.. Chess & Go have been around for thousands of years and they still are played en masse daily. This is irrelevant.
    - tyoverby 9y ago
      Human Chess players lost against Deep Blue in 1997; people still play Chess. In fact, they've learned a lot of strategies from the strategically superior AI.
    - taneq 9y ago
      We've had forklifts for decades and plenty of people still enter power lifting competitions.
bhntr3 9y ago
This was fun. It's our intern's last day on the ML infra team here. And he happens to be a competitive collegiate DOTA player. We were all crowded around watching the screen, shouting. Couldn't have planned a better send off. Nice job, OpenAI!
- popcorncolonel 9y ago
  As a ML researcher and an avid dota fan, I'm jealous! And it must have been great with Dendi the legend there too.
xfer 9y ago
Very impressive, even if there are some limitations. I look forward to more progress for a team of bots.
demonshalo 9y ago
MEH. I am not impressed. Restricting the number of parameters and variables in order to produce a bot that can do 1 sub-set of tasks really well is nothing special imo. Complexity and simulations of what you have not yet encountered is something humans can do with ease. This is not something a bot can do in a complex environment like DOTA. I'll be the first to admit that I was wrong and that AI is truly a thing once I see a bot like this one beat a pro-team in a 5v5. Until then, meh...
- criloz2 9y ago
  is an improvement against the other bots, it can be really good for training players, I would love to test it my self, but yeah dota is biggest that this, not only 5 v5, also 100+ heroes, and professional player can play a variety of heroes on mid with different matchups and its nuisances, the bot still need it a lot to even dominate 1v1.
kmnc 9y ago
How do things like reaction time, and actions per second work with something like this? Is it just an assumed advantage the ai gets, or does it simulate the limitations of a human? If it doesn't, how big of an advantage is it in an ai versus human competition?
- mAritz 9y ago
  That was pretty much the entire reason it won. It did some standard high-level 1v1 laning techniques, but if human "conditions"[1] were implemented it would look a lot different. Just the blocking of the lane creeps at the start is already super-human and gives the bot a huge advantage. [1] like lower actions per minute, latency, occasionally missclicking and not being 100% certain about distances
personjerry 9y ago
It was streaming and it's over, so here's the VOD: https://www.youtube.com/watch?v=ac1getNs2P8 https://www.youtube.com/watch?v=ac1getNs2P8
joefkelley 9y ago
This will be HUGE for competitive Dota, even if they never take it further. In much the same ways top chess players have learned from engines, I have to think top Dota players will practice against and study the hell out of this bot if OpenAI makes it available. Not everything is replicable by humans... for instance I noticed it constantly animation-canceling razes and only finishing it if it was going to hit; a human will definitely mess this up. But other things can definitely be used. It was positioning somewhat strangely, for example.
kibwen 9y ago
In the interest of a "fairer" comparison, I wonder how much of a difference it would make to force the AI to simulate mouse/keyboard input and interpret the raw screen buffer output, rather than using direct APIs into the game's guts, to more faithfully emulate its human opponent. I'm guessing the peripheral inputs wouldn't be much of a hurdle, but the image processing step could be very interesting.
- Zyst 9y ago
  Raw screen is not as big of a deal as it would seem, in DotA and Starcraft you can glimpse most of the information you can see on a screen through the minimap.
  - danielmorozoff 9y ago
    It would seem to me that this is an oversimplification. Doing screen based video processing is not simple and is not exact especially where camera movements are independent of game dynamics as well as the difficulties inherent in doing fast frame classification- there has been some good work in this direction recently but far from perfect(yolo, ssd). The route deep mind took in partnering with the game manufacturers to gain inputs directly while receiving global information about game state seems the simplest for their goal of training rl agents. We considered doing a project around this but figured it was a massive iceberg.
    - Zyst 9y ago
      Oh I misread the "Direct access to game API section", I thought they meant making it so that the machine can only see "what's on their screen", so if someone were to cast a long range skill for instance they wouldn't get that information. Which is a distinct advantage for machines. That is indeed, by no means trivial. I do think it's pointless though, that's just another exercise for the sake of making it another exercise.
- 147 9y ago
  How do you know it's not doing that already? I was under the assumption that open ai agents interacted with the environments through vnc.
exabrial 9y ago
OT: Is backdooring prevented by the game engine yet? Annoyed me that a useful tactic was "against the rules"
- banhfun 9y ago
  It's always been prevented in Dota 2. Buildings have backdoor protection, which makes them regenerate lost HP from recent attacks unless there are creeps nearby.
  - darrenkopp 9y ago
    T1 towers don't have backdoor protection though.
  - ryanlol 9y ago
    Not really "prevented", backdoor protection only provides 90HP/s.
    - Smaug123 9y ago
      Backdoor protection also gives a substantial damage reduction - 25%, and much more against illusions.
bronz 9y ago
this whole emphasis that openai and deepmind put on human/robot collaboration is a paper thin pr move in my opinion. the robots will be better than any human, humans will not be able to contribute a single thing soon. but they try to make us all feel safe by making it look like they benefit from our brains.
- cglouch 9y ago
  I know it's not the same type of AI, but in chess there's a whole scene for computer + human play. A chess engine on its own can have trouble seeing strategic ideas that humans can recognize (e.g. opposite-colored bishop endgames, certain closed positions, and fortresses) so an engine on its own will lose to that engine being assisted by a skilled human player. In other words, humans are still capable for contributing at least a little. That said, it's not much - I think a chess GM paired with an engine will probably only be able to beat an engine rated ~100 or so points higher than their own. It will be interesting for deep learning, though, where the ideas are a bit more abstract. Perhaps humans will be useful for a while longer.
- vbezhenar 9y ago
  AI uses many simulations to train its network. It's appropriate for chess or Dota, but you can't do that for real war, for example, there were not enough wars to learn from them and you can't simulate war good enough. Or making a business plan for Oracle corporation: it's unique situation, you don't have millions of Oracles bankrupting to learn from it. Humanity has this knowledge, but it's encoded in books, teachers and experts. So AI will need to communicate with people to adapt to our society, they can offer advices to experts, but they need to learn from those experts first.
terda12 9y ago
As a longtime DotA player and someone who's following the pro scene, this is very impressive. Especially considering how it's beaten Sumail, widely regarded as one of the best 1v1 players in the world. Can't wait to see what OpenAI have in store a year from now for 5v5.
- perishabledave 9y ago
  Sumail won once until they gave the bot insane creep blocking skills. https://twitter.com/Phillip_Aram/status/896162260455800832 https://twitter.com/Phillip_Aram/status/896162260455800832
  - 0x00000000 9y ago
    >The bot didn't recognize items on ground so he expended Mana picked up mango then killed. So yes, he won, but it was more gimping the bot. That's insane that he figured out how to beat it so quickly. I feel there are other ways to cheese it too. Like maybe survive until 6 -> rush shadow amulet -> smoke -> activate and walk into lane during fade time -> ult when the wave/bot is on top of you. I bet the bot has never seen invisibility and wouldn't know what to do
UnpossibleJim 9y ago
These types of MOBA games are a good precursor of military unit management. To be honest, I'm not totally sure whether to be happy or sad about this sort of thing. The strategic control of drone units on a combat field, without the loss of personel (on the countries with this technology, anyways) should make me happy, and does, to a point. BUT the potential for abuse (and, no, I haven't even begun to extrapolate towards the Terminator, nightmare scenarios) is rampant. Fewer people with a conscience on the battlefield or controlling the apparatuses of war may or may not be worth the cost of the lives of young men and women..... but I think wars should be fought by old men and women with swords, anyway. If you're old and can look i to the face of your enemy while you kill them, there's a better chance it's worth killing and dying for (by the numbers, anyways).... plus, the President, Congress and the Senate have to serve in combat positions, in my little fantasy scenerio =)
- bhouston 9y ago
  Yes, this is perfect for a future of swarms of drones fighting it out against each other.
  - trevor-e 9y ago
    I think it's scarily close to happening, or already has. Just the other day I saw the Drone Racing League on one of ESPN's alternative channels, and all I could think about was how effective they would be as kamikaze bombs. If these people can race drones up to 150+ mph and turn sharp corners through obstacles, then the military is already two steps ahead.
- 0xdada 9y ago
  DotA is not a MOBA and it's not a very good precursor for military unit management. I'd say Starcraft is a better comparison, and they are working on applying DeepMind to that right now [0]. [0] https://techcrunch.com/2017/08/09/blizzard-and-deepmind-turn-starcraft-ii-into-an-ai-research-lab/ https://techcrunch.com/2017/08/09/blizzard-and-deepmind-turn...
  - jsmthrowaway 9y ago
nether 9y ago
No!
ahh 9y ago
The developer interviewed claimed that there was no domain-specific knowledge and implied (though didn't explicitly state) there wasn't any training against non-OpenAI bots or players. (I'd love to know the reward function they used for whatever Q-learning variation they ran with.) If this is accurate, one of the things I'm most impressed with is that the bot figured out creep-blocking. (I can't find a good GIF, but this is walking in a wiggly path in front of the first wave of neutrals on your side, delaying their progress and pushing the lane towards you, which is good for ~reasons.) Creep blocking isn't all that hard in dexterity--I am a terrible dota player and I can more or less do it. And it's one of the most common pieces of dota knowledge; every pro player does it and since it's relatively easy compared to a lot of pro micro, everyone else rapidly learns they should. But nevertheless--the bot had enough games that it could randomly jump in front of the wave enough times that it noticed a win rate improvement for that slight wave push, and begin to do it intentionally? (And then get good at it?) Damn. One thing I don't really know about Q-learning and the typical nets used for it: I am guessing it is likely that internally to the bot's evaluation functions, there is some learned feature whose activation correlates well to the location of the wave equilibrium (since that's a feature that correlates well with winning!) At that point, is it likely that the bot can learn in smaller increments--that is, it knows that pushing equilibrium towards itself is good, and thus randomly creep blocking a little becomes reinforced (rather than having to notice the creep block's effect on game wins?)
- xapata 9y ago
  I expect your conjecture is correct. That's the whole point of deep learning -- there are many layers that automate what would otherwise be human feature extraction.
  - visarga 9y ago
    The magic here doesn't come solely from deep learning, but also from having access to massive simulation. Simulation can make an almost average human-level deep neural net become better than the best human. It happened for Go as well, where AlphaGo learned by self playing millions of games. I think there is a deep link between simulation and AGI. An AGI would need to be able to imagine how people and objects would act and react in any situation, which is the same as the ability to simulate the world, or to imagine. We might be able to create small simulations like Dota2, but the real world will be much harder.
Tangokat 9y ago
It is so weird reading all these comments. Almost half of them start out as "it's impressive.. but". Is it human nature or are HN commenters just so sceptical/negative of all the new tech? One of the OpenAI guys mentioned that they could potentially use the same technique in real life applications like surgery. Surgery is not "just" run on a computer it has a physical component too. Is that really the next step or was he just throwing out a random example people could understand?
- burkaman 9y ago
  HN culture strongly discourages unsubstantive comments like "wow, this is great" that don't add to the conversation. It's hard to comment positively and substantively on a story like this, because it requires you to understand the subject matter well enough to describe some interesting feature, or bring up related work. It's usually much easier to find a flaw, or something that looks like a flaw to fellow non-experts. This comment section actually seems ok, but usually you'll see mostly neutral or negative comments.
- kevinwang 9y ago
  >Is that really the next step or was he just throwing out a random example people could understand? More of the latter than the former. He was talking more in broad strokes about ai research in general then this specific project.
- AndrewKemendo 9y ago
  In my years on HN I think I've only seen a handful of comment sections that are predominantly positive or lauditory. That's not to say they are negative, but usually critical or cynical. I think it's good, keeps people on their toes.
- thinkfurther 9y ago
  Why not ask them directly, if you actually want to ask rather than make up? For me it makes as much sense as comparing someone sewing pieces of clothes together with someone pulling a zipper. Did people also have races against cars all the time? Is anyone getting excited over some plastic not changing texture when submerged in water for days or even years, while humans get elephant skin rather quickly? I do find all of that highly interesting, but it's more a morbid curiosity than being amazed. > One of the OpenAI guys mentioned that they could potentially use the same technique in real life applications like surgery. Oh yeah, it will all be for benefiting the elderly and the poor, I'm sure. At some point, ever around the corner. It won't benefit the people who need to have surgery in the first place because the so called civilized world can't even deal with warmongers and power mad cops, can't reign in sheer greed and sociopathy -- but nobody who has excuses today will still have them when faced with fully automatic enforcement systems. Generally, at least allow for the possibility that someone who is not utterly fascinated by something you like might not be less curious and progressive than you, but the opposite, and that what you think is the bigger picture being a fraction of what they see. At least until you actually asked the people whose comments you don't like.
habitue 9y ago
So an interesting thing you could do as a game company with an "unbeatable" bot is to use it to balance a metagame. Let the bots all play each other and tweak character stats etc until they win a proportional amount of the time. (This presupposes that the bot learns the game the way OpenAI claims it is, without needing to learn from player replays or from playing in a very different way than human players do with tree search etc)
- DannyDaemonic 9y ago
  I think balance is a bigger problem than people realize. It's much harder than you'd think, and the most common way of balancing things is to simply make them more "samey". I was part of the Warcraft 3 beta. When it started all the races felt so unique. It's a bit foggy now but I think, for example, all the elves buildings - which appeared tree like - would automatically regenerate. Once they started balancing the game, all these special traits fell away. All the races were homogenized and despite still being different, all played much more similarly. Honestly, it ruined the game for me. I feel if I had just picked it up upon release I would have enjoyed it, but watching all these unique traits and play styles fade away made me realize what could have been.
  - Zarath 9y ago
    Homogenization is a huge problem with nearly every major online game I've played (WoW, LoL, Hearthstone, etc.)
    - intended 9y ago
      Your looking at the blizzard school of balance. Think going to a theme park. It's very pretty and attractive but underneath it all, the experience is always on rails without any deviation. That's one way to deal with complexity. I hate those games, because the moment you become human, and try to do something intuitive but undefined and unwanted by the programmers, you get penalized. Dota doesnt do that, although the recent patch feels like a shadow of such design: but nothing like league or blizzard
darrenkopp 9y ago
Really was quite interesting to watch (https://www.twitch.tv/videos/166172514?t=7h3m10s https://www.twitch.tv/videos/166172514?t=7h3m10s). Honestly, the bot played _extremely_ well, but I think the biggest advantage was how much faster it's reaction time was and it's movements were likely much more precise than a human is with a mouse. I'm pretty interested in seeing their 5v5 results as well. It seems like that will have similar results as the bots can coordinate, but it's still a bit up in the air. I'm really not sure how well a bot like this would do with 4 human teammates though. I would guess the bot would be a strong laner, but fairly weak overall due to it's inability to communicate, though if it's really good at learning how it's current teammates are playing as it goes, it may do alright. I'm also pretty curious about the match limitations: no shrines (regen), no soul ring (mana regen at expense of some health), no raindrops (fixed amount of magic damage block)
- dirtyaura 9y ago
  It would be a fantastic side project to teach bots to learn to speak "Dota". Learning to communicate efficiently is likely much easier as the vocabulary and intentions behind them are constrained.
  - quadcore 9y ago
    Side question: if we consider a team of 5 bots, would they need to communicate?
    - sbarre 9y ago
      If you wanted them to play "within the rules", they would need some way to communicate via the game.. Text chat would make the most sense since adding in TTS / STT would seem unnecessary.. They could even communicate in some kind of shorthand language that only the bots understand..
      - Cyphase 9y ago
        I think the grandparent is asking whether the bots would need to communicate at all, as opposed to just "knowing" what the other bots are "thinking".
deleted 9y ago
[deleted]
ted12345 9y ago
I wonder if some of the things the guy says are misleading. In the video, we see the bot "creep blocking." For those unfamiliar with dota, players can use the model of the unit they control to obstruct the movement of allied computer controlled units in order to gain a favorable position. I suppose it's possible that over millions and millions of matches played against itself, the OpenAI bot "invented" this behavior for itself. But it seems more likely to me that the programmers "built that behavior in."
- mquander 9y ago
  Given that they said explicitly that the bot invented all of its behavior for itself from scratch, it seems more likely to me that it did so.
- popcorncolonel 9y ago
  It would be pretty much impossible for the programmers to "build the behavior in" to the neural network, unless you mean training on supervised data or something.
  - visarga 9y ago
    It's not impossible, it's called inverse reinforcement learning, where they learn a value function from an external demonstration. Then they use this value function for teaching the bot an action policy. Intuitively, the idea is to learn first what are a good state and a bad state, based on external demonstrations, then use that to teach the bot how to act. This kind of learning is similar to GANs, where the discriminator learns from real data and the generator learns from the discriminator.
    - popcorncolonel 9y ago
      Very interesting! Thanks for sharing -- I'll look more into this.
jmkapz 9y ago
I wonder if this a good move for Valve. How many people will be loose interest from the game if next year OpenAI manages to pull off the 5v5 challenge, thus bringing DotA 2 to the same level as GO and chess for AI, i.e. beating the best human(s). Has there been any study on the popularity of chess after AI moved in? How about prize pools, as a large driving force of DotA 2 is the competitive scene? I suppose Valve will find out soon enough and it may be a significant event to study of the impact of AI, if only by the scale (> 10 millions players [1]) and the traceability of the metrics. What if the numbers do indeed plunge? Maybe such public display of AI might be given second thoughts. And what of outside the gaming industry? [1] http://www.criticalhit.net/gaming/dota-2-vs-league-legends-updating-numbers/ http://www.criticalhit.net/gaming/dota-2-vs-league-legends-u...
- hmate9 9y ago
  Of course it's a good move. Millions watched AlphaGo, many of whom have never played Go before. Millions, almost every single gamer will watch AI vs Dota that will be amazing marketing for Valve.
  - jmkapz 9y ago
    I'm talking mid to long term here. To rephrase, is there a sizeable amount of players who will stop playing, knowing that they cannot ever be the best? Because when AI beats one, I don't believe there is a coming back. For the concrete AlphaGo example, of course it was hugely popular but I would be interested in the evolution on the number of players, how many of those viewers started playing balanced by how many existing players lost interest following the matches (excluding GO researchers :) ).
    - vecter 9y ago
      I don't think it will matter at all. Literally, zero impact. Chess computers crush the world's best humans, yet here we are in meatspace, still vying to win the human world chess championship. Also, 99.99999% of players realize they'll never ever be close to being the best in the world, yet they still play the game because it is inherently fun.
rafinha 9y ago
All these fancy stuff and valve still can't detect people feeding chicken...
obastani 9y ago
I think one of the biggest challenges in moving from 1v1 to 5v5 is the substantially increased possibility of creative strategies. In a 1v1, there isn't that much room for innovation, making it a perfect target for existing AI techniques, which are good at learning what is "in the data" but have not yet been shown to be capable of thinking outside the box. In contrast, in a 5v5 matchup, a much wider range of strategies is available. I'd imagine that in this more flexible setting, AIs would be highly vulnerable to cheese strategies, or in general strategies they have never seen before. Furthermore, if there are any exploitable quirks in the AI (which seems quite likely, given the established lack of robustness of neural nets), I have no doubt that clever human players will be quick to exploit them. On the flip side, I think that makes 5v5 a far more challenging and interesting goal that would be very exciting if achieved! As a side note: I think the most impressive aspect of this AI is that it was trained without watching any human games. In contrast, AlphaGo was bootstrapped from a bunch of existing human games, which I imagine was crucial to its training. Being able to learn how to fake out the human player without having seen this action before is quite impressive.
- electrograv 9y ago
  Without even getting into misconceptions and inaccuracies here, the following quotes illustrate "the moving goalposts of AI": The almost-metaphysical "mystique" that many attribute (perhaps subconsciously) to human intelligence vs machine intelligence: > AI techniques, which are good at learning what is "in the data" but have not yet been shown to be capable of thinking outside the box > AIs would be highly vulnerable > exploitable quirks in the AI > established lack of robustness of neural nets > clever human players will be quick to exploit them Of course, it is important to identify pitfalls, so as to address them. But it is sad that no matter how good AI gets, there will always be pessimistic reasoning downplaying its success. With every step forward, there are the predictions of impassible future failures -- "just around the corner"! Due to this mystical "cleverness" attributed exclusively to humans, I'm not convinced a majority of humans will ever allow machines to be considered 'intelligent', no matter how far the technology advances. I'd love to be proven wrong, though.
  - 9y ago
kahlonel 9y ago
Even with a lot of restrictions, I can't imagine how much variables this bot has to take into account; and then generate the output within a sub-millisecond time period. The most interesting part was aggressive positioning of the bot, and faking the spells to scare the shit out of Dendi (the pro player competing against it). Will definitely follow the progress of this project.
wnevets 9y ago
As a long time dota2 player, it was simply absurd just how good it was.
evc123 9y ago
Elon posted these AI fear/regulation tweets right after he retweeted the opeanai dota 2 blog post: https://twitter.com/elonmusk/status/896166762361704450 https://twitter.com/elonmusk/status/896166762361704450 https://twitter.com/elonmusk/status/896169801277517824 https://twitter.com/elonmusk/status/896169801277517824
donovanm 9y ago
this is pretty awesome, hopefully they'll release some more details about how it was implemented
Asdfbla 9y ago
I wonder what the input for the bot was. Just a screengrab like in other reinforcement learning examples (like Pong or GTA etc.) or did they use some Dota API to make their lives a bit more easy? Impressive nonetheless, like others said, knowing the reward function would be interesting.
ematvey 9y ago
To OpenAI folks: are you planning to publish a paper with implementation details?
Hroble 9y ago
If I remember correctly, Deep Mind recently releasead a kit to train bots to play Starcraft 2. Would anyone be able to compare the differences and difficulty of playing dota and starcraft for a bot?
csomar 9y ago
Is this the start of the end of online gaming? If bots outperform humans, what to prevent someone from a running bots for online games (even Poker and Chess) and being sure that everyone loses?
- simonebrunozzi 9y ago
  Or, quite the opposite. I played DotA a few dozen hours, mostly against the computer, to learn. Then I played only a few matches against people, but I generally didn't like them. Why? Because there were expectations that I would know certain things, or that I would perform at a certain level, and the other players were, on average, a bunch of young kids with tons of time to play DotA and their own weird jargon and acronyms for me. I would strongly prefer to play against bots only, and be able to adjust the difficulty, to make the game always challenging and interesting as I progress in skill level. It could also make the "competitor" in me happy by knowing what my (objective) ranking would be.
deleted 9y ago
[deleted]
VHRanger 9y ago
Where's the paper describing the exact methods used?
antouank 9y ago
Seems like the AI was beaten at least 50 times. https://twitter.com/riningear/status/896297256550252545 https://twitter.com/riningear/status/896297256550252545
Dominator 9y ago
You know how some people want a sports league where absolutely every drug is legal so we have the most insane juiced to the gills crazy motherfuckers out there? I want the AI version of that for video games. Completely unshackled, brokenly powerful AIs fighting against each other in just utterly bonkers displays.
Macrosmatic 9y ago
Anyone else interested in seeing this bot Vs itself?
Hekatron 9y ago
Figuring teamplay out is going to be a lot more complicated but a strong 1v1 bot is already very impressive. Props to openAI, that universe didn't take off was a bummer, it was a pretty cool project as well.
bitmapbrother 9y ago
I really don't think this is a big accomplishment. Dota 2 is a team game where 5 players all work together. I went in thinking I would see a 5v5 against bots. They promised a 5v5 against bots next year so we'll see how that plays out.
vfistri2 9y ago
Is there a research paper on this?

510 points by frisco 9y ago

dvt 9y ago

Saw this live and as someone that's played Dota for many years, it's not as impressive as people make it out to be. A Shadowfiend 1v1 is primarily a test of mechanical skill and not judgment. A similar comparison is if a Counter-Strike or Quake bot would instantly snap to your head getting a headshot. Like, yeah, cool I guess, and it would beat 100% of human players 100% of the time, but it's not even remotely as impressive as Chess or Go. The razes in particular are "skill shots" that are sometimes hard for humans to estimate whereas for a robot it's just simple math. I will say that the one impressive aspect of the bot was raze faking, but I think that it was one of those cases where it was "coached" (same as the creep blocking).

jlebar 9y ago
> but I think that it was one of those cases where it was "coached" (same as the creep blocking). What makes you think that? In the interview, they explicitly disclaim this.
- dvt 9y ago
  In the interview, they literally said they coached it to have certain behaviors.
  - cjbprime 9y ago
    I didn't get that from the interview. It implied the bot learned the game itself from a blank slate.
  - keerthiko 9y ago
    They said "yes, it does that"/"it's been known to do that", not "it was coached to do that". Just as a "We've seen it do that before."
    - dvt 9y ago
      Yeah, just re-watched the clip, I think you're right. My mistake (the word "coached" confused me).
    - nnnnnande 9y ago
      4:33 into the video: "[...] and coached it on what we thought was good or bad".
- nnnnnande 9y ago
  No, actually they explicitly state that they also coached it on what they though was good or bad (4:33 into the video). Having said that, I guess we still don't know which game mechanics were coached.
keerthiko 9y ago
It pulled some interesting tactical moves that I was keeping an eye out for. - In game 1, when the AI triggers his flask of health regen (which gets dispelled if you take hero damage) it zones Dendi out knowing he's going to come in for an attempt to dispel his regen. It does this with lots of non-committal, pre-emptive shadowrazes. It doesn't do this any other time. It's aggressively predicting what Dendi may favor based on its own cirucmstances. - In game 2, Dendi said he was going to try letting one creep ahead and see if it would give him a lane advantage. [very technical DOTA mechanics] What this does is if the opponent creep blocks better than you, their creep wave strikes your overextending creep without taking any damage, under their tower, thus significantly weakening your creep wave, making the tug of war swing very heavily in your favor. You counter this by also letting one of your creeps out ahead as soon as you realize it. The AI that was perfectly blocking all the way till the creeps met up in G1, as soon as it saw the leading creep let one of its creeps forward. A perfect response that is highly unlikely to have been coached but determined as the best choice to, what I'm guessing is to return the game to a familiar state. - The bot cornered Dendi between some trees and his tower, and then with 0 fakeouts/hesitation shadowrazed him. It was not just a matter of knowing the target was in range, but also accounting for the probability that he could dodge it. Not 100% certain about the mechanics here, but I believe: Shadowraze has a ~.2s cast animation. It can be canceled during the first .1s (fake out), but after that it is committed to, you will complete the animation, spend the mana, and cast the spell. If the cast crosses this window and the opponent is capable of moving the short distance needed to dodge in this time, it is theoretically not something the AI can guarantee on pure mechanical skill. So cornering is an approach advanced players use to reduce the probability and room for dodging a skillshot, and I found it taking instant advantage of that opportunity very impressive. Besides the obvious mechanical precision of a bot once it's made a decision, the other inhuman factor here is what Dendi mentions, that it has zero hesitation to take advantage of even the smallest opening it has available. A pro human who is even instantly aware of an opportunity, no matter how theoretically certain they are that they should take it, hesitates at least a little bit to think "is there a trap? Am I not considering something?" Hesitation is an incredibly human trait.
- Retric 9y ago
  Bots hesitate via thinking time. You see this in chess matches most obviously when a high level bot spends more time on some moves because it realizes the situation is unusually important or tricky.
pinouchon 9y ago
In this video listing "learned bot behaviors", they show creep blocking as one such behavior: https://youtu.be/wpa5wyutpGc?t=25s https://youtu.be/wpa5wyutpGc?t=25s. So it looks like the bot figured it on its own. (blog post: https://blog.openai.com/dota-2 https://blog.openai.com/dota-2)

brandonhsiao 9y ago

Is it taking in raw pixels or reading from game memory? How did they gather data to train for this? (Did Valve give them an API?)

nacs 9y ago
I'm sure they got some kind of official API since this seems to be a partnership between openAI and Valve. Blizzard's Starcraft did the same thing for their recently announced AI project where Starcraft 2 got an API for the AI developers to work with: https://techcrunch.com/2017/08/09/blizzard-and-deepmind-turn-starcraft-ii-into-an-ai-research-lab/ https://techcrunch.com/2017/08/09/blizzard-and-deepmind-turn...
mIREdeRMedeFLaO 9y ago
https://developer.valvesoftware.com/wiki/Dota_Bot_Scripting https://developer.valvesoftware.com/wiki/Dota_Bot_Scripting > Bot scripting in Dota is done via lua scripting. This is done at the server level, so there's no need to do things like examine screen pixels or simulate mouse clicks; instead scripts can query the game state and issue orders directly to units. Scripts have full access to all the entity locations, cooldowns, mana values, etc that a player on that team would expect to. The API is restricted such that scripts can't cheat -- units in FoW can't be queried, commands can't be issued to units the script doesn't control, etc.
- hhmc 9y ago
  I wouldn't be surprised if they had access to a less restricted AI. Until we see more concrete information I remain sceptical that they are only using the official API.
  - mAritz 9y ago
    Why? Everything it needed for that 1v1 was available. There was nothing in the FoW that needed to be known, it can keep timers about enemy cooldowns, gold and xp (approximations at least since it's somewhat random).

nacs 9y ago

Looks like its also being hosted on Twitch.tv in addition to Youtube: https://www.twitch.tv/dota2ti https://www.twitch.tv/dota2ti

justicezyx 9y ago
Twitch is the official streaming partner for 7 years.

justicezyx 9y ago

When will the match happen or did it already happen? I did not find any vods.

justicezyx 9y ago
OK it's happening right now, I saw machine the host's announcement
- BLanen 9y ago
  Weird website... No indication that a match is happening/happened/ or going to happen or with times or not. I guess I missed it and the stream link is now just a general link to the TI stream, which confused me for a bit.
  - justicezyx 9y ago
    They probably dont expect the dota tournament to have such a large influence scope...

thatsadude 9y ago

I love both Dota and ML, this is awesome. I would love to know whether they release the source code for us to play around.

popcorncolonel 9y ago
It is OpenAI, after all.

deleted 9y ago

[deleted]

sondr3 9y ago

Wow, that was not even close either. And it only takes it two weeks to reach that level of play? OpenAI should create a team of five (as they said they would) and let it play in the qualifiers next year.

hmate9 9y ago
5v5 is much more complex than 1v1 gameplay, it's not as simple as taking the current AI and making 5 of it.
- criloz2 9y ago
  also, it only will be fair if the bots are completely independents instead of a unique bot controlling 5 heroes
- sondr3 9y ago
  I'm not arguing that, just that it would be cool. The creators mentioned on stage that next year it would be a team of five bots. They also said it took the bot two weeks to reach the level it was at right now, so I'm curious to see how/if it'll evolve in the future.
- ronald_raygun 9y ago
  I don't want to be "that guy", but I used to play a lot of LOL, and I feel like 5v5 is in a different ballpark in terms of difficulty than a 1v1. Even champion select is strategically intensive. TSM (one of the best US teams), lost to an amateur team because they got out done in team selection. https://www.youtube.com/watch?v=g7l30-4A6WQ https://www.youtube.com/watch?v=g7l30-4A6WQ
  - hmate9 9y ago
    It is extremely difficult. 1v1 is like beating humans at chess. 5v5 is like beating humans at Go. That's the kind of jump it is.
    - unrealhoang 9y ago

eduren 9y ago

This is really cool outreach on OpenAI's part. So many young people watch The International and I'm guessing that at least a few of them are more interested in CS and STEM from seeing this.

savethefuture 9y ago

Well the presentation was a little meh but very impressive tech they built, excited to see the 5v5.

taion 9y ago

Is the "two weeks" of time in the Dota environment, or two weeks of training time running in parallel as with A3C or something?

hhmc 9y ago
From what the devs indicated on stream it sounded like processing hours (with a ratio of around 300 in-game hours to 1 processing hour).
- nopinsight 9y ago
  Interesting. That means it accumulates the equivalent of 24 * 14 * 300 = 100,800 hours of experience. That's about double the amount of practice one gets for playing 10 hours a day for 14 years.

cissou 9y ago

It looks like it's over…? Someone has a replay of the relevant part(s)?

cissou 9y ago
https://www.youtube.com/watch?v=ac1getNs2P8 https://www.youtube.com/watch?v=ac1getNs2P8

Analemma_ 9y ago

Not that this isn’t very cool, but 1v1 Dota isn’t anything like the full game, it’s mostly a competition of who has better micro. If it can beat a team of pros at 5v5– which is where the imperfect information, short-vs-long term strategy and inter-agent communication challenges come into play— then I’ll be impressed.

popinman322 9y ago
Honestly, you don't even have to make all the AI heroes separate. We automatically assume that each would be controlled by a separate virtual player, but one virtual player could very well control all controllable units on a team. That'd be interesting to watch.

justicezyx 9y ago

To be clear 1v1 mid lane is harder than 1 unit v 1 unit in sc. But it is at least 3 levels behind 5v5 human play. So first it's laining, which is showing in the game. Then is the ganking basically 2-3 player working together. Above that that's 5 men team fight. Then you have the strategy planning in the band pick phase, and in game movement coordination. There are other things like item choices, in game communication etc. I guess bots never tilt... This 1v1 match is definitely a proof of the strength and maturity of modern AI. It will be extremely interesting to see if they can train a bot to achieve the above intelligence. If so, I guess the game is pretty much losing a lot of its appeal.

hellbanner 9y ago
How does having a bot succeed cause appeal to be lost? First you're suggesting that the strategy will be "found" and never beaten (no room for improvement, not even in hero selection). Secondly.. Chess & Go have been around for thousands of years and they still are played en masse daily.
- justicezyx 9y ago
  > How does having a bot succeed cause appeal to be lost? You might think not beating bots is OK. Not for me and many people have been involved in competitive scene. The moment machine beats human, a large part of the competition is gone. The essence is that you want to be the best. But if all you can do is to beat some other inferior opponents, what's the point? > Secondly.. Chess & Go have been around for thousands of years and they still are played en masse daily. This is irrelevant.
  - tyoverby 9y ago
    Human Chess players lost against Deep Blue in 1997; people still play Chess. In fact, they've learned a lot of strategies from the strategically superior AI.
  - taneq 9y ago
    We've had forklifts for decades and plenty of people still enter power lifting competitions.

bhntr3 9y ago

This was fun. It's our intern's last day on the ML infra team here. And he happens to be a competitive collegiate DOTA player. We were all crowded around watching the screen, shouting. Couldn't have planned a better send off. Nice job, OpenAI!

popcorncolonel 9y ago
As a ML researcher and an avid dota fan, I'm jealous! And it must have been great with Dendi the legend there too.

xfer 9y ago

Very impressive, even if there are some limitations. I look forward to more progress for a team of bots.

demonshalo 9y ago

MEH. I am not impressed. Restricting the number of parameters and variables in order to produce a bot that can do 1 sub-set of tasks really well is nothing special imo. Complexity and simulations of what you have not yet encountered is something humans can do with ease. This is not something a bot can do in a complex environment like DOTA. I'll be the first to admit that I was wrong and that AI is truly a thing once I see a bot like this one beat a pro-team in a 5v5. Until then, meh...

criloz2 9y ago
is an improvement against the other bots, it can be really good for training players, I would love to test it my self, but yeah dota is biggest that this, not only 5 v5, also 100+ heroes, and professional player can play a variety of heroes on mid with different matchups and its nuisances, the bot still need it a lot to even dominate 1v1.

kmnc 9y ago

How do things like reaction time, and actions per second work with something like this? Is it just an assumed advantage the ai gets, or does it simulate the limitations of a human? If it doesn't, how big of an advantage is it in an ai versus human competition?

mAritz 9y ago
That was pretty much the entire reason it won. It did some standard high-level 1v1 laning techniques, but if human "conditions"[1] were implemented it would look a lot different. Just the blocking of the lane creeps at the start is already super-human and gives the bot a huge advantage. [1] like lower actions per minute, latency, occasionally missclicking and not being 100% certain about distances

personjerry 9y ago

It was streaming and it's over, so here's the VOD: https://www.youtube.com/watch?v=ac1getNs2P8 https://www.youtube.com/watch?v=ac1getNs2P8

joefkelley 9y ago

This will be HUGE for competitive Dota, even if they never take it further. In much the same ways top chess players have learned from engines, I have to think top Dota players will practice against and study the hell out of this bot if OpenAI makes it available. Not everything is replicable by humans... for instance I noticed it constantly animation-canceling razes and only finishing it if it was going to hit; a human will definitely mess this up. But other things can definitely be used. It was positioning somewhat strangely, for example.

kibwen 9y ago

In the interest of a "fairer" comparison, I wonder how much of a difference it would make to force the AI to simulate mouse/keyboard input and interpret the raw screen buffer output, rather than using direct APIs into the game's guts, to more faithfully emulate its human opponent. I'm guessing the peripheral inputs wouldn't be much of a hurdle, but the image processing step could be very interesting.

Zyst 9y ago
Raw screen is not as big of a deal as it would seem, in DotA and Starcraft you can glimpse most of the information you can see on a screen through the minimap.
- danielmorozoff 9y ago
  It would seem to me that this is an oversimplification. Doing screen based video processing is not simple and is not exact especially where camera movements are independent of game dynamics as well as the difficulties inherent in doing fast frame classification- there has been some good work in this direction recently but far from perfect(yolo, ssd). The route deep mind took in partnering with the game manufacturers to gain inputs directly while receiving global information about game state seems the simplest for their goal of training rl agents. We considered doing a project around this but figured it was a massive iceberg.
  - Zyst 9y ago
    Oh I misread the "Direct access to game API section", I thought they meant making it so that the machine can only see "what's on their screen", so if someone were to cast a long range skill for instance they wouldn't get that information. Which is a distinct advantage for machines. That is indeed, by no means trivial. I do think it's pointless though, that's just another exercise for the sake of making it another exercise.
147 9y ago
How do you know it's not doing that already? I was under the assumption that open ai agents interacted with the environments through vnc.

exabrial 9y ago

OT: Is backdooring prevented by the game engine yet? Annoyed me that a useful tactic was "against the rules"

banhfun 9y ago
It's always been prevented in Dota 2. Buildings have backdoor protection, which makes them regenerate lost HP from recent attacks unless there are creeps nearby.
- darrenkopp 9y ago
  T1 towers don't have backdoor protection though.
- ryanlol 9y ago
  Not really "prevented", backdoor protection only provides 90HP/s.
  - Smaug123 9y ago
    Backdoor protection also gives a substantial damage reduction - 25%, and much more against illusions.

bronz 9y ago

this whole emphasis that openai and deepmind put on human/robot collaboration is a paper thin pr move in my opinion. the robots will be better than any human, humans will not be able to contribute a single thing soon. but they try to make us all feel safe by making it look like they benefit from our brains.

cglouch 9y ago
I know it's not the same type of AI, but in chess there's a whole scene for computer + human play. A chess engine on its own can have trouble seeing strategic ideas that humans can recognize (e.g. opposite-colored bishop endgames, certain closed positions, and fortresses) so an engine on its own will lose to that engine being assisted by a skilled human player. In other words, humans are still capable for contributing at least a little. That said, it's not much - I think a chess GM paired with an engine will probably only be able to beat an engine rated ~100 or so points higher than their own. It will be interesting for deep learning, though, where the ideas are a bit more abstract. Perhaps humans will be useful for a while longer.
vbezhenar 9y ago
AI uses many simulations to train its network. It's appropriate for chess or Dota, but you can't do that for real war, for example, there were not enough wars to learn from them and you can't simulate war good enough. Or making a business plan for Oracle corporation: it's unique situation, you don't have millions of Oracles bankrupting to learn from it. Humanity has this knowledge, but it's encoded in books, teachers and experts. So AI will need to communicate with people to adapt to our society, they can offer advices to experts, but they need to learn from those experts first.

terda12 9y ago

As a longtime DotA player and someone who's following the pro scene, this is very impressive. Especially considering how it's beaten Sumail, widely regarded as one of the best 1v1 players in the world. Can't wait to see what OpenAI have in store a year from now for 5v5.

perishabledave 9y ago
Sumail won once until they gave the bot insane creep blocking skills. https://twitter.com/Phillip_Aram/status/896162260455800832 https://twitter.com/Phillip_Aram/status/896162260455800832
- 0x00000000 9y ago
  >The bot didn't recognize items on ground so he expended Mana picked up mango then killed. So yes, he won, but it was more gimping the bot. That's insane that he figured out how to beat it so quickly. I feel there are other ways to cheese it too. Like maybe survive until 6 -> rush shadow amulet -> smoke -> activate and walk into lane during fade time -> ult when the wave/bot is on top of you. I bet the bot has never seen invisibility and wouldn't know what to do

UnpossibleJim 9y ago

These types of MOBA games are a good precursor of military unit management. To be honest, I'm not totally sure whether to be happy or sad about this sort of thing. The strategic control of drone units on a combat field, without the loss of personel (on the countries with this technology, anyways) should make me happy, and does, to a point. BUT the potential for abuse (and, no, I haven't even begun to extrapolate towards the Terminator, nightmare scenarios) is rampant. Fewer people with a conscience on the battlefield or controlling the apparatuses of war may or may not be worth the cost of the lives of young men and women..... but I think wars should be fought by old men and women with swords, anyway. If you're old and can look i to the face of your enemy while you kill them, there's a better chance it's worth killing and dying for (by the numbers, anyways).... plus, the President, Congress and the Senate have to serve in combat positions, in my little fantasy scenerio =)

bhouston 9y ago
Yes, this is perfect for a future of swarms of drones fighting it out against each other.
- trevor-e 9y ago
  I think it's scarily close to happening, or already has. Just the other day I saw the Drone Racing League on one of ESPN's alternative channels, and all I could think about was how effective they would be as kamikaze bombs. If these people can race drones up to 150+ mph and turn sharp corners through obstacles, then the military is already two steps ahead.
0xdada 9y ago
DotA is not a MOBA and it's not a very good precursor for military unit management. I'd say Starcraft is a better comparison, and they are working on applying DeepMind to that right now [0]. [0] https://techcrunch.com/2017/08/09/blizzard-and-deepmind-turn-starcraft-ii-into-an-ai-research-lab/ https://techcrunch.com/2017/08/09/blizzard-and-deepmind-turn...
- jsmthrowaway 9y ago

nether 9y ago

No!

ahh 9y ago

The developer interviewed claimed that there was no domain-specific knowledge and implied (though didn't explicitly state) there wasn't any training against non-OpenAI bots or players. (I'd love to know the reward function they used for whatever Q-learning variation they ran with.) If this is accurate, one of the things I'm most impressed with is that the bot figured out creep-blocking. (I can't find a good GIF, but this is walking in a wiggly path in front of the first wave of neutrals on your side, delaying their progress and pushing the lane towards you, which is good for ~reasons.) Creep blocking isn't all that hard in dexterity--I am a terrible dota player and I can more or less do it. And it's one of the most common pieces of dota knowledge; every pro player does it and since it's relatively easy compared to a lot of pro micro, everyone else rapidly learns they should. But nevertheless--the bot had enough games that it could randomly jump in front of the wave enough times that it noticed a win rate improvement for that slight wave push, and begin to do it intentionally? (And then get good at it?) Damn. One thing I don't really know about Q-learning and the typical nets used for it: I am guessing it is likely that internally to the bot's evaluation functions, there is some learned feature whose activation correlates well to the location of the wave equilibrium (since that's a feature that correlates well with winning!) At that point, is it likely that the bot can learn in smaller increments--that is, it knows that pushing equilibrium towards itself is good, and thus randomly creep blocking a little becomes reinforced (rather than having to notice the creep block's effect on game wins?)

xapata 9y ago
I expect your conjecture is correct. That's the whole point of deep learning -- there are many layers that automate what would otherwise be human feature extraction.
- visarga 9y ago
  The magic here doesn't come solely from deep learning, but also from having access to massive simulation. Simulation can make an almost average human-level deep neural net become better than the best human. It happened for Go as well, where AlphaGo learned by self playing millions of games. I think there is a deep link between simulation and AGI. An AGI would need to be able to imagine how people and objects would act and react in any situation, which is the same as the ability to simulate the world, or to imagine. We might be able to create small simulations like Dota2, but the real world will be much harder.

Tangokat 9y ago

It is so weird reading all these comments. Almost half of them start out as "it's impressive.. but". Is it human nature or are HN commenters just so sceptical/negative of all the new tech? One of the OpenAI guys mentioned that they could potentially use the same technique in real life applications like surgery. Surgery is not "just" run on a computer it has a physical component too. Is that really the next step or was he just throwing out a random example people could understand?

burkaman 9y ago
HN culture strongly discourages unsubstantive comments like "wow, this is great" that don't add to the conversation. It's hard to comment positively and substantively on a story like this, because it requires you to understand the subject matter well enough to describe some interesting feature, or bring up related work. It's usually much easier to find a flaw, or something that looks like a flaw to fellow non-experts. This comment section actually seems ok, but usually you'll see mostly neutral or negative comments.
kevinwang 9y ago
>Is that really the next step or was he just throwing out a random example people could understand? More of the latter than the former. He was talking more in broad strokes about ai research in general then this specific project.
AndrewKemendo 9y ago
In my years on HN I think I've only seen a handful of comment sections that are predominantly positive or lauditory. That's not to say they are negative, but usually critical or cynical. I think it's good, keeps people on their toes.
thinkfurther 9y ago
Why not ask them directly, if you actually want to ask rather than make up? For me it makes as much sense as comparing someone sewing pieces of clothes together with someone pulling a zipper. Did people also have races against cars all the time? Is anyone getting excited over some plastic not changing texture when submerged in water for days or even years, while humans get elephant skin rather quickly? I do find all of that highly interesting, but it's more a morbid curiosity than being amazed. > One of the OpenAI guys mentioned that they could potentially use the same technique in real life applications like surgery. Oh yeah, it will all be for benefiting the elderly and the poor, I'm sure. At some point, ever around the corner. It won't benefit the people who need to have surgery in the first place because the so called civilized world can't even deal with warmongers and power mad cops, can't reign in sheer greed and sociopathy -- but nobody who has excuses today will still have them when faced with fully automatic enforcement systems. Generally, at least allow for the possibility that someone who is not utterly fascinated by something you like might not be less curious and progressive than you, but the opposite, and that what you think is the bigger picture being a fraction of what they see. At least until you actually asked the people whose comments you don't like.

habitue 9y ago

So an interesting thing you could do as a game company with an "unbeatable" bot is to use it to balance a metagame. Let the bots all play each other and tweak character stats etc until they win a proportional amount of the time. (This presupposes that the bot learns the game the way OpenAI claims it is, without needing to learn from player replays or from playing in a very different way than human players do with tree search etc)

DannyDaemonic 9y ago
I think balance is a bigger problem than people realize. It's much harder than you'd think, and the most common way of balancing things is to simply make them more "samey". I was part of the Warcraft 3 beta. When it started all the races felt so unique. It's a bit foggy now but I think, for example, all the elves buildings - which appeared tree like - would automatically regenerate. Once they started balancing the game, all these special traits fell away. All the races were homogenized and despite still being different, all played much more similarly. Honestly, it ruined the game for me. I feel if I had just picked it up upon release I would have enjoyed it, but watching all these unique traits and play styles fade away made me realize what could have been.
- Zarath 9y ago
  Homogenization is a huge problem with nearly every major online game I've played (WoW, LoL, Hearthstone, etc.)
  - intended 9y ago
    Your looking at the blizzard school of balance. Think going to a theme park. It's very pretty and attractive but underneath it all, the experience is always on rails without any deviation. That's one way to deal with complexity. I hate those games, because the moment you become human, and try to do something intuitive but undefined and unwanted by the programmers, you get penalized. Dota doesnt do that, although the recent patch feels like a shadow of such design: but nothing like league or blizzard

darrenkopp 9y ago

Really was quite interesting to watch (https://www.twitch.tv/videos/166172514?t=7h3m10s https://www.twitch.tv/videos/166172514?t=7h3m10s). Honestly, the bot played _extremely_ well, but I think the biggest advantage was how much faster it's reaction time was and it's movements were likely much more precise than a human is with a mouse. I'm pretty interested in seeing their 5v5 results as well. It seems like that will have similar results as the bots can coordinate, but it's still a bit up in the air. I'm really not sure how well a bot like this would do with 4 human teammates though. I would guess the bot would be a strong laner, but fairly weak overall due to it's inability to communicate, though if it's really good at learning how it's current teammates are playing as it goes, it may do alright. I'm also pretty curious about the match limitations: no shrines (regen), no soul ring (mana regen at expense of some health), no raindrops (fixed amount of magic damage block)

dirtyaura 9y ago
It would be a fantastic side project to teach bots to learn to speak "Dota". Learning to communicate efficiently is likely much easier as the vocabulary and intentions behind them are constrained.
- quadcore 9y ago
  Side question: if we consider a team of 5 bots, would they need to communicate?
  - sbarre 9y ago
    If you wanted them to play "within the rules", they would need some way to communicate via the game.. Text chat would make the most sense since adding in TTS / STT would seem unnecessary.. They could even communicate in some kind of shorthand language that only the bots understand..
    - Cyphase 9y ago
      I think the grandparent is asking whether the bots would need to communicate at all, as opposed to just "knowing" what the other bots are "thinking".

deleted 9y ago

[deleted]

ted12345 9y ago

I wonder if some of the things the guy says are misleading. In the video, we see the bot "creep blocking." For those unfamiliar with dota, players can use the model of the unit they control to obstruct the movement of allied computer controlled units in order to gain a favorable position. I suppose it's possible that over millions and millions of matches played against itself, the OpenAI bot "invented" this behavior for itself. But it seems more likely to me that the programmers "built that behavior in."

mquander 9y ago
Given that they said explicitly that the bot invented all of its behavior for itself from scratch, it seems more likely to me that it did so.
popcorncolonel 9y ago
It would be pretty much impossible for the programmers to "build the behavior in" to the neural network, unless you mean training on supervised data or something.
- visarga 9y ago
  It's not impossible, it's called inverse reinforcement learning, where they learn a value function from an external demonstration. Then they use this value function for teaching the bot an action policy. Intuitively, the idea is to learn first what are a good state and a bad state, based on external demonstrations, then use that to teach the bot how to act. This kind of learning is similar to GANs, where the discriminator learns from real data and the generator learns from the discriminator.
  - popcorncolonel 9y ago
    Very interesting! Thanks for sharing -- I'll look more into this.

jmkapz 9y ago

I wonder if this a good move for Valve. How many people will be loose interest from the game if next year OpenAI manages to pull off the 5v5 challenge, thus bringing DotA 2 to the same level as GO and chess for AI, i.e. beating the best human(s). Has there been any study on the popularity of chess after AI moved in? How about prize pools, as a large driving force of DotA 2 is the competitive scene? I suppose Valve will find out soon enough and it may be a significant event to study of the impact of AI, if only by the scale (> 10 millions players [1]) and the traceability of the metrics. What if the numbers do indeed plunge? Maybe such public display of AI might be given second thoughts. And what of outside the gaming industry? [1] http://www.criticalhit.net/gaming/dota-2-vs-league-legends-updating-numbers/ http://www.criticalhit.net/gaming/dota-2-vs-league-legends-u...

hmate9 9y ago
Of course it's a good move. Millions watched AlphaGo, many of whom have never played Go before. Millions, almost every single gamer will watch AI vs Dota that will be amazing marketing for Valve.
- jmkapz 9y ago
  I'm talking mid to long term here. To rephrase, is there a sizeable amount of players who will stop playing, knowing that they cannot ever be the best? Because when AI beats one, I don't believe there is a coming back. For the concrete AlphaGo example, of course it was hugely popular but I would be interested in the evolution on the number of players, how many of those viewers started playing balanced by how many existing players lost interest following the matches (excluding GO researchers :) ).
  - vecter 9y ago
    I don't think it will matter at all. Literally, zero impact. Chess computers crush the world's best humans, yet here we are in meatspace, still vying to win the human world chess championship. Also, 99.99999% of players realize they'll never ever be close to being the best in the world, yet they still play the game because it is inherently fun.

rafinha 9y ago

All these fancy stuff and valve still can't detect people feeding chicken...

obastani 9y ago

I think one of the biggest challenges in moving from 1v1 to 5v5 is the substantially increased possibility of creative strategies. In a 1v1, there isn't that much room for innovation, making it a perfect target for existing AI techniques, which are good at learning what is "in the data" but have not yet been shown to be capable of thinking outside the box. In contrast, in a 5v5 matchup, a much wider range of strategies is available. I'd imagine that in this more flexible setting, AIs would be highly vulnerable to cheese strategies, or in general strategies they have never seen before. Furthermore, if there are any exploitable quirks in the AI (which seems quite likely, given the established lack of robustness of neural nets), I have no doubt that clever human players will be quick to exploit them. On the flip side, I think that makes 5v5 a far more challenging and interesting goal that would be very exciting if achieved! As a side note: I think the most impressive aspect of this AI is that it was trained without watching any human games. In contrast, AlphaGo was bootstrapped from a bunch of existing human games, which I imagine was crucial to its training. Being able to learn how to fake out the human player without having seen this action before is quite impressive.

electrograv 9y ago
Without even getting into misconceptions and inaccuracies here, the following quotes illustrate "the moving goalposts of AI": The almost-metaphysical "mystique" that many attribute (perhaps subconsciously) to human intelligence vs machine intelligence: > AI techniques, which are good at learning what is "in the data" but have not yet been shown to be capable of thinking outside the box > AIs would be highly vulnerable > exploitable quirks in the AI > established lack of robustness of neural nets > clever human players will be quick to exploit them Of course, it is important to identify pitfalls, so as to address them. But it is sad that no matter how good AI gets, there will always be pessimistic reasoning downplaying its success. With every step forward, there are the predictions of impassible future failures -- "just around the corner"! Due to this mystical "cleverness" attributed exclusively to humans, I'm not convinced a majority of humans will ever allow machines to be considered 'intelligent', no matter how far the technology advances. I'd love to be proven wrong, though.
- 9y ago

kahlonel 9y ago

Even with a lot of restrictions, I can't imagine how much variables this bot has to take into account; and then generate the output within a sub-millisecond time period. The most interesting part was aggressive positioning of the bot, and faking the spells to scare the shit out of Dendi (the pro player competing against it). Will definitely follow the progress of this project.

wnevets 9y ago

As a long time dota2 player, it was simply absurd just how good it was.

evc123 9y ago

Elon posted these AI fear/regulation tweets right after he retweeted the opeanai dota 2 blog post: https://twitter.com/elonmusk/status/896166762361704450 https://twitter.com/elonmusk/status/896166762361704450 https://twitter.com/elonmusk/status/896169801277517824 https://twitter.com/elonmusk/status/896169801277517824

donovanm 9y ago

this is pretty awesome, hopefully they'll release some more details about how it was implemented

Asdfbla 9y ago

I wonder what the input for the bot was. Just a screengrab like in other reinforcement learning examples (like Pong or GTA etc.) or did they use some Dota API to make their lives a bit more easy? Impressive nonetheless, like others said, knowing the reward function would be interesting.

ematvey 9y ago

To OpenAI folks: are you planning to publish a paper with implementation details?

Hroble 9y ago

If I remember correctly, Deep Mind recently releasead a kit to train bots to play Starcraft 2. Would anyone be able to compare the differences and difficulty of playing dota and starcraft for a bot?

csomar 9y ago

Is this the start of the end of online gaming? If bots outperform humans, what to prevent someone from a running bots for online games (even Poker and Chess) and being sure that everyone loses?

simonebrunozzi 9y ago
Or, quite the opposite. I played DotA a few dozen hours, mostly against the computer, to learn. Then I played only a few matches against people, but I generally didn't like them. Why? Because there were expectations that I would know certain things, or that I would perform at a certain level, and the other players were, on average, a bunch of young kids with tons of time to play DotA and their own weird jargon and acronyms for me. I would strongly prefer to play against bots only, and be able to adjust the difficulty, to make the game always challenging and interesting as I progress in skill level. It could also make the "competitor" in me happy by knowing what my (objective) ranking would be.

deleted 9y ago

[deleted]

VHRanger 9y ago

Where's the paper describing the exact methods used?

antouank 9y ago

Seems like the AI was beaten at least 50 times. https://twitter.com/riningear/status/896297256550252545 https://twitter.com/riningear/status/896297256550252545

Dominator 9y ago

You know how some people want a sports league where absolutely every drug is legal so we have the most insane juiced to the gills crazy motherfuckers out there? I want the AI version of that for video games. Completely unshackled, brokenly powerful AIs fighting against each other in just utterly bonkers displays.

Macrosmatic 9y ago

Anyone else interested in seeing this bot Vs itself?

Hekatron 9y ago

Figuring teamplay out is going to be a lot more complicated but a strong 1v1 bot is already very impressive. Props to openAI, that universe didn't take off was a bummer, it was a pretty cool project as well.

bitmapbrother 9y ago

I really don't think this is a big accomplishment. Dota 2 is a team game where 5 players all work together. I went in thinking I would see a 5v5 against bots. They promised a 5v5 against bots next year so we'll see how that plays out.

vfistri2 9y ago

Is there a research paper on this?