24 ms·

Grandmaster level in StarCraft II using multi-agent reinforcement learning

• altmind 7y ago
  There's a great video highlighting irregularities in AlphaStar MMR calculations and Blizzard matchmaking, notably, Alphastar, playing protoss, on average played with people 700 MMR below and victories for much weaker enemies gave unexpectedly huge boost to calculared MMR. When only matched with >6000MMR players, alphastar final had 6 win and 15 losses. There is a problem with MMR calculation used for alphastar. More specific, there is matchmaking problem when alphastar did not get matched against equal-MMR and most of his protos games were against much lower-MMR players, skewing the data for MMR calculations. The example given in video: you won 10 times against 5100, you lost against 7200. The calculated MMR would probably be around 6300. The problem is you wont be matched like this in real game. When matched mostly against the weaker enemies, MMR calculations have a lot of uncertainty. The video, unfortunately in russian https://youtu.be/mpAUufSzaUo?t=1323 https://youtu.be/mpAUufSzaUo?t=1323
• cjbprime 7y ago
  I hope they don't stop here without doing a world champion challenge. It looks like they're still significantly below professional player strength.
  • sanxiyn 7y ago
    They said they will stop here.
• plopz 7y ago
  > placing within the top 0.15% of the region's 90,000 players > 61 wins out of 90 games against high-ranking players This doesn't seem to be quite as commanding as it was in Go. Do we know what MMR it reached or if it consistently beat players like Serral?
  • nightcracker 7y ago
    It hasn't beat Serral once, from the article: "The AI wasn’t able to beat the best player in the world".
    • dmichulke 7y ago
      To play devil's advocate: They didn't say anywhere they scored a win against the top 10 either. To elaborate: to be among the "top 0.15%" of 90k is place ~14 (13.5) in the worst case, so nowhere near the "commanding" abilities mentioned by GP
      • swarnie_ 7y ago
        You'll struggle to work it out, go look at the top 20 of any of the ladders IIiiIII1iI1iII!II
  • CydeWeys 7y ago
    It seems like it's only a matter of time though. StarCraft II also has a rock-paper-scissors nature to it though, so you wouldn't expect even a perfect player to win 100% of the time. There are some strategies that are hard-counters to other strategies, and because of the imperfect information nature of the game, by the time you scout your opponent and see what they're doing, it may be too late to shift and deal with it.
    • Erlich_Bachman 7y ago
      > by the time you scout your opponent and see what they're doing, it may be too late to shift and deal with it. Unless you're Serral and your observers magically come out of nowhere and cover every single pixel of your base.
      • Erlich_Bachman 7y ago
        Like this is happens when you try to surprise Serral https://www.youtube.com/watch?v=HTMIh9wzDjo https://www.youtube.com/watch?v=HTMIh9wzDjo
• mc32 7y ago
  How soon is this put into the hands of battlefield commanders? Not presuming it’s a bad thing, but I wonder when we’ll hear the outcome of a battle was aided by this kind of intelligence.
  • dmichulke 7y ago
    I assume obtaining a decent-sized training dataset is gonna be a bloody business.
    • xrisk 7y ago
      Isn't this trained via playing against itself? I don't think there's a dataset involved.
      • 0-_-0 7y ago
        Right, so you would stage a war in a far away desert between 2 AI armies to train a neural network. For science. I'm calling dibs on the movie script!
      • glckr 7y ago
        Yes, but Starcraft II is the perfect Starcraft II simulator to play against itself in. I don't think we have one of those for war.
      • Voloskaya 7y ago
        AlphaStar is first trained in a supervised way using the publicly available replays from SCII matches. Even if you just consider self-play, I don't see how you would create a simulation that is realistic enough so that you could generalize afterwards to reality. RL today has an extremely hard time dealing with different distributions. You can see the amount of work OpenAI [1] had to do to go from a virtual hand+rubik's cube to real ones, even though you can make a very accurate simulation much more easily. [1] https://openai.com/blog/solving-rubiks-cube/ https://openai.com/blog/solving-rubiks-cube/
        • 7y ago
• nightcracker 7y ago
  > After 50 games, however, DeepMind hit a snag. Some players had noticed that three user accounts on the Battle.net gaming platform had played the exact same number of StarCraft II games over a similar time frame — the three accounts that AlphaStar was secretly using. When watching replays of these matches, players noticed that the account owner was performing actions that would be extremely difficult, if not impossible, for a human. In response, DeepMind began using a number of tricks to keep the trial blind and stop players spotting AlphaStar, such as switching accounts regularly. So instead of fixing the unfairness... they tried to hide it?
  • Nimitz14 7y ago
    Funny how they put it as: > When watching replays of these matches, players noticed that the account owner was performing actions that would be extremely difficult, if not impossible, for a human. framing it like the agent was doing some sort insane play, when the reality is the main evidence of a player being alphastar (other than its garbage decision making) was the fact that it wasn't using hotkeys!
    • deleted 7y ago
      [deleted]
    • ionforce 7y ago
      The correct terminology would be "control groups". The configuration of control groups is visible in replays; AlphaStar's replay data does not have any control groups in it. Though, via its API, it is able to select arbitrary groups of units from anywhere on the map. Personally, I would have loved to see it work through control group management because I think that is something that is important and steals attention away from humans. But probably to the researchers, it is just annoying data to model that doesn't get to the "core" of StarCraft.
  • Iv 7y ago
    To play against alphastar, you have to opt-in. IIRC it states Alphastar will hide itself. One of the things people noticed in replays was the lack of control groups and in the case of zergs, the ability to select larvas directly, which no player ever does. It could have been as simple as removing these quirks.
• hamilyon2 7y ago
  Replays are in replays.zip in supplementary data
  • dpcx 7y ago
    Is there a link to that? A video of this would be interesting to watch.
    • casefields 7y ago
      https://deepmind.com/research/open-source/alphastar-resources https://deepmind.com/research/open-source/alphastar-resource...
• hoseja 7y ago
  There has always been the issue of interface when playing videogames AI vs human. Either give the human a brain-computer interface or give the AI a mouse, keyboard, monitor, robot hands and a camera. Anything else seems inherently unfair.
  • Erlich_Bachman 7y ago
    If you read the paper, this is exactly what they have done, and this is why this achievement is so interesting.
  • Aperocky 7y ago
    Or connect the computer to human neurons directly.
    • Symmetry 7y ago
      You'd be able to save the time it takes a nerve impulse from head to hand that way. It's not much but apparently it can be significant enough that in some DotA games shorter players are disproportionately successful in the roles requiring the fastest reflexes.
      • Shaanie 7y ago
        Is that supposed to be a joke? DotA doesn't rely on reflexes nearly enough for that to matter.
        • UserIsUnused 7y ago
          Data doesn't lie. When so many player play a game, even an almost irrelevant advantage can tip the scale. Or maybe it's a sociological factor, not a phisics ones. But the data doesn't lie.
          • VIPMaster 7y ago
            Sure, data doesn't lie, but where is the data? I'm a pretty seasoned DotA player myself and this is the first I'm hearing of any such phenomenon. I can't think of any hero that shorter players have been more successful on than taller players, or vice versa.
• king-rat 7y ago
  Not being robust to strategies it hasn't seen before is a serious shortcoming in a real time strategy game. That also indicates an interesting flaw in how this model is trained - in the millions of games it plays against itself, how do you ensure that it tries every viable (and some inviable) strategies? Sure, it couldn't best Serral but I wonder how it would fare against Has, a player known for some pretty off the wall builds.
  • jplayer01 7y ago
    > Not being robust to strategies it hasn't seen before is a serious shortcoming in a real time strategy game. I'm not sure if you've actually played or followed competitive SC2. This is absolutely normal. Players will pull something completely unexpected out of a hat and win. The losing player will learn from it in future games. That's just how it goes. Unexpected strategies are really hard to counter when you've never seen them before, and they're often employed to directly counter what you're doing right then and there. So you've been countered and dealt a devastating blow, which means figuring out how to come back from that can be hard to impossible. It's a thoroughly human failing in every way. I'd be more concerned if the AI wasn't able to learn to counter it in future matches.
    • king-rat 7y ago
      I do actually follow competitive SC2. My point is that for DeepMind to master this game in the same way it has with Go and chess, it must be able to anticipate counter plays ad infinitum. Otherwise humans will continue to beat it.
      • chillacy 7y ago
        Are new strategies more important in SC2 than go/chess? From the blog it seems like the AI learns from self-play so if it doesn't come up with a strategy, it won't know how to counter it. But that seems like it should apply to chess and go as well.
    • Nimitz14 7y ago
      I don't think you understood his point. A player doesn't need to have seen a strategy before to react correctly to it. Sometimes a player will pull something completely unexpected out of a hat and lose, because the other guy reacted correctly thanks to his game experience. If you watch any high GM player stream half the time his reaction to what his (worse) opponent is doing is "WTF is this?" as he then proceeds to crush it. Alphastar cannot do that.
• Nimitz14 7y ago
  The title seems to contradict the subtitle: > Google AI beats top human players at strategy game StarCraft II vs > DeepMind’s AlphaStar beat all but the very best humans at the fast-paced sci-fi video game.
  • diggan 7y ago
    Yeah, subtitle seems much better. My impression before reading was that it did beat the top players but reading the articles makes it clear it beats everyone BUT the top players... Also, this part seems a bit weird from the article: > The AI wasn’t able to beat the best player in the world, as AIs have in chess and Go, but DeepMind considers its benchmark met, and says it has completed the StarCraft II challenge. So they didn't manage to beat the best players but consider the challenge complete anyways? I thought the goal was to build something that could things better than humans.
    • xorfish 7y ago
      The difference between top 200 players and the very best players is quite significant in Starcraft II. Some argue that the difference in skill between the best players and a top 200 player is the same as between a top 200 player and a median player.
    • Nimitz14 7y ago
      They're framing it positively to distract from the fact they can't do it, the reality is they've spent millions of $ on compute and yet their agent is still terrible at strategy (and tactics sometimes too), they probably decided it's best to stop now before sinking even more money into it.
      • TaupeRanger 7y ago
        You're getting downvoted for tone probably, but I think you're right overall. They simply can't "win" this game in the same way they "won" Go.
    • UnFleshedOne 7y ago
• sjunlee 7y ago
  I won’t agree until they beat South Korean players at Starcraft I.
  • ajudson 7y ago
    They didn't even beat South Koreans at Starcraft 2 in the article
• jplayer01 7y ago
  Honestly, the article needs to be replaced with https://deepmind.com/blog/article/AlphaStar-Grandmaster-level-in-StarCraft-II-using-multi-agent-reinforcement-learning https://deepmind.com/blog/article/AlphaStar-Grandmaster-leve... which actually goes into some technical detail. Nature.com's article is purely for laypersons and, imo, not particularly useful for HN's crowd because of how little insight it gives. It also provides the paper and an archive of all of the AI's matches for anybody who wants to take a closer look. These can be viewed with the free version of SC2 (afaik). Further links: https://doi.org/10.1038/s41586-019-1724-z https://doi.org/10.1038/s41586-019-1724-z (supplementary data available here in json form) https://rdcu.be/bVI7G https://rdcu.be/bVI7G (public paper) https://deepmind.com/research/open-source/alphastar-resources https://deepmind.com/research/open-source/alphastar-resource... (replays) https://www.youtube.com/playlist?list=PLtFBLTxDxWOSrWZ8krQt6eDNXTpG67Xpf https://www.youtube.com/playlist?list=PLtFBLTxDxWOSrWZ8krQt6... (list of older AlphaStar matches cast by an SC2 player) https://www.youtube.com/watch?v=l82wBa3UoZU https://www.youtube.com/watch?v=l82wBa3UoZU (one of the newer matches cast by another player) https://old.reddit.com/r/starcraft/comments/dpaunw/deepminds_alphastar_ai_has_achieved/f5u9fx8/ https://old.reddit.com/r/starcraft/comments/dpaunw/deepminds... (win/loss rates across all the played matches by race, includes apm)
  • dang 7y ago
    This comment was originally posted on https://news.ycombinator.com/item?id=21408024 https://news.ycombinator.com/item?id=21408024, but we've merged it into the earlier submission, which used the link you mentioned.
    • diminoten 7y ago
      The viewers of HN need Nature just as much as anybody, stop deluding yourself into thinking this is some kind of ivory tower...
      • nemonemo 7y ago
        For those who wonder, the Nature article is here: https://www.nature.com/articles/d41586-019-03298-6 https://www.nature.com/articles/d41586-019-03298-6
• imvetri 7y ago
  Why they wouldn't solve more impactful problems rather than playing games?
  • username90 7y ago
    Games are a lot easier to work with than real problems since they are easy to simulate and evaluate. So using games as a stepping stone to learn how to make AI's do complex things seems reasonable to me.
  • linuxftw 7y ago
    Because that's all they're capable of doing with it. Games can be easily repeated, have clear outcomes. There's no need for nuanced thinking or actual problem-solving. Just approximate the current scenario against previously successful strategies and pursue that strategy. A real challenge would be to invent a new game, have both a human and an "AI" read the rules for the first time, and then compete in their first game together. Would the "AI" be able to formulate a winning strategy without historical data? If not, then it's not "AI."
    • sinuhe69 7y ago
      There is no deep learning without massive data. What do you expect? :D
      • linuxftw 7y ago
        I expect less hype and more substance. I don't think computers beating people at video games (or really, any games) is particularly novel at this point, nor do I think it qualifies as "AI".
    • chillacy 7y ago
      > Just approximate the current scenario against previously successful strategies and pursue that strategy. Just about sums up business school.
• raldu 7y ago
  Link to all game replays: https://deepmind.com/research/open-source/alphastar-resources https://deepmind.com/research/open-source/alphastar-resource...
• xorfish 7y ago
  https://www.youtube.com/watch?v=3HqwCrDBdTE https://www.youtube.com/watch?v=3HqwCrDBdTE Here is a video of someone finding out that he played against alphastar. (He won)
• ryanmercer 7y ago
  Am I the only one that doesn't care about software beating humans at video games? I'm sure there's some sort of useful learning being done here by the people that created the software that might some day help them create software that can better predict what is needed for a specific application but to me it just feels like entities like OpenAI and Google researching this predictive software are just wasting obscene amounts of money. How about train the stuff on better typed OCR? Better handwriting OCR? This will have actual commercial application. Why not make something for grading papers. English papers, math homework, etc? Start at a first grade level and as you train the software up move on to higher levels of education. My fiance is a high school teacher, she currently only teaches math but previously has done math and English, she sits there grading papers off the clock while watching television in the evenings and on the weekends... MANY high school teachers are in this situation, think of how much free time could be reclaimed by training this instead of teaching software how to beat humans in video games!!! This would even help teachers have more time during school hours to help struggling students, if you aren't trying to grade papers in class while students are doing work you free up time that you could actively be assisting one or more students. Instead, these "AI" researchers keep training software to be the best at video games... facepalm Then take that and apply it to something like my job. I clear international freight through customs for a living, I look at paperwork all day and then have to determine what tariff number I should use for something (cell phone 8517120050, silver ring over 1.50 USD 7113115000) and classify every single line on an invoice by using familiarity with the tariff schedule/description(s) on the invoice/any other supporting documentation/customer profiles for customers that have paid to have product databases on file with us. My employer, we do thousands of these things a day and have to keep everything for several years (5 IIRC) in the evening CBP or any other OGA wants to see it during that time period. So take that huge, pre-existing, data set like that, identify which shipments were classified correctly and which were not, and then let the software have a stab at trying to do it with that better OCR you created. Then, you know, actually get rid of (or drastically reduce) soul crushing, mind-numbing, highly repetitive digital paperwork jobs like mine.
• CzarnyZiutek 7y ago
  calculator beats top human at multiplication (1957) AI is just a f.... calculator.
  • Sohcahtoa82 7y ago
    I learned how to use a calculator in Elementary school, that doesn't mean I knew how to solve calculus problems. AI needs to know what to calculate. According to the article, at any step in the game, there are 10^26 possible actions to take. Good luck calculating which ones to try via brute force.
• bsaul 7y ago
  Has anyone read the actual paper ? This summary really makes it look like "mission accomplished", but this was much much more interesting than that. We saw AI do "obviously stupid things", and we also saw them improve a lot in the middle of the trial, as many youtubers showed. AI was also much more interesting when playing the protoss race, and really felt like it was responding to the opponents actions, and the other races not so much. But most surprising is that it didn't make any "breathtaking" moves or actions, as opposed to AlphaGO. Actually not a single game made pro player realize something new about the game. Which is really embarassing, because it suggests that the AI just was able to correctly reproduce existing strategies and build orders, that it probably "learned" from existing pro games in the training sample. I was really hoping for a more interesting report, honestly explaining the shortcomings of the technics used and giving hints for the obvious blunders. As well as a roadmap for a second round, this time aiming at beating the very top players.
  • the_af 7y ago
    > Actually not a single game made pro player realize something new about the game. Which is really embarassing, because it suggests that the AI just was able to correctly reproduce existing strategies and build orders, that it probably "learned" from existing pro games in the training sample. Why is it embarrassing if the AI behaved like a collection of all the knowledge of existing pro players? Obviously it'd be more interesting if we also learned something new about the game. Or AI, or both.
  • Jyaif 7y ago
    > Actually not a single game made pro player realize something new about the game. People are now over saturating their mineral line (making more probes than before), so I don't think that's true.
    • jcampbell1 7y ago
      There were other cases of this such as 3 rather than 2 early air units. The AI seemed to anticipate losses in a way that is almost more rational than humans.
• interblag 7y ago
  This is a really interesting one to digest. As with previous announcements about AlphaStar, much of the feedback (here and elsewhere) is about the fundamental challenge of assessing human vs. machine in an RTS. These points are very valid - stepping back however, this still feels like a pretty incredible accomplishment. I'm a gold league SC2 player, so maybe in the 30th-50th percentile. Three years ago, when DeepMind started this project (and after nearly two decades of research into SC/SC2) I could probably have beaten the best non-cheating AI. Now, after 3 years, this AI is playing at a Grandmaster level, under at least a reasonable approach to fairness. By comparison, according to the AlphaGo paper [1] the best Go AIs prior to AlphaGo were playing at a 6 Dan level, which looks to be somewhere in the 90-98th percentile [2]. The speed at which AlphaStar overtook previous AIs seems to me to be nearly unprecedented in AI research. This is like if the world's best chess AI had gone from losing high school tournaments to being competitive with Kasparov in less than 3 years. Valid criticisms aside, this feels like an incredible achievement. [1] https://www.nature.com/articles/nature16961.pdf https://www.nature.com/articles/nature16961.pdf [2] https://senseis.xmp.net/?KGSRankHistogram https://senseis.xmp.net/?KGSRankHistogram
  • baddox 7y ago
    > The speed at which AlphaStar overtook previous AIs seems to me to be nearly unprecedented in AI research. Is it not simply the case that, before AlphaStar, very little money and effort was being put into developing AIs for Starcraft 2?
    • minblaster 7y ago
      Starcraft is an incredibly complex game. 10^26 possible moves at any point (you can click/drag anywhere on the screen, pressing a keyboard button as you do so), imperfect information, real-time constraints, etc.
      • nuclx 7y ago
        Yes it is, but you can prune the decision space significantly by discarding bogus choices, or more concretely by modelling higher level goals and then compiling them into actual moves.
• cryptofits 7y ago
  I wonder which game is more complex, start-craft or DOTA
  • deleted 7y ago
    [deleted]
  • tim58 7y ago
    The complexity of DOTA comes, in large part, to teamwork.
• Razengan 7y ago
  1. Create a realistic military combat game with AI that learns from thousands of players and builds on that knowledge across multiple games/matches. 2. Load that AI into real-world Terminators.
  • uname_a 7y ago
    Yes, this was exactly my reaction: this work seems to be depressingly applicable to real-world warmaking, yet we are only focused on the technical successes, and not on the systemic dangers that we are heedlessly creating. Assuming that everyone looking at this work is a friend who will not turn it to military advantage is extraordinarily naive on our part.
  • gpm 7y ago
    3. Discover that your realistic military combat game neglected the effects of the sun heating your robots and that they now consistently overshoot targets during the day and undershoot targets during the night. (And the million other complexities that exist in the real world). Fundamentally I think the biggest advance in AI for real world applications (military and otherwise), is one that can learn quickly enough that we can train it in the real world situations.
  • friendlybus 7y ago
    3. Discover that a kid holding a fork can walk up to your terminator due to RoE and disconnect it's battery pack.
• Mikeb85 7y ago
  As impressive as this is, APM matters a lot in an RTS such as Starcraft and computers have a massive advantage here. I'd like to see them tackle an RTS like Civ where APM doesn't matter whatsoever, only decision making.
  • nullbyte 7y ago
    Interestingly, AlphaStar has very low APM compared to human players. This is because humans do a lot of insignificant actions when button mashing, whereas AlphaStar is extremely precise with its moves. That's why when you compare APM graphs between Alpha and a human player, the human almost always ranks higher in actions-per-minute.
    • Mikeb85 7y ago
      It's still a high APM game. Reducing redundant button presses still doesn't change the fact that APM is very significant and that being able to process information and act quickly is more important than pure strategy.
      • nullbyte 7y ago
        That's the difference between APM vs EPM in Starcraft.
• Trav5 7y ago
  What do you guys think about having the ai communicate to a person and having them execute the moves?
  • UnFleshedOne 7y ago
    This is how SkyNet starts -- out of sheer frustration.
  • gpm 7y ago
    Having played starcraft - the idea doesn't work. Starcraft happens to quickly, with too much precision, and too low a tolerance for latency to wait for a human to absorb understand and execute instructions. Having programmed simple NNs the idea doesn't work. The amount of time you'd have to spend having humans execute instructions during training would be astronomical. They were training 16000 games simultaneously, for 44 days, most likely running at some multiple of how fast the game normally runs. Moreover you lose out on supervised learning, because we don't data sets of "the human told the other human to do this", we only have data sets (almost a million games large) of "the human did this".
• yters 7y ago
  What would be interesting is to limit the AI processing speed to human capacity, which is something like 60 bits per second. In all these AI v. Human games I see, it is really apples to oranges because the human consumes vastly less resources and compute cycles to perform at the same level as the AI. And when I say 'vast' I mean Vast. There is like a quintillion factor difference between the AI and the human. There is no way the AI is even comparable to the human. To be comparable, we'd have to parallelize the game and lock millions of humans onto the game full time for millions of years. At the end of the day, the AI is just a more sophisticated lookup table. There is as yet no analogous AI to human play.
  • Mangalor 7y ago
    There's a lot of knobs to wiggle here. I'm more interested in finding out what emerges in the system that we can actually call "AI" the more restraints we put on it.
  • whichquestion 7y ago
    It would be interesting to see if there are any meta reports on people doing research on the topic of resource constrained AI. Has anyone explored whether or not an AI can improve on its own algorithm while being heavily resource constrained? Is the ability to use less resources a limit of our current hardware? Do we see the current hardware performance improvement trajectory being able to reduce the amount of resources an AI consumes to perform a task? Are our algorithms simply not tuned well for using smaller resources? Could we build better algorithms around resource deficient environments?
    • yters 7y ago
      Great questions. From what I can tell, AI is just as resource hungry as it ever was, if not more. The only difference is now we have more resources to throw at the traditional algorithms.
  • chillacy 7y ago
    I think we should just accept that humans and computers will be good at different things. Otherwise when AlphaStar 5 walks on stage and smiles at the camera we'll be criticizing its use of fuel cells instead of digesting food.
• ogig 7y ago
  So they added exploiter agents showing the AI what strats are on the meta. When the meta changes, will it still perform? Maybe im reading it wrong but seems much less impressive with the exploiter agents in the mix.
  • aketchum 7y ago
    My understanding is that the exploiter agents are not based on the meta. My take is that the exploiter agents have a different goal than the main agent. While the main agent is trying to develop a strategy to win against as many opponents as possible, the exploiters are focused on finding new ways to beat just the main agent. It seems similar to the inspiration behind GANs
    • ogig 7y ago
      I think you are right, from the paper: >During training we used three agent types that differ only in the distribution of opponents they train against, when they are snapshotted to create a new player, and the probability of resetting to the supervised parameters. So exploiter agents aren't fed a specific strat, instead they discover the weak spots in the same way as the main agent tries to win. The GANs similarity is there.
• angel_j 7y ago
  I don't think folks realize how much of a game changer AlphaStar is. If it can do the same "job" those high level players, it can probably do any job that doesn't require lifting. It could be playing all the traffic lights in a city. It could manage a fleet of factory drones. It could coach humans engaged in sport or business. It could probably figure out spreadsheets. Drill down, and perhaps it can play with chemical synthesis and quantum theories, or drive a truck.
  • aketchum 7y ago
    It is definitely amazing and a huge step forward for RL. However, the paper says that it started off with a policy that beats 84% of players based on imitation learning, so it didn't learn this strategy all on its own from scratch. Also, this required hundreds of thousands (millions? some large number) of simulations to learn, it is much more difficult reach that scale of learning in the real world.
  • empath75 7y ago
    Or control a fleet of armed drones, more likely. I’d be surprised if darpa, etc aren’t investigating applications like that right now.
    • 83457 7y ago
      Isn't it likely they have been doing that for decades?
  • 6gvONxR4sf7o 7y ago
    Sure, if you give it a few bazillion years of practice. We haven't figured out sample complexity yet (the amount of experience required to learn).
• readparryrepost 7y ago
  Love the work done into this, SC2 is such a 'human' game I'm impressed that it can be so successful but I'm wondering whether the model takes the raw graphical information or whether it takes some kind of full map representation. Some kind of machine representation might feel like cheating just a little as it's not quite competing on the same playing field as a human player.
  • sanxiyn 7y ago
    It does not use raw graphical information, information is machine readable.
• jcampbell1 7y ago
  I know this is an endless discussion about fairness, but they handicapped it to have the capability in terms of EAPM of the worlds best player at peak. The result is that it could beat all but the top players. While the mechanics of not using hot keys was strange, I never saw it do anything the elite players couldn’t do. It is worth watching the match between alpha star and Serral who is current best player. Serral beats it like a walk in the park. https://m.youtube.com/watch?v=_BOp10v8kuM https://m.youtube.com/watch?v=_BOp10v8kuM
• nullbyte 7y ago
  But can it beat Serral?
  • vkou 7y ago
    Not at the moment.
• tossAfterUsing 7y ago
  yeah, but can it should political messages on a livecast?
• narrator 7y ago
  We need to create an open source organization to build these brains at home because: "Recomputing the AlphaGo Zero weights will take about 1700 years on commodity hardware."[1] I see these huge AI brains creating a new class divide between people who have access to these new AI brains and those who don't. The mission of open source has always been to break down these barriers to empowerment with technology. Thus, this is a great area for open source innovation. [1] https://github.com/leela-zero/leela-zero https://github.com/leela-zero/leela-zero
  • sanxiyn 7y ago
    AlphaGo Zero was extremely computation inefficient because in (IMO misguided) attempt to be "general", DeepMind avoided even slightly specializing training to Go. KataGo optimized training to Go, slightly, and obtained 100x speedup, and that was low hanging fruit.
    • narrator 7y ago
      1700 years sped up 100x is still 17 years though. Even so, for the models that actually need it, Google and other people with huge GPU farms will have an enormous advantage over a single individual and only a large community can have any hope of challenging their ability to dominate.
• gambler 7y ago
  From the paper: >Humans play StarCraftthrough a screen that displays only part of the map along with a high-level view of the entire map, to e.g. avoid information overload. The agent interacts with the game through a similar camera-like interface What exactly does that mean? Does it or does it not play by operating purely on image data human players would see on the screen? How much of the system's interaction with game's interface is learned as opposed to hand-crafred and filtered through APIs? It's amazing that most people here seem to think that system's ranking in a computer game are more important than its ability to learn from and interact with unstructured data.
  • redkoala 7y ago
    It's meant to operate purely on data viewable on the screen to mimic a human player's experience and provide an even playing ground.
    • gambler 7y ago
      Can you cite something that confirms your assertion? From what I'm reading in the paper, it sounds like there is some custom interface in play: >AlphaStar can target locations more accurately than humans outside the camera, although less accurately within it because target locations (selected on a 256x256 grid) are treated the same inside and outside the camera. It's really hard to parse what such statements mean. The fact that someone who is cited as a co-author of the paper approved the interface as "fair" isn't particularly reassuring.
      • Erlich_Bachman 7y ago
        > The fact that someone who is cited as a co-author of the paper They are not just some random person, they were top SCII players. Who else if not them would know this well enough to make assumptions? There is a custom interface in the sense that the bot does not read pixels from the screen - it reads the information through the API, same information that is usually presented on a screen. But the amount of information is limited to exactly what a human would see at the same time using the standard SCII interface.
• pmoriarty 7y ago
  Starcraft is a relatively simple game (which is one of the main reasons for its enormous popularity), so seeing an AI do well at it is not particularly impressive. I'd have been far more impressed had an AI beaten some of the best wargamers at a complex wargame, or had it beaten some of the best text-adventure game players at novel text adventure games neither of them had played before. The former would be difficult because of the enormous search space that results from the large variety of units, possible actions, and board size in complex wargames, and the latter is from the creativity required to do well at unfamiliar text-adventure games (which are hugely varied and can't be boiled down to any simple set of rules).[1] Text adventure games would present a further difficulty in that not only do they not have very many players compared to games like chess, go, or Starcraft, but their players aren't normally ranked relative to one another, and there's no enormous record of human-played games to draw on. Further, in text adventures sometimes there are clear winning conditions and sometimes not, and sometimes clear signs of progress and sometimes not. It's also hard to objectively rate a text adventure game's difficulty. Without these features researchers would have a much harder time training an AI to do well against humans. Another game which I briefly thought about nominating as a challenge for AI's was Factorio, due to its enormous complexity. However, when I thought about it for a little while I realized that this was a game that computers would find it easy to beat humans because they'd do a much better and faster job at number crunching and optimizing. Factorio is really more of a game where a human almost tries to emulate a computer, much like programmers do when they're mentally evaluating or stepping through an algorithm or trying to optimize something. Of course, seeing an AI beat humans at any of these games would still make the front page of HN, but most gamers (nevermind most non-gamers) who would probably just say "What?" ... and that's probably yet another reason for the reasearchers choosing Starcraft. It's relatively easy, low-hanging publicity fruit, where they could claim success and a relatively large number of people would kind of understand why it was an accomplishment. [1] - Yes, I'm aware of the recent HN post about fuzzing done on the Z-machines, but by the author's own admission that fuzzer cheated by having access to the game's word dictionary, and it also cheated by having access to the game's internal state -- neither of which a human player would have access to. Also, many Z-machine games are ancient and relatively simple and straightforward compared to more modern games.
  • 7y ago
• sgillen 7y ago
  I see a lot of comments downplaying this achievement, saying it's not impressive or it won't be impressive until X condition is met. I welcome skepticism and criticism for this sort of thing, and think most of it that I've seen here is well founded. But I would like to take a second to explain why I think this, and really all the progress in this area, is actually a really impressive achievement to me. Let me try and frame this from the computers perspective. Let's assume a resolution of 1024x780. I'm not sure what size frames they actually feed their agent but it's not that important to the discussion, the point is it's a big image, and according to the article this agent is learning from pixels. So, you the computer are given let's say 1024*780 = 798720 numbers to look at. You then choose a number between 0 and 798720 (or the crazy 10^26 number the article gives as the possible number of actions at each frame) as your action for that frame, and then you get another 798720 numbers to look at. After the round is over (on average 20 minutes, if you make a decision every frame that's 20x60x60 = 72000 rounds). You get one number telling you how well you did. You repeat the process and get a new number. It's higher this time! But what is the cause? was it that click you made on frame 22456? or maybe that unlikely move you made on frame 4567? Obviously I'm oversimplifying here, and the numbers are probably wrong. but I still think what I've said gives the right idea for what kind of task we (as a society/community/whatever) have somehow gotten a computer to solve. Computers are DUMB, the fact that it's able to play this game at all, let alone at a high level, is still a minor miracle to me.
  • timeattack 7y ago
    I think that downplaying the achievement is more like defensive mechanism which exists to protect illusion of human brain superiority.
    • sgillen 7y ago
      Part of the problem is the hype in the other direction. I guess there are those who claim that this sort of result constitutes super human intelligence or something close to AGI. When in reality this is a hyper specialized system that has taken many man years of work to realize.
    • TaupeRanger 7y ago
• Strom 7y ago
  > Agents were capped at a max of 22 agent actions per 5 seconds, where one agent action corresponds to a selection, an ability and a target unit or point, which counts as up to 3 actions towards the in-game APM counter. Moving the camera also counts as an agent action, despite not being counted towards APM. I'm happy to see that they've greatly improved the APM cap. During the earlier showmatch they had an extremely naive cap on the average APM during the whole game. Which meant that the AI could take it easy for most of the time and then during battle sequences hit peaks of 2000 APM. Also the previous cap was based on the in-game APM counter, which doesn't count some things like moving the camera, which is now addressed. The current state sounds a lot better. However it seems still superhuman in mechanical non-strategy ways, e.g. a human can misclick (click too early/late when moving the mouse, or just miss the target with the cursor completely) or do accidental double clicks etc. These end up being very costly mistakes against a zero mechanical mistake AI. Which in turn means that the AI can win with an inferior strategy due to the extra gains it has via superior mechanical accuracy. In other words this artificial intelligence is still under significacnt artificial motor skills welfare, and thus even if it starts beating pro players we shouldn't be too quick to talk about how it's the intelligence part that got it the win. All of that said, I'm liking the progress and am excited to see what they achieve with this next. Would love another showmatch against pro players.
  • Erlich_Bachman 7y ago
    Yes, those might have some impact, but it is clear that the progress is there, with this new APM cap and camera movement etc. You can also see in replays that the AI often makes mechanical mistakes, missing spells, missing units, even ordering wrong units from outside the screen - so it surely seems that if it's win rate was conditioned in any strong way on its sheer mechanical ability, it would have learned to not make any such mistakes. Since the mistakes are clearly there - it seems that its power comes primarily from somewhere else, likely from the AI ability to choose the right actions, not from mechanical power of executing them perfectly. Thus it is not likely that "this artificial intelligence is still under significacnt artificial motor skills welfare". They have consulted top players before they set this AI up, and if those details were important, they would have baked them into the limitations list already.
• hartator 7y ago
  Instead of playing the APM rate, maybe they can slow down the game for the human players? If you play at 0.25x or 0.5x, it will be harder for the AI to out micro. Couple of micro strategies used by the AI, like staying at a 7 range when enemy can fire at 6 range is still a big reason why they are so effective.
• Madmallard 7y ago
  Time to do League of Legends next. I wonder how they would model teammates behavior?
• hervature 7y ago
  Having a hard time to parse what is the action space here. The paper claims: AlphaStar’s action space is defined as a set of functions with typed arguments Looking at citation 7, it seems like they are structuring the action space as (First pick high level action)->(Pick argument 1 for action)->...->(Pick argument n for action). If this is the case, this seems to be "cheating" calling this AI as humans have completely picked out the actions. That is, the achievement here this: given what humans consider useful actions, AlphaStar can play at a grandmaster level. The achievement here is mostly engineering in my opinion. One that extends far further than the 40ish people list on the paper. Probably an effort of over 1,000 people. From casually looking over the paper, there is nothing significantly different than AlphaZero or previous art. Again, the achievement here is listed under the infrastructure section of the paper. In summary, this is a great step forward but now we need to start developing techniques to learn these action space hierarchies instead of throwing more power at increasingly difficult games.
  • sanxiyn 7y ago
    It is extremely different from AlphaZero... In fact, they heavily rely on human knowledge, which is like opposite of AlphaZero. To quote the paper, "We found our use of human data to be critical in achieving good performance with reinforcement learning".
    • hervature 7y ago
      Ok, you’re right. I should’ve said AlphaGo. But that in itself shows what I mean that this is almost a step backwards.
      • sanxiyn 7y ago
        AlphaZero was miraculously good, almost to the point of straining credibility. AlphaGo and AlphaStar are more like normal advances. They are mostly engineering, although theoretical contributions are not trivial. (Using reinforcement learning for value network in case of AlphaGo, and multi-agent self-play setup in case of AlphaStar, since straight self-play doesn't work.)
• jdpigeon 7y ago
  I'm being lazy and don't want to dig, but does anyone know which of the AlphaStar networks (Protoss, Terran, Zerg) is best at beating the others?
  • sanxiyn 7y ago
    They didn't publish cross-race self-play statistics, but AlphaStar's MMR is 6275 for Protoss, 6048 for Terran, and 5835 for Zerg. So it is best at Protoss, and then Terran and Zerg, in that order.