5 ms·

I'm starting to get annoyed with these DeepMind publicity stunts. They don't release any code, and don't let anyone verify their results. Their chess AI beating

I'm starting to get annoyed with these DeepMind publicity stunts. They don't release any code, and don't let anyone verify their results. Their chess AI beating Stockfish involved some at least a bit questionable setup. And here? I'm a big fan of TLO, but he's currently not even in the top 50 even with his main race. With Protoss, he's clearly making amateur-level mistakes. Why choose this setting to show off your supposedly superhuman AI? (Edit: This comment was made during the stream and it just looks like my point will be addressed before the stream's even finished, they'll let the AI play against a true top Protoss. Yeah!)

samcday 8y ago
but he's currently not even in the top 50 even with his main race Does that really matter? AFAICT he's in the top #100 [1]. For a game as complex as SC2, it's pretty significant that DeepMind is showing these kinds of results already. It's only going to improve from here right? [1]: https://www.gosugamers.net/starcraft2/players/14064-tlo https://www.gosugamers.net/starcraft2/players/14064-tlo
- codeflo 8y ago
  With Protoss, he's not remotely near that. And that does matter if you intend to make huge a press release in the shape of "DeepMind AI beats top pro player". We'll see in an hour if that's what they're going for.
  - aurelwu 8y ago
    there is a LiquidMana coming on now ;)
    - codeflo 8y ago
      I'm excited. :)
solaxun 8y ago
Even so - to me this is kind of a big deal. Go is fully observable, there might be an intractable number of combinations, but the game is well defined and everything is out in the open. In Starcraft you don't have full knowledge of the game (fog of war), there are infinitely more game paths , you have to scout, plan for the long term (macro), react tactically in the short-term, micromanage, etc. I'm actually shocked at it's performance.
Sweetness 8y ago
Because it's still insanely impressive. I would have been blown away if the AI was merely at the platinum level. This is an AI playing starcraft. Starcraft!
- gambler 8y ago
  >This is an AI playing starcraft. Starcraft! Yeah, it's not like computers were ever able to play StarCraft before. Oh, wait...
  - deleted 8y ago
    [deleted]
  - Zaita 8y ago
    The AI normally picks a set strategy and attempts that. On harder difficulty the AI gets to cheat (more income per mineral collected). The SC2 AI is extremely primitive and at best Bronze level. The AI never really "reacts" to your actions, it's basically always on a set path. DeepMind is orders of magnitude better than the Starcraft AI. FWIW cheating AI in RTS games is extremely common. Often they don't get fog of war, or have higher income rates. RTS AI is very very hard, so this is very impressive.
candiodari 8y ago
In all of the games Alphastar was slower than TLO in the build order. I agree that TLO could have been faster, but this was definitely not the deciding factor. I think the main factor was Alpha* just had incredibly accurate assessment of exactly what part of it's army it needed to defend/attack and was very careful.
- Buttons840 8y ago
  What's AlphaStar's APM. Can it just move every single unit perfectly in a single frame?
  - aurelwu 8y ago
    it's not perfect but definitely superhuman from what I see. These are demonstration matches and afaik they want to tune the "mechanical" abilities of the agents to be somewhat in the realm of the strongest human players but it's not a trivial thing as not every action is equal and just picking a APM number would make the AI way weaker than a human in some areas (splitting marines, building/morphing lots of units ) and way stronger in other areas (multitasking)
  - samfriedman 8y ago
    Commentator just mentioned that the APM they've observed from AlphaStar is comparable to pro players.
    - Buttons840 8y ago
      Yeah, in game 4 vs Mana the commentators mentioned there are APM limitations to keep the APM within the realm of human possibility, but the decision making and multitasking are beyond even the best players. "There is not a pro in the world that can control stalkers that way" said Artosis (a commentator).
    - nightcracker 8y ago
      It really isn't. Human APM is inflated and isn't true APM, humans spam click, repeat actions, make pointless actions a lot, etc. A human can have 300 APM but only make one true action per second. DeepMind never missclicks, doesn't have to move the mouse or fingers, never spamclicks, doesn't make pointless actions, so its APM isn't inflated. I do believe that with this progress AlphaStar with more training could beat humans even with a more realistic APM limit.
Permit 8y ago
>I'm a big fan of TLO, but he's currently not even in the top 50 even with his main race For what it's worth, this comment reminds me a lot of[1]: > FYI, Fan Hui, the current European Go champion that Google DeepMind defeated, is ONLY a 2nd Dan Go player. The Highest Dan ranking in Go is 9th Dan! [1] https://news.ycombinator.com/item?id=10983898 https://news.ycombinator.com/item?id=10983898
superfx 8y ago
> Their chess AI beating Stockfish involved some at least a bit questionable setup. I believe their eventual Science paper addressed these concerns.
- gambler 8y ago
  Which paper? How did they address it?
  - superfx 8y ago
    Science paper: http://science.sciencemag.org/content/362/6419/1140 http://science.sciencemag.org/content/362/6419/1140 NYT article summarizing some of the issues addressed: https://www.nytimes.com/2018/12/26/science/chess-artificial-intelligence.html https://www.nytimes.com/2018/12/26/science/chess-artificial-...
  - gamegoblin 8y ago
    They played a newer version of SF (newest version at the time the paper was written), let SF use opening book, stronger CPU for SF, non-fixed time control for SF.
archgoon 8y ago
>but he's currently not even in the top 50 even with his main race. I don't understand this part of the complaint, and it also seems to go counter to the idea that they should be releasing more data. Do we only want to see the state of these systems once they're beating everyone? I prefer to see the progress of these systems over time. Seeing an AI getting stomped on by a pro player doesn't really show us what an AI is currently capable if most people would also get stomped by a pro player.
apetresc 8y ago
The fact that the community has been able to use the published paper to replicate all of AlphaZero's results in both chess, go (and a number of other similar combinatorial games) and get similar performance should lay your concerns to rest.
- gambler 8y ago
  Can you provide some links?
  - Veedrac 8y ago
    http://zero.sjeng.org/ http://zero.sjeng.org/ for AlphaZero playing Go, and http://lczero.org/ http://lczero.org/ is at least par with AlphaZero Chess based off match reproductions.
  - methodover 8y ago
    LeelaChess is one example for Chess. https://github.com/LeelaChessZero/lczero https://github.com/LeelaChessZero/lczero
    - gambler 8y ago
      This project actually proves the point of the root comment. The community spent tons of time on tuning/training this network, and it still routinely loses to Stockfish, which runs on inferior hardware. It illustrates that: 1. Deep mind kept a lot of information about their methods undisclosed. 2. Despite all the claims of generality, the algorithm requires insane amounts of fiddling to train. Just read their blog. 3. The hype around AlphaZero crushing every other algorithm in chess is overblown. It's competitive, but not clearly superior. Still, Kudos to people running Lela project for doing what DeepMind should have done - describing how things really work and testing in real-life conditions.
- CJefferson 8y ago
  Where?

samcday 8y ago

but he's currently not even in the top 50 even with his main race Does that really matter? AFAICT he's in the top #100 [1]. For a game as complex as SC2, it's pretty significant that DeepMind is showing these kinds of results already. It's only going to improve from here right? [1]: https://www.gosugamers.net/starcraft2/players/14064-tlo https://www.gosugamers.net/starcraft2/players/14064-tlo

codeflo 8y ago
With Protoss, he's not remotely near that. And that does matter if you intend to make huge a press release in the shape of "DeepMind AI beats top pro player". We'll see in an hour if that's what they're going for.
- aurelwu 8y ago
  there is a LiquidMana coming on now ;)
  - codeflo 8y ago
    I'm excited. :)

solaxun 8y ago

Even so - to me this is kind of a big deal. Go is fully observable, there might be an intractable number of combinations, but the game is well defined and everything is out in the open. In Starcraft you don't have full knowledge of the game (fog of war), there are infinitely more game paths , you have to scout, plan for the long term (macro), react tactically in the short-term, micromanage, etc. I'm actually shocked at it's performance.

Sweetness 8y ago

Because it's still insanely impressive. I would have been blown away if the AI was merely at the platinum level. This is an AI playing starcraft. Starcraft!

gambler 8y ago
>This is an AI playing starcraft. Starcraft! Yeah, it's not like computers were ever able to play StarCraft before. Oh, wait...
- deleted 8y ago
  [deleted]
- Zaita 8y ago
  The AI normally picks a set strategy and attempts that. On harder difficulty the AI gets to cheat (more income per mineral collected). The SC2 AI is extremely primitive and at best Bronze level. The AI never really "reacts" to your actions, it's basically always on a set path. DeepMind is orders of magnitude better than the Starcraft AI. FWIW cheating AI in RTS games is extremely common. Often they don't get fog of war, or have higher income rates. RTS AI is very very hard, so this is very impressive.

candiodari 8y ago

In all of the games Alphastar was slower than TLO in the build order. I agree that TLO could have been faster, but this was definitely not the deciding factor. I think the main factor was Alpha* just had incredibly accurate assessment of exactly what part of it's army it needed to defend/attack and was very careful.

Buttons840 8y ago
What's AlphaStar's APM. Can it just move every single unit perfectly in a single frame?
- aurelwu 8y ago
  it's not perfect but definitely superhuman from what I see. These are demonstration matches and afaik they want to tune the "mechanical" abilities of the agents to be somewhat in the realm of the strongest human players but it's not a trivial thing as not every action is equal and just picking a APM number would make the AI way weaker than a human in some areas (splitting marines, building/morphing lots of units ) and way stronger in other areas (multitasking)
- samfriedman 8y ago
  Commentator just mentioned that the APM they've observed from AlphaStar is comparable to pro players.
  - Buttons840 8y ago
    Yeah, in game 4 vs Mana the commentators mentioned there are APM limitations to keep the APM within the realm of human possibility, but the decision making and multitasking are beyond even the best players. "There is not a pro in the world that can control stalkers that way" said Artosis (a commentator).
  - nightcracker 8y ago
    It really isn't. Human APM is inflated and isn't true APM, humans spam click, repeat actions, make pointless actions a lot, etc. A human can have 300 APM but only make one true action per second. DeepMind never missclicks, doesn't have to move the mouse or fingers, never spamclicks, doesn't make pointless actions, so its APM isn't inflated. I do believe that with this progress AlphaStar with more training could beat humans even with a more realistic APM limit.

Permit 8y ago

>I'm a big fan of TLO, but he's currently not even in the top 50 even with his main race For what it's worth, this comment reminds me a lot of[1]: > FYI, Fan Hui, the current European Go champion that Google DeepMind defeated, is ONLY a 2nd Dan Go player. The Highest Dan ranking in Go is 9th Dan! [1] https://news.ycombinator.com/item?id=10983898 https://news.ycombinator.com/item?id=10983898

superfx 8y ago

> Their chess AI beating Stockfish involved some at least a bit questionable setup. I believe their eventual Science paper addressed these concerns.

gambler 8y ago
Which paper? How did they address it?
- superfx 8y ago
  Science paper: http://science.sciencemag.org/content/362/6419/1140 http://science.sciencemag.org/content/362/6419/1140 NYT article summarizing some of the issues addressed: https://www.nytimes.com/2018/12/26/science/chess-artificial-intelligence.html https://www.nytimes.com/2018/12/26/science/chess-artificial-...
- gamegoblin 8y ago
  They played a newer version of SF (newest version at the time the paper was written), let SF use opening book, stronger CPU for SF, non-fixed time control for SF.

archgoon 8y ago

>but he's currently not even in the top 50 even with his main race. I don't understand this part of the complaint, and it also seems to go counter to the idea that they should be releasing more data. Do we only want to see the state of these systems once they're beating everyone? I prefer to see the progress of these systems over time. Seeing an AI getting stomped on by a pro player doesn't really show us what an AI is currently capable if most people would also get stomped by a pro player.

apetresc 8y ago

The fact that the community has been able to use the published paper to replicate all of AlphaZero's results in both chess, go (and a number of other similar combinatorial games) and get similar performance should lay your concerns to rest.

gambler 8y ago
Can you provide some links?
- Veedrac 8y ago
  http://zero.sjeng.org/ http://zero.sjeng.org/ for AlphaZero playing Go, and http://lczero.org/ http://lczero.org/ is at least par with AlphaZero Chess based off match reproductions.
- methodover 8y ago
  LeelaChess is one example for Chess. https://github.com/LeelaChessZero/lczero https://github.com/LeelaChessZero/lczero
  - gambler 8y ago
    This project actually proves the point of the root comment. The community spent tons of time on tuning/training this network, and it still routinely loses to Stockfish, which runs on inferior hardware. It illustrates that: 1. Deep mind kept a lot of information about their methods undisclosed. 2. Despite all the claims of generality, the algorithm requires insane amounts of fiddling to train. Just read their blog. 3. The hype around AlphaZero crushing every other algorithm in chess is overblown. It's competitive, but not clearly superior. Still, Kudos to people running Lela project for doing what DeepMind should have done - describing how things really work and testing in real-life conditions.
CJefferson 8y ago
Where?