2 ms·

> starcraft isn't really an interesting problem for AI to solve I completely disagree. This is much more interesting than the toy problems that most academics

by NavinF 5y ago

> starcraft isn't really an interesting problem for AI to solve I completely disagree. This is much more interesting than the toy problems that most academics work on. RL problems in this space are inherently difficult since you only get 1bit of “ground truth” information (win/loss) at the end of a game and have to guess which of your actions contributed to that outcome. Sure you can throw hardware at the problem, but you’ve got a huge incentive to minimize costs. It’s only recently that the default consensus for RL problems changed from “this is an inherently creative endeavor and needs human input” to “you can probably do it if you throw enough GPUs at the problem”.

gwern 5y ago
I'd say SC2 is still in the "inherently creative" part. Even with the very interesting AlphaStar League partially solving the self-play diversity and exploration problem, it's not at all obvious that just sinking in another 10-100x TPU-time into AlphaStar would patch up all of the errors and poor strategy AS exhibited - in fact, I'd bet heavily against it, since spending a large-but-still-feasible amount of compute doesn't appear to have fixed OA5 or AlphaGo's similar-looking problems, but AlphaZero was required. Now, applying MuZero might be powerful to solve SC2 with much more compute, but that is still not something many DRL researchers would give you large odds on, and you still probably need to come up with some ways to abstract it and condense the game tree into something tractable for planning over, especially with the partial information making modeling the future much harder. (VQ-VAE and related generative modeling approaches are promising in this regard... but still no slam dunk. If DM announced tomorrow that MuZero+VQ-VAE had defeated AS and a bunch of human pros, people would be extremely shocked, even as they scrambled to tweet about how they saw it coming all along & also are very offended by the waste of CO2 and how unreplicable it is on a grad student budget.)

by NavinF 5y ago

gwern 5y ago

I'd say SC2 is still in the "inherently creative" part. Even with the very interesting AlphaStar League partially solving the self-play diversity and exploration problem, it's not at all obvious that just sinking in another 10-100x TPU-time into AlphaStar would patch up all of the errors and poor strategy AS exhibited - in fact, I'd bet heavily against it, since spending a large-but-still-feasible amount of compute doesn't appear to have fixed OA5 or AlphaGo's similar-looking problems, but AlphaZero was required. Now, applying MuZero might be powerful to solve SC2 with much more compute, but that is still not something many DRL researchers would give you large odds on, and you still probably need to come up with some ways to abstract it and condense the game tree into something tractable for planning over, especially with the partial information making modeling the future much harder. (VQ-VAE and related generative modeling approaches are promising in this regard... but still no slam dunk. If DM announced tomorrow that MuZero+VQ-VAE had defeated AS and a bunch of human pros, people would be extremely shocked, even as they scrambled to tweet about how they saw it coming all along & also are very offended by the waste of CO2 and how unreplicable it is on a grad student budget.)