4 ms·

He's also wrong about a lot. For all his insistence on accuracy, he himself is misleading or ignorant. Like his slam of the Dartmouth project - even if you knew

by gwern 7y ago

He's also wrong about a lot. For all his insistence on accuracy, he himself is misleading or ignorant. Like his slam of the Dartmouth project - even if you knew nothing about it, all you have to do is click through to see the claims of 'solving vision' are sheer projection and urban legend. And he's happy to make up claims out of whole cloth: for example, when he says "AlphaGo works fine on a 19x19 board, but would need to be retrained to play on a rectangular board; the lack of transfer is telling.", for which he provides precisely zero evidence, he ignores the fact that AG training works fine on a mixture of board sizes and such progressive growing/curriculum training in fact seems to accelerate training (https://arxiv.org/abs/1902.10565 https://arxiv.org/abs/1902.10565) and that rectangular convolutions are a thing that exist, and there would be plenty of transfer if anyone tried. If no one has tried that exact thing, it's because it would be pointless and it's obvious to everyone not named 'Marcus' that it'd work fine, being rectangular doesn't mystically stop it from working anymore than using a 13x13 rather than 19x19 board makes AG-style training stop working. (This is not the first time Marcus has claimed that something didn't work; I first realized that he doesn't actually keep up with the AI literature when I pointed out to him on Twitter that plenty of knowledge graphs were in use combined with DL, and Google Search was the biggest example of this, and he had no idea what I was talking about.) No matter what DL or DRL does, or how little his own preferred paradigm does, Marcus will never ever admit anything. AlphaGo beats humans? Well, it just copied humans. AlphaZero learns from scratch? Well, he wrote a whole paper explaining how akshully it still copies humans because the tree search encodes the rules. MuZero throws out even the tree search's knowledge of rules? Crickets and essays about 'misinformation'.

YeGoblynQueenne 7y ago
Gary Marcus' tone amounts to base trolling and it's a shame that's how he chooses to carry out his criticism. I understand your frustration. Regarding the lack of transfer, yes, AlphaGo, AlphaZero and most of their variants have boards of fixed size and shape hard-coded in their architecture (as they have the types of piece moves-hard coded) and need architectural modifications and re-training before they can play on different boards or with different pieces (e.g. AlphaGo can't play Chess and Shoggi unmodified). The KataGo paper (the paper you linked) is one exception to this. Personally, I don't know others. Anyway general game-playing is a hard task and nobody claims it's solved by AlphaGo. Regarding KataGo its main contribution is a significant reduction to the cost of training an AlpahGo variant while maintaining a competitive performance. This is very promising- after DeepBlue, creating a chess engine became cheaper and cheaper until they could run on a smartphone. We are far from that with Go computer players. However, in the KataGo paper, major gains are claimed to come from a) game-playing specific or MCTS-specific improvements (playout cap randomisation, forced playouts and policy target pruning) or architecture-specific improvements (global pooling) or, b) domain-specific improvements (auxiliary ownership and score targets). Finally, KataGo has a few game-specific features (liberties, pass-alive regions and ladder features). The KataGo paper itself says it very clearly. I quote, snipping for brevity: Second, our work serves as a case study that there is still a significant efficiency gap between AlphaZero's methods and what is possible from self-play. We find nontrivial further gains from some domain-specific methods (...) We also find that a set of standard game-specific input features still significantly accelerates learning, showing that AlphaZero does not yet obsolete even simple additional tuning. Finally, "it would obviously work so nobody tried" would make sense if it wasn't for the extremely competitive nature of machine learning research where every novel result is presented as a big breakthrough. Also, if something is obvious but never seems to make it to publication the chances are someone has tried and it didn't work as expected so they shelved the paper. We all know what happens to negative results in machine learning.
- gwern 7y ago
  > The KataGo paper (the paper you linked) is one exception to this. And my point is that KataGo shows that if you make the relatively minor architectural changes necessary to do this at all, it works just fine. None of the other tweaks it makes, useful as they may be, have anything to do with fixing transfer learning, because there's nothing to fix. It's a pretty absurd claim to claim that a CNN which works fine on 19x19 will suddenly collapse and show no transfer on, say, 17x17, and KataGo demonstrates that this does not happen. > Also, if something is obvious but never seems to make it to publication the chances are someone has tried and it didn't work as expected so they shelved the paper. 'What if Go but rectangular boards' is pretty dumb when you have chess and shogi and other domains showing that A0 works, so I feel confident that no one like DM seriously tried and simply buried their failures. (Publication bias requires there to be a literature that can be differentially published, and competition presumes the existence of >0 entities competing; there is no active field of 'rectangular Go'.)
YeGoblynQueenne 7y ago
Regarding MuZero- I confess to not have read the paper very carefully, but I am confused by its claim that the new system achieves superhuman performance without knowing the game rules. Specifically, MuZero uses MCTS and MCTS needs to have at the very least a move generator in order to produce actions that can then be evaluated for their results. The trained MuZero model learns the transition function and evaluation function but I don't see in the paper where it learns what actions are legal in the domain. And I don't understand how any architecture could model the possible moves in a game without observing examples of external play (i.e. not self-play). MuZero reuses the AlphaZero architecture so most likely the moves of the pieces for Chess, Shoggi and Go are hard-coded in the architecture, as they are in AlphaZero. There's also probably some similar hard-coding of Atari actions, which I'm probably missing in the paper.
- yazr 7y ago
  > but I am confused by its claim ... without knowing the game rules. > probably some similar hard-coding of Atari actions Nope, no hard coding. Consider trying to MCTS on an Atari game. You have to "learn to predict" the <next frame, action> pairs. Initially this guess is very bad, but eventually your predictions are good enough that rolling out a tree of predictions improves your action selection For Go, and chess, we twist our self into NOT using the game rules in the simulator e.g. for each move, just indicate if GAME LOSS WIN Whether this paper worthy of a new Nature hype cycle is a separate debate
  - YeGoblynQueenne 7y ago
    >> You have to "learn to predict" the <next frame, action> pairs. But where do the actions come from? For example, if I play chess, I could pick up a piece and throw it at my opponent's head. Similarly, if I play Atari I could chuck the controller at the monitor. These are actions I can perform that are available to me because of my basic human anatomy and because of the laws of physics (I can grab and throw and a thrown object flies through the air untl it hits a target or gravity wins). In the case of MuZero, what actions can the system perform and where do they come from? I don't see where that is described in the paper. >> For Go, and chess, we twist our self into NOT using the game rules in the simulator e.g. for each move, just indicate if GAME LOSS WIN Similarly - what determines "each move"? EDIT: I can see in the MuZero paper that "Final outcomes {lose,draw,win} in board games are treated as rewards $u_t \in {-1,0,+1}$ occurring at the final step of the episode" but I also can't see where these come from, what tells the model that a loss, draw or win has occurred at the end of an episode. I mean, if you're telling the model what actions can be performed and what end-states values are, then what game rules are you _not_ giving to the system?

by gwern 7y ago

YeGoblynQueenne 7y ago

Gary Marcus' tone amounts to base trolling and it's a shame that's how he chooses to carry out his criticism. I understand your frustration. Regarding the lack of transfer, yes, AlphaGo, AlphaZero and most of their variants have boards of fixed size and shape hard-coded in their architecture (as they have the types of piece moves-hard coded) and need architectural modifications and re-training before they can play on different boards or with different pieces (e.g. AlphaGo can't play Chess and Shoggi unmodified). The KataGo paper (the paper you linked) is one exception to this. Personally, I don't know others. Anyway general game-playing is a hard task and nobody claims it's solved by AlphaGo. Regarding KataGo its main contribution is a significant reduction to the cost of training an AlpahGo variant while maintaining a competitive performance. This is very promising- after DeepBlue, creating a chess engine became cheaper and cheaper until they could run on a smartphone. We are far from that with Go computer players. However, in the KataGo paper, major gains are claimed to come from a) game-playing specific or MCTS-specific improvements (playout cap randomisation, forced playouts and policy target pruning) or architecture-specific improvements (global pooling) or, b) domain-specific improvements (auxiliary ownership and score targets). Finally, KataGo has a few game-specific features (liberties, pass-alive regions and ladder features). The KataGo paper itself says it very clearly. I quote, snipping for brevity: Second, our work serves as a case study that there is still a significant efficiency gap between AlphaZero's methods and what is possible from self-play. We find nontrivial further gains from some domain-specific methods (...) We also find that a set of standard game-specific input features still significantly accelerates learning, showing that AlphaZero does not yet obsolete even simple additional tuning. Finally, "it would obviously work so nobody tried" would make sense if it wasn't for the extremely competitive nature of machine learning research where every novel result is presented as a big breakthrough. Also, if something is obvious but never seems to make it to publication the chances are someone has tried and it didn't work as expected so they shelved the paper. We all know what happens to negative results in machine learning.

gwern 7y ago
> The KataGo paper (the paper you linked) is one exception to this. And my point is that KataGo shows that if you make the relatively minor architectural changes necessary to do this at all, it works just fine. None of the other tweaks it makes, useful as they may be, have anything to do with fixing transfer learning, because there's nothing to fix. It's a pretty absurd claim to claim that a CNN which works fine on 19x19 will suddenly collapse and show no transfer on, say, 17x17, and KataGo demonstrates that this does not happen. > Also, if something is obvious but never seems to make it to publication the chances are someone has tried and it didn't work as expected so they shelved the paper. 'What if Go but rectangular boards' is pretty dumb when you have chess and shogi and other domains showing that A0 works, so I feel confident that no one like DM seriously tried and simply buried their failures. (Publication bias requires there to be a literature that can be differentially published, and competition presumes the existence of >0 entities competing; there is no active field of 'rectangular Go'.)

YeGoblynQueenne 7y ago

Regarding MuZero- I confess to not have read the paper very carefully, but I am confused by its claim that the new system achieves superhuman performance without knowing the game rules. Specifically, MuZero uses MCTS and MCTS needs to have at the very least a move generator in order to produce actions that can then be evaluated for their results. The trained MuZero model learns the transition function and evaluation function but I don't see in the paper where it learns what actions are legal in the domain. And I don't understand how any architecture could model the possible moves in a game without observing examples of external play (i.e. not self-play). MuZero reuses the AlphaZero architecture so most likely the moves of the pieces for Chess, Shoggi and Go are hard-coded in the architecture, as they are in AlphaZero. There's also probably some similar hard-coding of Atari actions, which I'm probably missing in the paper.

yazr 7y ago
> but I am confused by its claim ... without knowing the game rules. > probably some similar hard-coding of Atari actions Nope, no hard coding. Consider trying to MCTS on an Atari game. You have to "learn to predict" the <next frame, action> pairs. Initially this guess is very bad, but eventually your predictions are good enough that rolling out a tree of predictions improves your action selection For Go, and chess, we twist our self into NOT using the game rules in the simulator e.g. for each move, just indicate if GAME LOSS WIN Whether this paper worthy of a new Nature hype cycle is a separate debate
- YeGoblynQueenne 7y ago
  >> You have to "learn to predict" the <next frame, action> pairs. But where do the actions come from? For example, if I play chess, I could pick up a piece and throw it at my opponent's head. Similarly, if I play Atari I could chuck the controller at the monitor. These are actions I can perform that are available to me because of my basic human anatomy and because of the laws of physics (I can grab and throw and a thrown object flies through the air untl it hits a target or gravity wins). In the case of MuZero, what actions can the system perform and where do they come from? I don't see where that is described in the paper. >> For Go, and chess, we twist our self into NOT using the game rules in the simulator e.g. for each move, just indicate if GAME LOSS WIN Similarly - what determines "each move"? EDIT: I can see in the MuZero paper that "Final outcomes {lose,draw,win} in board games are treated as rewards $u_t \in {-1,0,+1}$ occurring at the final step of the episode" but I also can't see where these come from, what tells the model that a loss, draw or win has occurred at the end of an episode. I mean, if you're telling the model what actions can be performed and what end-states values are, then what game rules are you _not_ giving to the system?