4 ms·

The title of the paper is literally a lie. It's not grandmaster strength, and it is using Stockfish explicitly during the games, because the network they traine

The title of the paper is literally a lie. It's not grandmaster strength, and it is using Stockfish explicitly during the games, because the network they trained fails to finish games properly. Instead of presenting that result, they modify it with Stockfish to get stronger results, so they can claim "grandmaster strength" in the title. They claim it outperforms Alpha Zero's policy network when Alpha is not the state of the art for its architecture anymore and hasn't been for years. Leela is. A fair comparison would use Leela, not Alpha Zero. And of course, as usual, they seem not to provide any source code or weights, so no one can check their research independently.

danielmarkbruce 3y ago
Would you be fine if it were titled: "2895 lichess blitz elo without search (except for edge cases due to the threefold rep rule)" ? The non optimal endgame play wouldn't really be a problem except for the threefold rule, in practice (i think). It would eventually bumble around and win.
- mtlmtlmtlmtl 3y ago
  No. Much more interesting would be to eliminate this calling out to Stockfish entirely. It's just not just a matter of the title, the data in the paper is fundamentally tainted by it. You get less data on how the modell actually plays this way, how strong it really is, and how it could be integrated with search in all positions, which the obvious next step(i.e the model + some search algorithm, not simply calling out to another engine). The fact that they're willing to do that just to get a better headline is pretty telling.
  - danielmarkbruce 3y ago
    That title would call out the edge cases. I don't think it is tainted. In practical terms, those games have been won. Also, the idea of this + search as being the next step is wrong. That was the last step, ie it's been done. No search is the next step.
    - mtlmtlmtlmtl 3y ago
      In chess, games are never won until checkmate or resignation by the opponent. If it fails to get there, it just didn't win the games. It doesn't matter how winning the position was at any other point during the game. That's why it's tainted. The results are changed, and the games themselves are changed. Therefore the comparisons to humans and other engines is changed. And hence, what we learn about the approach presented is diminished and obsfuscated. I think wrong is a strong word here. You can keep making a bigger model, that's one direction to go for sure. But adding search is certainly an interesting one at the very least. And it's definitely the way forward to see if this is an approach that can advance the strength of chess playing programs. I seriously doubt this would ever outperform a search based paradigm on its own. For one thing it gets no advantage from extra time(which is another reason the grandmaster strength claim is dubious btw; it would be outplayed in longer time controls). For another, search might be necessary to make this thing able to train without the help of Stockfish, by playing itself(iterated amplification and distillation, how AlphaZero was trained). For a third, due to the game complexity of chess, it's likely that model size without search would have to scale exponentially to become stronger. And all this brings me back to my previous points about releasing weights and code. If they had done so, I would be trying all of these things out myself right now instead of arguing about it on the internet. But time and time again Deepmind have demonstrated that they won't engage with the wider computer chess community. And so their research, which is no doubt innovative and inspiring, fails to have the impact it could have, because the community has to redo all the work with less money and compute. We saw this with Alpha Zero; it took years and a lot of hard work by volunteers for Leela just to get where Alpha Zero was originally and eventually surpass it. And it will probably take a long time to see state of the art competitive engines based on this new approach in the wild. Because Deepmind don't really care about that beyond making a splash with their initial publication. They're only peripherally interested in chess, which is fine, but as a chess player and computer chess enthusiast it makes me sad that they won't engage with the community in an open way.
      - danielmarkbruce 3y ago
        Adding search isn't an interesting direction. It's been done. AlphaZero has search. Leela has search. You are right though, they are only peripherally interested in chess.

danielmarkbruce 3y ago

Would you be fine if it were titled: "2895 lichess blitz elo without search (except for edge cases due to the threefold rep rule)" ? The non optimal endgame play wouldn't really be a problem except for the threefold rule, in practice (i think). It would eventually bumble around and win.

mtlmtlmtlmtl 3y ago
No. Much more interesting would be to eliminate this calling out to Stockfish entirely. It's just not just a matter of the title, the data in the paper is fundamentally tainted by it. You get less data on how the modell actually plays this way, how strong it really is, and how it could be integrated with search in all positions, which the obvious next step(i.e the model + some search algorithm, not simply calling out to another engine). The fact that they're willing to do that just to get a better headline is pretty telling.
- danielmarkbruce 3y ago
  That title would call out the edge cases. I don't think it is tainted. In practical terms, those games have been won. Also, the idea of this + search as being the next step is wrong. That was the last step, ie it's been done. No search is the next step.
  - mtlmtlmtlmtl 3y ago
    In chess, games are never won until checkmate or resignation by the opponent. If it fails to get there, it just didn't win the games. It doesn't matter how winning the position was at any other point during the game. That's why it's tainted. The results are changed, and the games themselves are changed. Therefore the comparisons to humans and other engines is changed. And hence, what we learn about the approach presented is diminished and obsfuscated. I think wrong is a strong word here. You can keep making a bigger model, that's one direction to go for sure. But adding search is certainly an interesting one at the very least. And it's definitely the way forward to see if this is an approach that can advance the strength of chess playing programs. I seriously doubt this would ever outperform a search based paradigm on its own. For one thing it gets no advantage from extra time(which is another reason the grandmaster strength claim is dubious btw; it would be outplayed in longer time controls). For another, search might be necessary to make this thing able to train without the help of Stockfish, by playing itself(iterated amplification and distillation, how AlphaZero was trained). For a third, due to the game complexity of chess, it's likely that model size without search would have to scale exponentially to become stronger. And all this brings me back to my previous points about releasing weights and code. If they had done so, I would be trying all of these things out myself right now instead of arguing about it on the internet. But time and time again Deepmind have demonstrated that they won't engage with the wider computer chess community. And so their research, which is no doubt innovative and inspiring, fails to have the impact it could have, because the community has to redo all the work with less money and compute. We saw this with Alpha Zero; it took years and a lot of hard work by volunteers for Leela just to get where Alpha Zero was originally and eventually surpass it. And it will probably take a long time to see state of the art competitive engines based on this new approach in the wild. Because Deepmind don't really care about that beyond making a splash with their initial publication. They're only peripherally interested in chess, which is fine, but as a chess player and computer chess enthusiast it makes me sad that they won't engage with the community in an open way.
    - danielmarkbruce 3y ago
      Adding search isn't an interesting direction. It's been done. AlphaZero has search. Leela has search. You are right though, they are only peripherally interested in chess.