3 ms·

This may look low: ELO for mediocre players is 1500. But if it is obeying the rules of the game, then this is big. This is a signal that if it learns some exper

This may look low: ELO for mediocre players is 1500. But if it is obeying the rules of the game, then this is big. This is a signal that if it learns some expertise, like discovering how to use or create better search algorithms (like MCTS and heuristics to evaluate a state) and improve by itself (somewhat like alphazero did), then it may eventually reach superhuman level. It may then reach superhuman level in any task simpler than chess which is enough to destroy many human jobs. EDIT: From the article: "With this prompt ChatGPT almost always plays fully legal games." Relax: we're still far from that.

gfd 4y ago
According to https://chess.stackexchange.com/questions/2550/what-are-the-average-elo-and-uscf-ratings https://chess.stackexchange.com/questions/2550/what-are-the-... median rating is 1148 (252,989 Players). So it's beating half of humanity at a mind sport and it wasn't even specifically trained for it.
- jabloczko 4y ago
  There's a difference between chess.com and USCF ratings.
- sapiogram 4y ago
  That's USCF ratings, chess.com ratings are massively inflated in comparison.
- nottathrowaway3 4y ago
  The median chess player is usually described as mediocre (if you ask chess players). They suck as badly as the median clarinet player in your high school band/orchestra.
LeanderK 4y ago
I think current LLM architectures are limiting the strategies it learned. MCTS requires recursion but GPT is always executing a fixed number of steps. Allowing language models more flexibility by a variable number of steps, for example through recursion of the model, would eliminate this hard bound but they are harder to design and train. We have just been able to train GPT-sized models.
- sebzim4500 4y ago
  I'm sure the MuZero chess policy network would reach much higher than 1400, and that has no notion of recursion either. (And also wasn't taught the rules explicitly)
  - LeanderK 4y ago
    Yes, I wouldn't say it's impossible, but that's just a hard limit from the architecture. MCTS would have to terminate after a few iterations. And the number of steps is not that big, just massively parallel.

gfd 4y ago

According to https://chess.stackexchange.com/questions/2550/what-are-the-average-elo-and-uscf-ratings https://chess.stackexchange.com/questions/2550/what-are-the-... median rating is 1148 (252,989 Players). So it's beating half of humanity at a mind sport and it wasn't even specifically trained for it.

jabloczko 4y ago
There's a difference between chess.com and USCF ratings.
sapiogram 4y ago
That's USCF ratings, chess.com ratings are massively inflated in comparison.
nottathrowaway3 4y ago
The median chess player is usually described as mediocre (if you ask chess players). They suck as badly as the median clarinet player in your high school band/orchestra.

LeanderK 4y ago

I think current LLM architectures are limiting the strategies it learned. MCTS requires recursion but GPT is always executing a fixed number of steps. Allowing language models more flexibility by a variable number of steps, for example through recursion of the model, would eliminate this hard bound but they are harder to design and train. We have just been able to train GPT-sized models.

sebzim4500 4y ago
I'm sure the MuZero chess policy network would reach much higher than 1400, and that has no notion of recursion either. (And also wasn't taught the rules explicitly)
- LeanderK 4y ago
  Yes, I wouldn't say it's impossible, but that's just a hard limit from the architecture. MCTS would have to terminate after a few iterations. And the number of steps is not that big, just massively parallel.