3 ms·

From what I understand of large language models, it won't play the best move it can, but the most likely to be played based on its training set. If it was trai

From what I understand of large language models, it won't play the best move it can, but the most likely to be played based on its training set. If it was trained on chess.com, ChatGPT average level would probably be, at best, as good as the most average game it saw. Correct?

throwaway2214 4y ago
you assume they just reproduces the training set, but when they get big enough they start to "understand" things, when the input is really big it actually can never "guess" the next word in the batch, so it has to "learn" concepts
bigstumpy 4y ago
If you specifically asked for the best moves, presumably it would recite moves that correlated with won games, not just common moves.
swatcoder 4y ago
Not quite. That's just not a useful way to think about it because the way it works is much "foggier" than any earnest chess player you compare it against. It doesn't consider rules and legal plays, or strategy, in the way a chess.com player might. It composes each next play (and each analysis) based on a lot of game and non-game factors in a small, moving window of context. It can forget what it's doing, it can lose track of formal rules, it can slip from one pattern into another, it can start reciting chess poetry instead, etc It's more like playing chess with someone suffering from dementia. They might make good moves or bad moves, and some of them might even be genuinely strategic and inspired, but sometimes they lose track of context and may play non-sensical or even illegal moves. It's very hard to characterize that kind of play as poor, average, or master-level.
Dr_Birdbrain 4y ago
If you play moves that are common across low-rating games, the average completions will be low-level moves. If you play moves that are common in high-rated moves, it should play high rated moves. In practice, it mostly plays gibberish that doesn’t follow the rules.

throwaway2214 4y ago

you assume they just reproduces the training set, but when they get big enough they start to "understand" things, when the input is really big it actually can never "guess" the next word in the batch, so it has to "learn" concepts

bigstumpy 4y ago

If you specifically asked for the best moves, presumably it would recite moves that correlated with won games, not just common moves.

swatcoder 4y ago

Not quite. That's just not a useful way to think about it because the way it works is much "foggier" than any earnest chess player you compare it against. It doesn't consider rules and legal plays, or strategy, in the way a chess.com player might. It composes each next play (and each analysis) based on a lot of game and non-game factors in a small, moving window of context. It can forget what it's doing, it can lose track of formal rules, it can slip from one pattern into another, it can start reciting chess poetry instead, etc It's more like playing chess with someone suffering from dementia. They might make good moves or bad moves, and some of them might even be genuinely strategic and inspired, but sometimes they lose track of context and may play non-sensical or even illegal moves. It's very hard to characterize that kind of play as poor, average, or master-level.

Dr_Birdbrain 4y ago

If you play moves that are common across low-rating games, the average completions will be low-level moves. If you play moves that are common in high-rated moves, it should play high rated moves. In practice, it mostly plays gibberish that doesn’t follow the rules.