4 ms·

Except they can play chess better than 90% of humans... Just by luck of course..No understanding.

by golol 3y ago

pxmpxm 3y ago
I've got a disposable gift calculator in front of me that can do division better than every human ever. No one seems to anthropomorphize it, however.
- golol 3y ago
  It has learned an algorithm that plays chess purely by reading chess logs. It can play at different skill levels depending on the surrounding natural language How is that not understanding? By the way, would you say that Stockfish 15 understands how to play chess well?
xrisk 3y ago
I’m pretty sure you can write a simple tree search based program that plays better than 90% of humans, sheerly by crunching numbers quickly. I’m sure you wouldn’t claim that that program “understands” chess in the way a human does. Of course it’s a different question as to how exactly do humans understand chess. I personally don’t believe humans have some sort of “special” intelligence, whatever we do must be computable and therefore doable by a computer. It’s just that todays LLMs don’t seem to be it.
- johnthewise 3y ago
  Sure, that wouldn't be impressive if you wrote tree search for chess. However, if you wrote a program that doesn't do any search and still plays well, that'd be impressive. I don't understand your claim that LLMs playing chess is anything sort of understanding. related: https://parrotchess.com/ https://parrotchess.com/
  - xrisk 3y ago
    How do you know there’s no search involved?
    - Jensson 3y ago
      It does more illegal moves the more you deviate from standard chess openings and positions and becomes unable to play at some point. If it did a tree search based on chess rules that wouldn't happen. If it encoded a lot of chess openings and some tiny ad hoc logic around those then that is exactly what we would expect.
      - golol 3y ago
        This is not true anymore. It has been discovered that gpt-3.5-instruct when prompeted with purely PGN notation.playes at >1700 Elo and makes no/little illegal moves.
eloisant 3y ago
They're only as good as the players that played the games they're trained on. Meaning that for computer programs they're very bad at chess.
- golol 3y ago
  That is extremely impressive for a model that can do literally millions of tasks. It is not trained to play chess. It can write stories, code, recipes, franslate text, create JSON AND play chess. Don't you see how amazing this is compared to everything we had previously? LLMs are not just generalists that can't do anything specific properly. They can play chess better than most humans and I'm pretty sure soon they will play better than any human. And this in a general AI system!
lewhoo 3y ago
> Except they can play chess better than 90% of humans... Just by luck of course..No understanding. I've been thinking about this and I think there is a simple trick to make LLM's play chess well. Any chess position can be represented by a string, like so: rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1 (Forsyth–Edwards Notation) If we think of a chess game as a sentence or a sequence of such strings or tokens, then after feeding tons of those to the model it is reasonable to suspect it will do what it does best, that is predict the next token. There is no spacial awareness or anything beyond that going on I think.
- xrisk 3y ago
  There are far too many possible positions aka tokens for this to be feasible.
  - lewhoo 3y ago
    Same could be said about language but here we are. There are far more possible combinations of words in any given language than there are positions in chess for a typical length of a sentence I suspect.
    - xrisk 3y ago
      Definitely, but treating fen positions as tokens isn’t going to work, you need something smaller, something you can decompose into. What that is I don’t know.
      - lewhoo 3y ago
        Well, every character of the string can be a token. From a transformer's perspective it shouldn't matter if it is Chinese, English or "chess" language.
    - johnthewise 3y ago
      You are right, but evidence of LLMs working well with language is not a sign that they might be memorizing Chess. But rather opposite, infeasability of memorization of chess and yet LLMs still doing well on them might be a sign that LLMs could be doing the same 'thing' with language.
EddTheSDET 3y ago
If they’re trained on a database of hundreds if thousands of top ranked games they’ll beat 90% by following those by rote
- golol 3y ago
  Just to make it clear: the database doesn't have to contain exclusively top rated games. It can be 99% low ELO games, as long as there are suffiiciently many high ELO games (in absolute number), the LLM is large enough, and there is sufficient context for the model to distinguish high ELO from low ELO games, it can learn to play well given fhe right prompting.

Except they can play chess better than 90% of humans... Just by luck of course..No understanding.

by golol 3y ago

Except they can play chess better than 90% of humans... Just by luck of course..No understanding.

I've got a disposable gift calculator in front of me that can do division better than every human ever. No one seems to anthropomorphize it, however.

golol 3y ago
It has learned an algorithm that plays chess purely by reading chess logs. It can play at different skill levels depending on the surrounding natural language How is that not understanding? By the way, would you say that Stockfish 15 understands how to play chess well?

xrisk 3y ago

I’m pretty sure you can write a simple tree search based program that plays better than 90% of humans, sheerly by crunching numbers quickly. I’m sure you wouldn’t claim that that program “understands” chess in the way a human does. Of course it’s a different question as to how exactly do humans understand chess. I personally don’t believe humans have some sort of “special” intelligence, whatever we do must be computable and therefore doable by a computer. It’s just that todays LLMs don’t seem to be it.

johnthewise 3y ago
Sure, that wouldn't be impressive if you wrote tree search for chess. However, if you wrote a program that doesn't do any search and still plays well, that'd be impressive. I don't understand your claim that LLMs playing chess is anything sort of understanding. related: https://parrotchess.com/ https://parrotchess.com/
- xrisk 3y ago
  How do you know there’s no search involved?
  - Jensson 3y ago
    It does more illegal moves the more you deviate from standard chess openings and positions and becomes unable to play at some point. If it did a tree search based on chess rules that wouldn't happen. If it encoded a lot of chess openings and some tiny ad hoc logic around those then that is exactly what we would expect.
    - golol 3y ago
      This is not true anymore. It has been discovered that gpt-3.5-instruct when prompeted with purely PGN notation.playes at >1700 Elo and makes no/little illegal moves.

eloisant 3y ago

They're only as good as the players that played the games they're trained on. Meaning that for computer programs they're very bad at chess.

golol 3y ago
That is extremely impressive for a model that can do literally millions of tasks. It is not trained to play chess. It can write stories, code, recipes, franslate text, create JSON AND play chess. Don't you see how amazing this is compared to everything we had previously? LLMs are not just generalists that can't do anything specific properly. They can play chess better than most humans and I'm pretty sure soon they will play better than any human. And this in a general AI system!

lewhoo 3y ago

> Except they can play chess better than 90% of humans... Just by luck of course..No understanding. I've been thinking about this and I think there is a simple trick to make LLM's play chess well. Any chess position can be represented by a string, like so: rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1 (Forsyth–Edwards Notation) If we think of a chess game as a sentence or a sequence of such strings or tokens, then after feeding tons of those to the model it is reasonable to suspect it will do what it does best, that is predict the next token. There is no spacial awareness or anything beyond that going on I think.

xrisk 3y ago
There are far too many possible positions aka tokens for this to be feasible.
- lewhoo 3y ago
  Same could be said about language but here we are. There are far more possible combinations of words in any given language than there are positions in chess for a typical length of a sentence I suspect.
  - xrisk 3y ago
    Definitely, but treating fen positions as tokens isn’t going to work, you need something smaller, something you can decompose into. What that is I don’t know.
    - lewhoo 3y ago
      Well, every character of the string can be a token. From a transformer's perspective it shouldn't matter if it is Chinese, English or "chess" language.
  - johnthewise 3y ago
    You are right, but evidence of LLMs working well with language is not a sign that they might be memorizing Chess. But rather opposite, infeasability of memorization of chess and yet LLMs still doing well on them might be a sign that LLMs could be doing the same 'thing' with language.

EddTheSDET 3y ago

If they’re trained on a database of hundreds if thousands of top ranked games they’ll beat 90% by following those by rote

golol 3y ago
Just to make it clear: the database doesn't have to contain exclusively top rated games. It can be 99% low ELO games, as long as there are suffiiciently many high ELO games (in absolute number), the LLM is large enough, and there is sufficient context for the model to distinguish high ELO from low ELO games, it can learn to play well given fhe right prompting.