7 ms·

I have the exact same problem with this article that I had with the previous one - the author fails to provide any data on the frequency of illegal moves. Thus

I have the exact same problem with this article that I had with the previous one - the author fails to provide any data on the frequency of illegal moves. Thus it's impossible to draw any meaningful conclusions. It would be similar to if I claimed that an LLM is an expert doctor, but in my data I've filtered out all of the times it gave incorrect medical advice.

falcor84 2y ago
I world argue that it's more akin to filtering out the chit-chat with the patient, where the doctor explained things in an imprecise manner, keeping only the formal and valid medical notation
- caddemon 2y ago
  There is no legitimate reason to make an illegal move in chess though? There are reasons why a good doctor might intentionally explain things imprecisely to a patient.
  - hnthrowaway6543 2y ago
    > There is no legitimate reason to make an illegal move in chess though? If you make an illegal move and the opponent doesn't notice it, you gain a significant advantage. LLMs just have David Sirlin's "Playing to Win" as part of their training data.
    - fluoridation 2y ago
      You raise an interesting point. If the filtered out illegal moves were disadvantageous, it could be that if the model had been allowed to make any moves it wanted it would have played to a much worse level than it did.
- ses1984 2y ago
  It’s like the doctor saying, “you have cancer? Oh you don’t? Just kidding. Parkinson’s. Oh it’s not that either? How about common cold?”
  - falcor84 2y ago
    Big the difference is that valid bad moves (equivalents of "cancer") were included in the analysis, it's only invalid ones (like "your body is kinda outgrowing itself") that were excluded from the analysis
    - ses1984 2y ago
      What makes a chess move invalid is the state of the board. I don’t think moves like “pick up the pawn and throw it across the room” were considered.
      - toast0 2y ago
        That's a valid move in Monopoly though. Although it's much prefered to pick up the table and throw it.
sigmar 2y ago
Don't think that analogy works unless you could write a script that automatically removes incorrect medical advice, because then you would indeed have an LLM-with-a-script that was an expert doctor (which you can do for illegal chess move, but obviously not for evaluating medical advice)
- kcbanner 2y ago
  It would be possible to employ an expert doctor, instead of writing a script.
  - ben_w 2y ago
    Which is cheaper: 1. having a human expert creating every answer or 2. having an expert check 10 answers each of which have a 90% chance of being right and then manually redoing the one which was wrong Now add a complications that: • option 1 also isn't 100% correct • nobody knows which things in option 2 are correlated or not and if those are or aren't correlated with human errors so we might be systematically unable to even recognise the errors • even if we could, humans not only get lazy without practice but also get bored if the work is too easy, so a short-term study in efficiency changes doesn't tell you things like "after 2 years you get mass resignations by the competent doctors, while the incompetent just say 'LGTM' to all the AI answers"
- wavemode 2y ago
  You can write scripts that correct bad math, too. In fact most of the time ChatGPT will just call out to a calculator function. This is a smart solution, and very useful for end users! But, still, we should not try to use that to make the claim that LLMs have a good understanding of math.
  - henryfjordan 2y ago
    At what point does "knows how to use a calculator" equate to knowing how to do math? Feels pretty close to me...
famouswaffles 2y ago
3-turbo-instruct makes about 5 or less illegal moves in 8205. It's not here but turbo instruct has been evaled before. https://github.com/adamkarvonen/chess_gpt_eval https://github.com/adamkarvonen/chess_gpt_eval
timjver 2y ago
> It would be similar to if I claimed that an LLM is an expert doctor, but in my data I've filtered out all of the times it gave incorrect medical advice. Computationally it's trivial to detect illegal moves, so it's nothing like filtering out incorrect medical advice.
- wavemode 2y ago
  As I wrote in another comment - you can write scripts that correct bad math, too. But we don't use that to claim that LLMs have a good understanding of math.
  - ben_w 2y ago
    I'd say that's because we don't understand what we mean by "understand". Hardware that accurately performs maths faster than all of humanity combined is so cheap as to be disposable, but I've yet to see anyone claim that a Pi Zero has "understanding" of anything. An LLM can display the viva voce approach that Turing suggested[0], and do it well. Ironically for all those now talking about "stochastic parrots", the passage reads: """… The game (with the player B omitted) is frequently used in practice under the name of viva voce to discover whether some one really understands something or has ‘learnt it parrot fashion’. …" Showing that not much has changed on the philosophy of this topic since it was invented. [0] https://academic.oup.com/mind/article/LIX/236/433/986238 https://academic.oup.com/mind/article/LIX/236/433/986238
    - danparsonson 2y ago
      > I'd say that's because we don't understand what we mean by "understand". I'll have a stab at it. The idea of LLMs 'understanding' maths is that, once having been trained on a set of maths-related material, the LLM will be able to generalise to solve other maths problems that it hasn't encountered before. If an LLM sees all the multiplication tables up to 10x10, and then is correctly able to compute 23x31, we might surmise that it 'understands' multiplication - i.e. that it has built some generalised internal representation of what multiplication is, rather than just memorising all possible answers. Obviously we don't expect generalisation from a Pi Zero without specifically being coded for it, because it's a fixed function piece of hardware. Personally I think this is highly unlikely given that maths and natural language are very different things, and being good at the latter does not bear any relationship to being good at the former (just ask anyone who struggles with maths - plenty of people do!). Not to mention that it's also much easier to test for understanding of maths because there is (usually!) a single correct answer regardless of how convoluted the problem - compared to natural language where imitation and understanding are much more difficult to tell apart.
theptip 2y ago
This is a crazy goal-post move. TFA is proving a positive capability, and rejecting the null hypothesis that “LLMs can’t think they just regurgitate”. Making some illegal moves doesn’t invalidate the demonstrated situational logic intelligence required to play at ELO 1800. (Another angle: a human on Chess.com also has any illegal move they try to make ignored, too.)
- wavemode 2y ago
  It's not a goalpost move. As I've already said, I have the exact same problem with this article as I had with the previous one. My goalposts haven't moved, and my standards haven't changed. Just provide the data! How hard can it be? Why leave it out in the first place?
- photonthug 2y ago
  > Making some illegal moves doesn’t invalidate the demonstrated situational logic intelligence That’s exactly what it does. 1 illegal move in 1 million or 100 million or any other sample size you want to choose means it doesn’t understand chess. People in this thread are really distracted by the medical analogy so I’ll offer another: you’ve got a bridge that allows millions of vehicles to cross, and randomly falls down if you tickle it wrong, maybe a car of rare color. One key aspect of bridges is that they work reliably for any vehicle, and once they fail they don’t work with any vehicle. A bridge that sometimes fails and sometimes doesn’t isn’t a bridge as much as a death trap.
  - famouswaffles 2y ago
    >1 illegal move in 1 million or 100 million or any other sample size you want to choose means it doesn’t understand chess Highly rated chess players make illegal moves. It's rare but it happens. They don't understand chess ?
    - emptyfile 2y ago
      [dead]
    - 2y ago
sixo 2y ago
When I play chess I filter out all kinds of illegal moves. I also filter out bad moves. Human is more like "recursively thinking of ideas and then evaluating them with another part of your model", why not let the LLMs do the same?
- skydhash 2y ago
  Because that’s not what happens? We learn through symbolic meaning and rules which then form a consistent system. Then we can have a goal and continuously evaluate if we’re within the system and transitionning towards that goal. The nice thing is that we don’t have to compute the whole simulation in our brains and can start again from the real world. The more you train, the better your heuristics become and the more your efficiency increases. The internal model of a LLM is statistical text. Which is linear and fixed. Not great other than generating text similar to what was ingested.
  - hackinthebochs 2y ago
    >The internal model of a LLM is statistical text. Which is linear and fixed. Not at all. Like seriously, not in the slightest.
    - skydhash 2y ago
      What does it encode? Images? Scent? Touch? Some higher dimensional qualia?
      - hackinthebochs 2y ago
        Well, a simple description is that they discover circuits that reproduce the training sequence. It turns out that in the process of this, they recover relevant computational structures that generalize the training sequence. The question of how far they generalize is certainly up for debate. But you can't reasonably deny that they generalize to a certain degree. After all, most sentences they are prompted on are brand new and they mostly respond sensibly. Their representation of the input is also not linear. Transformers use self-attention which relies on the softmax function, which is non-linear.
GuB-42 2y ago
> Thus it's impossible to draw any meaningful conclusions. It would be similar to if I claimed that an LLM is an expert doctor, but in my data I've filtered out all of the times it gave incorrect medical advice. Not really, you can try to make illegal moves in chess, and usually, you are given a time penalty and get to try again, so even in a real chess game, illegal moves are "filtered out". And for the "medical expert" analogy, let's say that you compare to systems based on the well being of the patients after they follow the advise. I think it is meaningful even if you filter out advise that is obviously inapplicable, for example because it refers to non-existing body parts.
koolala 2y ago
I want to see graphs of moves the author randomly made too. Maybe even plotting a random-move player on the performance graphs vs. the AIs. It's beginner chess and beginners make moves at random all the time.
- benediktwerner 2y ago
  1750 elo is extremely far from beginner chess. The random mover bot on Lichess has like 700 rating. And the article does show various graphs of the badly playing models which will hardly play worse than random but are clearly far below the good models.
Der_Einzige 2y ago
Correct - Dynamic grammar based/constrained sampling can be used to, at each time-step, force the model to only make valid moves (and you don't have to do it in the prompt like this article does!!!) I have NO idea why no one seems to do this. It's a similar issue with LLM-as-judge evaluations. Often they are begging to be combined with grammar based/constrained/structured sampling. So much good stuff in LLM land isn't used for no good reason! There are several libraries for implementing this easily, outlines, guidance, lm-format-enforcer, and likely many more. You can even do it now with OpenAI! Oobabooga text gen webUI literally has chess as one of it's candidate examples of grammar based sampling!!!
rcxdude 2y ago
I don't think is super relevant. I mean, it would be interesting (especially if there was a meaningful difference in the number of illegal move attempts between the different approaches, doubly so if that didn't correlate with the performance when illegal moves are removed), but I don't think it really affects the conclusions of the article: picking randomly from the set of legal moves makes for a truly terrible chess player, so clearly the LLMs are bringing something to the party such that sampling from their output performs significantly better. Splitting hairs about the capability of the LLM on its own (i.e. insisting on defining attempts at an illegal move as a game loss for the purposes of rating) seems pretty besides the point.
hansvm 2y ago
There's a subtle distinction though; if you're able to filter out illegal behavior, the move quality conditioned on legality can be extremely different from arbitrary move quality (and, as you might see in LLM json parsing, conditioning per-token can be very different from conditioning per-response). If you're arguing that the singularity already happened then your criticism makes perfect sense; these are dumb machines, not useful yet for most applications. If you just want to use the LLM as a tool though, the behavior when you filter out illegal responses (assuming you're able to do so) is the only reasonable metric. Analogizing to a task I care a bit about: Current-gen LLMs are somewhere between piss-poor and moderate at generating recipes. With a bit of prompt engineering most recipes pass my "bar", but they're still often lacking in one or more important characteristics. If you do nothing other than ask it to generate many options and then as a person manually filter to the subset of ideas (around 1/20) which look stellar, it's both very effective at generating good recipes, and they're usually much better than my other sources of stellar recipes (obviously not generally applicable because you have to be able to tell bad recipes from good at a glance for that workflow to make sense). The fact that most of the responses are garbage doesn't really matter; it's still an improvement to how I cook.
teleforce 2y ago
> It would be similar to if I claimed that an LLM is an expert doctor, but in my data I've filtered out all of the times it gave incorrect medical advice Sharp eyes, similarly Andrew Ng and his Stanford University team pulled the same trick by having overfitting training to testing ratio for his famous cardiologist-level paper published in Nature Medicine [1]. The training ratio is more than 99% and testing less than 1% which failed AI validation 101. The paper would not stand in most AI conference but being published in Nature Medicine, one of the highest impact factor journal there is and has many citations for AI in healthcare and medicine. [1] Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network: https://www.nature.com/articles/s41591-018-0268-3 https://www.nature.com/articles/s41591-018-0268-3

falcor84 2y ago

I world argue that it's more akin to filtering out the chit-chat with the patient, where the doctor explained things in an imprecise manner, keeping only the formal and valid medical notation

caddemon 2y ago
There is no legitimate reason to make an illegal move in chess though? There are reasons why a good doctor might intentionally explain things imprecisely to a patient.
- hnthrowaway6543 2y ago
  > There is no legitimate reason to make an illegal move in chess though? If you make an illegal move and the opponent doesn't notice it, you gain a significant advantage. LLMs just have David Sirlin's "Playing to Win" as part of their training data.
  - fluoridation 2y ago
    You raise an interesting point. If the filtered out illegal moves were disadvantageous, it could be that if the model had been allowed to make any moves it wanted it would have played to a much worse level than it did.
ses1984 2y ago
It’s like the doctor saying, “you have cancer? Oh you don’t? Just kidding. Parkinson’s. Oh it’s not that either? How about common cold?”
- falcor84 2y ago
  Big the difference is that valid bad moves (equivalents of "cancer") were included in the analysis, it's only invalid ones (like "your body is kinda outgrowing itself") that were excluded from the analysis
  - ses1984 2y ago
    What makes a chess move invalid is the state of the board. I don’t think moves like “pick up the pawn and throw it across the room” were considered.
    - toast0 2y ago
      That's a valid move in Monopoly though. Although it's much prefered to pick up the table and throw it.

sigmar 2y ago

Don't think that analogy works unless you could write a script that automatically removes incorrect medical advice, because then you would indeed have an LLM-with-a-script that was an expert doctor (which you can do for illegal chess move, but obviously not for evaluating medical advice)

kcbanner 2y ago
It would be possible to employ an expert doctor, instead of writing a script.
- ben_w 2y ago
  Which is cheaper: 1. having a human expert creating every answer or 2. having an expert check 10 answers each of which have a 90% chance of being right and then manually redoing the one which was wrong Now add a complications that: • option 1 also isn't 100% correct • nobody knows which things in option 2 are correlated or not and if those are or aren't correlated with human errors so we might be systematically unable to even recognise the errors • even if we could, humans not only get lazy without practice but also get bored if the work is too easy, so a short-term study in efficiency changes doesn't tell you things like "after 2 years you get mass resignations by the competent doctors, while the incompetent just say 'LGTM' to all the AI answers"
wavemode 2y ago
You can write scripts that correct bad math, too. In fact most of the time ChatGPT will just call out to a calculator function. This is a smart solution, and very useful for end users! But, still, we should not try to use that to make the claim that LLMs have a good understanding of math.
- henryfjordan 2y ago
  At what point does "knows how to use a calculator" equate to knowing how to do math? Feels pretty close to me...

famouswaffles 2y ago

3-turbo-instruct makes about 5 or less illegal moves in 8205. It's not here but turbo instruct has been evaled before. https://github.com/adamkarvonen/chess_gpt_eval https://github.com/adamkarvonen/chess_gpt_eval

timjver 2y ago

> It would be similar to if I claimed that an LLM is an expert doctor, but in my data I've filtered out all of the times it gave incorrect medical advice. Computationally it's trivial to detect illegal moves, so it's nothing like filtering out incorrect medical advice.

wavemode 2y ago
As I wrote in another comment - you can write scripts that correct bad math, too. But we don't use that to claim that LLMs have a good understanding of math.
- ben_w 2y ago
  I'd say that's because we don't understand what we mean by "understand". Hardware that accurately performs maths faster than all of humanity combined is so cheap as to be disposable, but I've yet to see anyone claim that a Pi Zero has "understanding" of anything. An LLM can display the viva voce approach that Turing suggested[0], and do it well. Ironically for all those now talking about "stochastic parrots", the passage reads: """… The game (with the player B omitted) is frequently used in practice under the name of viva voce to discover whether some one really understands something or has ‘learnt it parrot fashion’. …" Showing that not much has changed on the philosophy of this topic since it was invented. [0] https://academic.oup.com/mind/article/LIX/236/433/986238 https://academic.oup.com/mind/article/LIX/236/433/986238
  - danparsonson 2y ago
    > I'd say that's because we don't understand what we mean by "understand". I'll have a stab at it. The idea of LLMs 'understanding' maths is that, once having been trained on a set of maths-related material, the LLM will be able to generalise to solve other maths problems that it hasn't encountered before. If an LLM sees all the multiplication tables up to 10x10, and then is correctly able to compute 23x31, we might surmise that it 'understands' multiplication - i.e. that it has built some generalised internal representation of what multiplication is, rather than just memorising all possible answers. Obviously we don't expect generalisation from a Pi Zero without specifically being coded for it, because it's a fixed function piece of hardware. Personally I think this is highly unlikely given that maths and natural language are very different things, and being good at the latter does not bear any relationship to being good at the former (just ask anyone who struggles with maths - plenty of people do!). Not to mention that it's also much easier to test for understanding of maths because there is (usually!) a single correct answer regardless of how convoluted the problem - compared to natural language where imitation and understanding are much more difficult to tell apart.

theptip 2y ago

This is a crazy goal-post move. TFA is proving a positive capability, and rejecting the null hypothesis that “LLMs can’t think they just regurgitate”. Making some illegal moves doesn’t invalidate the demonstrated situational logic intelligence required to play at ELO 1800. (Another angle: a human on Chess.com also has any illegal move they try to make ignored, too.)

wavemode 2y ago
It's not a goalpost move. As I've already said, I have the exact same problem with this article as I had with the previous one. My goalposts haven't moved, and my standards haven't changed. Just provide the data! How hard can it be? Why leave it out in the first place?
photonthug 2y ago
> Making some illegal moves doesn’t invalidate the demonstrated situational logic intelligence That’s exactly what it does. 1 illegal move in 1 million or 100 million or any other sample size you want to choose means it doesn’t understand chess. People in this thread are really distracted by the medical analogy so I’ll offer another: you’ve got a bridge that allows millions of vehicles to cross, and randomly falls down if you tickle it wrong, maybe a car of rare color. One key aspect of bridges is that they work reliably for any vehicle, and once they fail they don’t work with any vehicle. A bridge that sometimes fails and sometimes doesn’t isn’t a bridge as much as a death trap.
- famouswaffles 2y ago
  >1 illegal move in 1 million or 100 million or any other sample size you want to choose means it doesn’t understand chess Highly rated chess players make illegal moves. It's rare but it happens. They don't understand chess ?
  - emptyfile 2y ago
    [dead]
  - 2y ago

sixo 2y ago

When I play chess I filter out all kinds of illegal moves. I also filter out bad moves. Human is more like "recursively thinking of ideas and then evaluating them with another part of your model", why not let the LLMs do the same?

skydhash 2y ago
Because that’s not what happens? We learn through symbolic meaning and rules which then form a consistent system. Then we can have a goal and continuously evaluate if we’re within the system and transitionning towards that goal. The nice thing is that we don’t have to compute the whole simulation in our brains and can start again from the real world. The more you train, the better your heuristics become and the more your efficiency increases. The internal model of a LLM is statistical text. Which is linear and fixed. Not great other than generating text similar to what was ingested.
- hackinthebochs 2y ago
  >The internal model of a LLM is statistical text. Which is linear and fixed. Not at all. Like seriously, not in the slightest.
  - skydhash 2y ago
    What does it encode? Images? Scent? Touch? Some higher dimensional qualia?
    - hackinthebochs 2y ago
      Well, a simple description is that they discover circuits that reproduce the training sequence. It turns out that in the process of this, they recover relevant computational structures that generalize the training sequence. The question of how far they generalize is certainly up for debate. But you can't reasonably deny that they generalize to a certain degree. After all, most sentences they are prompted on are brand new and they mostly respond sensibly. Their representation of the input is also not linear. Transformers use self-attention which relies on the softmax function, which is non-linear.

GuB-42 2y ago

> Thus it's impossible to draw any meaningful conclusions. It would be similar to if I claimed that an LLM is an expert doctor, but in my data I've filtered out all of the times it gave incorrect medical advice. Not really, you can try to make illegal moves in chess, and usually, you are given a time penalty and get to try again, so even in a real chess game, illegal moves are "filtered out". And for the "medical expert" analogy, let's say that you compare to systems based on the well being of the patients after they follow the advise. I think it is meaningful even if you filter out advise that is obviously inapplicable, for example because it refers to non-existing body parts.

koolala 2y ago

I want to see graphs of moves the author randomly made too. Maybe even plotting a random-move player on the performance graphs vs. the AIs. It's beginner chess and beginners make moves at random all the time.

benediktwerner 2y ago
1750 elo is extremely far from beginner chess. The random mover bot on Lichess has like 700 rating. And the article does show various graphs of the badly playing models which will hardly play worse than random but are clearly far below the good models.

Der_Einzige 2y ago

Correct - Dynamic grammar based/constrained sampling can be used to, at each time-step, force the model to only make valid moves (and you don't have to do it in the prompt like this article does!!!) I have NO idea why no one seems to do this. It's a similar issue with LLM-as-judge evaluations. Often they are begging to be combined with grammar based/constrained/structured sampling. So much good stuff in LLM land isn't used for no good reason! There are several libraries for implementing this easily, outlines, guidance, lm-format-enforcer, and likely many more. You can even do it now with OpenAI! Oobabooga text gen webUI literally has chess as one of it's candidate examples of grammar based sampling!!!

rcxdude 2y ago

I don't think is super relevant. I mean, it would be interesting (especially if there was a meaningful difference in the number of illegal move attempts between the different approaches, doubly so if that didn't correlate with the performance when illegal moves are removed), but I don't think it really affects the conclusions of the article: picking randomly from the set of legal moves makes for a truly terrible chess player, so clearly the LLMs are bringing something to the party such that sampling from their output performs significantly better. Splitting hairs about the capability of the LLM on its own (i.e. insisting on defining attempts at an illegal move as a game loss for the purposes of rating) seems pretty besides the point.

hansvm 2y ago

There's a subtle distinction though; if you're able to filter out illegal behavior, the move quality conditioned on legality can be extremely different from arbitrary move quality (and, as you might see in LLM json parsing, conditioning per-token can be very different from conditioning per-response). If you're arguing that the singularity already happened then your criticism makes perfect sense; these are dumb machines, not useful yet for most applications. If you just want to use the LLM as a tool though, the behavior when you filter out illegal responses (assuming you're able to do so) is the only reasonable metric. Analogizing to a task I care a bit about: Current-gen LLMs are somewhere between piss-poor and moderate at generating recipes. With a bit of prompt engineering most recipes pass my "bar", but they're still often lacking in one or more important characteristics. If you do nothing other than ask it to generate many options and then as a person manually filter to the subset of ideas (around 1/20) which look stellar, it's both very effective at generating good recipes, and they're usually much better than my other sources of stellar recipes (obviously not generally applicable because you have to be able to tell bad recipes from good at a glance for that workflow to make sense). The fact that most of the responses are garbage doesn't really matter; it's still an improvement to how I cook.

teleforce 2y ago

> It would be similar to if I claimed that an LLM is an expert doctor, but in my data I've filtered out all of the times it gave incorrect medical advice Sharp eyes, similarly Andrew Ng and his Stanford University team pulled the same trick by having overfitting training to testing ratio for his famous cardiologist-level paper published in Nature Medicine [1]. The training ratio is more than 99% and testing less than 1% which failed AI validation 101. The paper would not stand in most AI conference but being published in Nature Medicine, one of the highest impact factor journal there is and has many citations for AI in healthcare and medicine. [1] Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network: https://www.nature.com/articles/s41591-018-0268-3 https://www.nature.com/articles/s41591-018-0268-3