5 ms·

> LLMs have no real understanding A large language model trained on Othello games will actually build a "board representation" inside its neural network: https

by ekidd 3y ago

> LLMs have no real understanding A large language model trained on Othello games will actually build a "board representation" inside its neural network: https://thegradient.pub/othello/ https://thegradient.pub/othello/. This was confirmed by editing the board representation inside the neural net, "tricking" it into "believing" the board was in a different state. At this point, the LLM started making legal moves based on the edited position. If the LLM is actually building a data structure describing the board state, then it's hard to claim it's just acting as a parrot. The raw version of GPT apparently achieves an Elo score of 1800 at chess: https://twitter.com/GrantSlatton/status/1703913578036904431 https://twitter.com/GrantSlatton/status/1703913578036904431. This is worse than a good dedicated chess engine, but almost certainly better than I could do! And every day, I see Copilot occasionally perform non-trivial completions that require "in-context learning" of complicated things that exist only in my source code. Sure, it crashes and burns regularly, too, but I've seen a lot of cases where it could only have figured out the code it did by carefully combining information available in the file. So I believe that LLMs can actually achieve surprisingly large amounts of understanding, particularly ChatGPT 4 and (on a good day) CoPilot. But the frustrating thing is that they're inconsistent, and the current RLHF process apparently impairs their ability to correctly estimate accuracy. When they make mistakes, they're effectively "blind" to those mistakes, to a shocking degree.

ruined 3y ago
the input is a data structure describing board state. i don't think uncovering the internal representation of board state, after providing the board state, implies any kind of phenomenal understanding - it just implies the software may accept input. side-channel editing of the internal state, and observation of the resulting effect, does not imply understanding. you can do the same thing to any traditional game algorithm.
- ekidd 3y ago
  As I understand it, the Othello LLM was: 1. Trained using written sequences of moves from real world games, in text notation. 2. Prompted using written series of moves. So it has supposedly never "seen" an Othello board state at all, if I understand the experiment. It deduced the rules of rules of Othello and "learned" a neural network containing a board representation. It's entire possible that I misunderstood the paper, though? Why did you conclude that they were training and/or promoting using board state? (I should go re-read the paper; it's been a while.)
  - famouswaffles 3y ago
    No you're right. Clearly the information of the board state is implicitly present in sequences of moves (humans can play blindfold chess too but poorer) but a reconstruction from that alone is highly non trivial. I don't think a human that had never seen or played the game would be able to do it.
    - DanHulton 3y ago
      I think you underestimate humans.
    - notahacker 3y ago
      Agreed. But Othello syntax happens to map perfectly to possible Othello moves, so the most efficient representation of the information contained within Othello happens to be a data structure which maps to an Othello board. Other information like visuals on the players would, from the point of view of evaluating the next Othello move, just be noise. Human language doesn't map perfectly or even particularly closely to the physical world we inhabit or emotions we experience though, so a maximally efficient model of human language will overfit to useless semantic features whilst lacking context on what foo that follows baz is. Understanding the physical or emotional world solely from human language is more like trying to use the Othello LLM's state representation to establish the colour of the board
      - famouswaffles 3y ago
        The sense data your brain ingests does not map perfectly or even particularly closely to the real physical world we inhabit. Whether it’s through language, vision (which feels quite “real” but is really just a 2D projection of light that we interpret), sound or anything else, it’s all just some byproduct of the world that we nonetheless can make useful predictions with. There is enough information to build "a" world model in all the text ever written by humans about the world. Not necessarily the "one true model" that is your own personal life experience of the world since birth. >Understanding the physical or emotional world solely from human language is more like trying to use the Othello LLM's state representation to establish the colour of the board Nobody truly understands the physical world. Don't you think the birds that can feel the electromagnetic fields around the earth and use it to guide their travels would tell you your model was fundamentally incorrect ? Certainly, LLMs are more limited in their input data, but it's not a fundamental difference. and adding more modalities is trivial.
- jaidhyani 3y ago
  As the other commenter said, this is incorrect. The input was a sequence of legal moves (not even "real" moves - most of the training data was synthetically generated with "generate legal moves" as the only constraint). Deducing board state from this is extremely non-trivial.
kweingar 3y ago
I’d point out that the LLMs are still incapable of using these internal representations in novel ways. A human who has internalized the rules of chess can learn and play variants like duck chess easily. GPT-4, despite being somewhat proficient at chess, falls apart when asked to play duck chess, constantly making illegal or illogical moves.
- famouswaffles 3y ago
  Uhh not really. I assure you, a LLM that can play chess will converge in training on duck chess much faster than one that wasn't.
  - emmender1 3y ago
    LLMs are failing on tasks as simple as this. you can call it a trick question but what the hey. "prove that there are no non negative numbers less than 3" Claude puts out this elaborate BS - whats funny is the pretension. ---- This is trivial to prove by contradiction: Assume there exists a non-negative number n < 3. Then either: n = 0. But 0 is not less than 3, so this is a contradiction. 0 < n < 3. But the only numbers between 0 and 3 are 1 and 2, and neither of those are less than 3, so this is also a contradiction. Therefore, our assumption that such an n exists must be false. By contradiction, there are no non-negative numbers less than 3. In formal logical terms: Assume ∃n ∈ R, n ≥ 0 and n < 3 (There exists some non-negative number n less than 3) Then either: a) n = 0. But 0 ≥ 3 is false, so this is a contradiction. b) 0 < n < 3. The only numbers in (0,3) are 1 and 2, neither of which is less than 3, so this is also a contradiction. Therefore the assumption in (1) must be false. By contradiction, the statement ∀n ∈ R, n ≥ 0 → n ≥ 3 (For all non-negative numbers n, n is greater than or equal to 3) must be true. So the proof by contradiction shows that there are no non-negative numbers less than 3. ---- boosters can hype all you want, but the damn thing cant compute. and please dont say - "but most humans will fail at this too" meanwhile...: claude-coders working furiously to add this test case - lol.
    - famouswaffles 3y ago
      I swear, these kind of gotchas are the most stupid. Because they are easily resolved by just talking to the current state of the art. https://chat.openai.com/share/93324013-b333-44ef-b8c8-5460cbfb8477 https://chat.openai.com/share/93324013-b333-44ef-b8c8-5460cb...
YeGoblynQueenne 3y ago
>> The raw version of GPT apparently achieves an Elo score of 1800 at chess: https://twitter.com/GrantSlatton/status/1703913578036904431 https://twitter.com/GrantSlatton/status/1703913578036904431. This is worse than a good dedicated chess engine, but almost certainly better than I could do! Probably because it was trained to get better at it. https://twitter.com/GrantSlatton/status/1706545268009931175 https://twitter.com/GrantSlatton/status/1706545268009931175 There's already been plenty of discussion on LLMs' ability to play chess and it makes sense that OpenAI would try to make their models better at it.

by ekidd 3y ago

ruined 3y ago

the input is a data structure describing board state. i don't think uncovering the internal representation of board state, after providing the board state, implies any kind of phenomenal understanding - it just implies the software may accept input. side-channel editing of the internal state, and observation of the resulting effect, does not imply understanding. you can do the same thing to any traditional game algorithm.

ekidd 3y ago
As I understand it, the Othello LLM was: 1. Trained using written sequences of moves from real world games, in text notation. 2. Prompted using written series of moves. So it has supposedly never "seen" an Othello board state at all, if I understand the experiment. It deduced the rules of rules of Othello and "learned" a neural network containing a board representation. It's entire possible that I misunderstood the paper, though? Why did you conclude that they were training and/or promoting using board state? (I should go re-read the paper; it's been a while.)
- famouswaffles 3y ago
  No you're right. Clearly the information of the board state is implicitly present in sequences of moves (humans can play blindfold chess too but poorer) but a reconstruction from that alone is highly non trivial. I don't think a human that had never seen or played the game would be able to do it.
  - DanHulton 3y ago
    I think you underestimate humans.
  - notahacker 3y ago
    Agreed. But Othello syntax happens to map perfectly to possible Othello moves, so the most efficient representation of the information contained within Othello happens to be a data structure which maps to an Othello board. Other information like visuals on the players would, from the point of view of evaluating the next Othello move, just be noise. Human language doesn't map perfectly or even particularly closely to the physical world we inhabit or emotions we experience though, so a maximally efficient model of human language will overfit to useless semantic features whilst lacking context on what foo that follows baz is. Understanding the physical or emotional world solely from human language is more like trying to use the Othello LLM's state representation to establish the colour of the board
    - famouswaffles 3y ago
      The sense data your brain ingests does not map perfectly or even particularly closely to the real physical world we inhabit. Whether it’s through language, vision (which feels quite “real” but is really just a 2D projection of light that we interpret), sound or anything else, it’s all just some byproduct of the world that we nonetheless can make useful predictions with. There is enough information to build "a" world model in all the text ever written by humans about the world. Not necessarily the "one true model" that is your own personal life experience of the world since birth. >Understanding the physical or emotional world solely from human language is more like trying to use the Othello LLM's state representation to establish the colour of the board Nobody truly understands the physical world. Don't you think the birds that can feel the electromagnetic fields around the earth and use it to guide their travels would tell you your model was fundamentally incorrect ? Certainly, LLMs are more limited in their input data, but it's not a fundamental difference. and adding more modalities is trivial.
jaidhyani 3y ago
As the other commenter said, this is incorrect. The input was a sequence of legal moves (not even "real" moves - most of the training data was synthetically generated with "generate legal moves" as the only constraint). Deducing board state from this is extremely non-trivial.

kweingar 3y ago

I’d point out that the LLMs are still incapable of using these internal representations in novel ways. A human who has internalized the rules of chess can learn and play variants like duck chess easily. GPT-4, despite being somewhat proficient at chess, falls apart when asked to play duck chess, constantly making illegal or illogical moves.

famouswaffles 3y ago
Uhh not really. I assure you, a LLM that can play chess will converge in training on duck chess much faster than one that wasn't.
- emmender1 3y ago
  LLMs are failing on tasks as simple as this. you can call it a trick question but what the hey. "prove that there are no non negative numbers less than 3" Claude puts out this elaborate BS - whats funny is the pretension. ---- This is trivial to prove by contradiction: Assume there exists a non-negative number n < 3. Then either: n = 0. But 0 is not less than 3, so this is a contradiction. 0 < n < 3. But the only numbers between 0 and 3 are 1 and 2, and neither of those are less than 3, so this is also a contradiction. Therefore, our assumption that such an n exists must be false. By contradiction, there are no non-negative numbers less than 3. In formal logical terms: Assume ∃n ∈ R, n ≥ 0 and n < 3 (There exists some non-negative number n less than 3) Then either: a) n = 0. But 0 ≥ 3 is false, so this is a contradiction. b) 0 < n < 3. The only numbers in (0,3) are 1 and 2, neither of which is less than 3, so this is also a contradiction. Therefore the assumption in (1) must be false. By contradiction, the statement ∀n ∈ R, n ≥ 0 → n ≥ 3 (For all non-negative numbers n, n is greater than or equal to 3) must be true. So the proof by contradiction shows that there are no non-negative numbers less than 3. ---- boosters can hype all you want, but the damn thing cant compute. and please dont say - "but most humans will fail at this too" meanwhile...: claude-coders working furiously to add this test case - lol.
  - famouswaffles 3y ago
    I swear, these kind of gotchas are the most stupid. Because they are easily resolved by just talking to the current state of the art. https://chat.openai.com/share/93324013-b333-44ef-b8c8-5460cbfb8477 https://chat.openai.com/share/93324013-b333-44ef-b8c8-5460cb...

YeGoblynQueenne 3y ago

>> The raw version of GPT apparently achieves an Elo score of 1800 at chess: https://twitter.com/GrantSlatton/status/1703913578036904431 https://twitter.com/GrantSlatton/status/1703913578036904431. This is worse than a good dedicated chess engine, but almost certainly better than I could do! Probably because it was trained to get better at it. https://twitter.com/GrantSlatton/status/1706545268009931175 https://twitter.com/GrantSlatton/status/1706545268009931175 There's already been plenty of discussion on LLMs' ability to play chess and it makes sense that OpenAI would try to make their models better at it.