5 ms·

> My current (tentative) resolution of the surprise is that language encoded way more information about reality than we thought it did. (Enough information that

> My current (tentative) resolution of the surprise is that language encoded way more information about reality than we thought it did. (Enough information that you can fully derive reality from language seems improbable, but iirc it did derive Othello and partly derived chess and I would have thought there wasn’t enough information in language to derive those without playing the games as well, so I can’t rule it out.) The information isn't in language itself, it's in language as actually used by humans. GPT4 knows about chess because it's "read" a significant fraction of everything we've ever written about chess. A human being who did that without ever playing a game would also start out better than a typical novice.

fwlr 3y ago
> A human being who did that without ever playing a game would also start out better than a typical novice. I am quite skeptical of these arguments along the lines of “imagine a human read everything written on the topic…”. What humans are doing when they read something is not what neural nets are doing when they read something. Humans are (idealistically) doing something like Feynman’s description of how he reads (or in this case, listens to) a theorem: “I had a scheme, which I still use today when somebody is explaining something that I’m trying to understand: I keep making up examples. For instance, the mathematicians would come in with a terrific theorem, and they’re all excited. As they’re telling me the conditions of the theorem, I construct something which fits all the conditions. You know, you have a set (one ball) – disjoint (two balls). Then the balls turn colors, grow hairs, or whatever, in my head as they put more conditions on. Finally they state the theorem, which is some dumb thing about the ball which isn’t true for my hairy green ball thing, so I say, ‘False!’"” Bret Victor’s description of what “really good programmers” are doing is also related: “[showing the code for binary search] In order to write code like this, you have to imagine an array in your head, and you essentially have to ‘play computer’. You have to simulate in your head what each line of code would do on a computer. And to a large extent those who we consider to be skilled software engineers are just those people who are really good at playing computer.” I think when we imagine an LLM as a human who’s read everything ever written in chess but never played an actual game, we’re actually tricking ourselves - because that hypothetical human would be ‘playing chess’ inside their head by imagining the pieces and moving them according to the rules they had read[1]. LLMs are not doing anything like that when they read about chess. So it’s a very restricted (or perhaps more accurately, a very different) kind of ‘reading’ that we don’t have any intuition for. Since the ‘reading’ that we do have an intuition for is smuggling in exactly the kind of “modeling the world” ability we’re looking for, it’s not surprising that this argument would incorrectly lead us to believe we’ve found it in LLMs. 1: In fact the very best computer chess is achieved by AlphaZero which was trained exclusively on “playing chess in its head”, and it beats even the most powerful and optimized search algorithms like Stockfish looking 20 moves ahead.
- anybodyz 3y ago
  What LLM's are doing when they imagine playing chess is what we do when we stand up after sitting on the floor, or what we do when we see a few million individual samples of color and light intensity and realize there's an apple and a knife in front of us. I think what is almost impossible for most people to understand is that AI's do not need to be structured like the human brain and use the crutches we use to solve problems the way we do because evolution did not provide us with a way of instantly understanding complex physics or instantly absorbing the structure of a computer program by seeing it's code in one shot.
  - groestl 3y ago
    Also, there is no reason to believe that playing chess in our head is anything else but us pattern matching a mental process on a higher level, recognizing a simulation there, and feeding that info back into the loop below. Nature provided us with a complex, layered and circular architecture of the brain, but the rest is pretty much training that structure. And we know that different architectures with similar outcome are possible, since there are vast variations across our own species, and other species as well, with essentially the same capabilities.
krainboltgreene 3y ago
> it's "read" a significant fraction of everything we've ever written about chess I doubt heavily that a significant fraction of chess's writings are even available in digital format, much less inside of CommonCrawl and correctly trained on.
- beardedwizard 3y ago
  Why would common crawl NOT contain a chess manual? the rules are explained in detail on wikipedia. the simplest conclusion is that it has indeed been trained on a chess manual and is good at predicting what the next word in a chess manual is. it is not synthesizing anything.
  - krainboltgreene 3y ago
    I never said that CommonCrawl doesn't have a chess manual, whatever the hell that might be. > the simplest conclusion is that it has indeed been trained on a chess manual That is an absurd conclusion that is based on literally fantasy.
  - galaxyLogic 3y ago
    I haven't seen this mentioned but can LLMs actually play chess? I'm sure they have read rules of chess online, but if you ask them to play chess with you, what happens? Can they apply the rules? Can they apply them intelligently and win the game? My point is that even though LLMs "know" what the rules of chess are, they don't really "understand" them, unless they can use them to play the game and play it well.
    - anon84873628 3y ago
      You win chess by "applying the rules intelligently" the same way you launch rockets to space by applying the laws of physics intelligently. True, yes, but not as simple as it sounds.
    - consilient 3y ago
      GPT4 can play chess, but not especially well. For example: https://lichess.org/@/oopsallbots-gpt-4 https://lichess.org/@/oopsallbots-gpt-4
wombatpm 3y ago
So if we gathered all books, writings, games, etc on ever published on chess could we develop a grandmaster player without having to build a deep blue type system?
- anon84873628 3y ago
  In that scenario, I believe a current LLM would "understand" chess. It would know how to play the game, it would know what being a grandmaster means, and it would know the heuristics and algorithms for evaluating a given game state. But it would not be able to implement an optimized solver to actually play the game efficiently. To do that would require it to also have all the knowledge about computer science. Hence the dream of singularity. Can you teach ChatGPT to build AlphaZero?

fwlr 3y ago

> A human being who did that without ever playing a game would also start out better than a typical novice. I am quite skeptical of these arguments along the lines of “imagine a human read everything written on the topic…”. What humans are doing when they read something is not what neural nets are doing when they read something. Humans are (idealistically) doing something like Feynman’s description of how he reads (or in this case, listens to) a theorem: “I had a scheme, which I still use today when somebody is explaining something that I’m trying to understand: I keep making up examples. For instance, the mathematicians would come in with a terrific theorem, and they’re all excited. As they’re telling me the conditions of the theorem, I construct something which fits all the conditions. You know, you have a set (one ball) – disjoint (two balls). Then the balls turn colors, grow hairs, or whatever, in my head as they put more conditions on. Finally they state the theorem, which is some dumb thing about the ball which isn’t true for my hairy green ball thing, so I say, ‘False!’"” Bret Victor’s description of what “really good programmers” are doing is also related: “[showing the code for binary search] In order to write code like this, you have to imagine an array in your head, and you essentially have to ‘play computer’. You have to simulate in your head what each line of code would do on a computer. And to a large extent those who we consider to be skilled software engineers are just those people who are really good at playing computer.” I think when we imagine an LLM as a human who’s read everything ever written in chess but never played an actual game, we’re actually tricking ourselves - because that hypothetical human would be ‘playing chess’ inside their head by imagining the pieces and moving them according to the rules they had read[1]. LLMs are not doing anything like that when they read about chess. So it’s a very restricted (or perhaps more accurately, a very different) kind of ‘reading’ that we don’t have any intuition for. Since the ‘reading’ that we do have an intuition for is smuggling in exactly the kind of “modeling the world” ability we’re looking for, it’s not surprising that this argument would incorrectly lead us to believe we’ve found it in LLMs. 1: In fact the very best computer chess is achieved by AlphaZero which was trained exclusively on “playing chess in its head”, and it beats even the most powerful and optimized search algorithms like Stockfish looking 20 moves ahead.

anybodyz 3y ago
What LLM's are doing when they imagine playing chess is what we do when we stand up after sitting on the floor, or what we do when we see a few million individual samples of color and light intensity and realize there's an apple and a knife in front of us. I think what is almost impossible for most people to understand is that AI's do not need to be structured like the human brain and use the crutches we use to solve problems the way we do because evolution did not provide us with a way of instantly understanding complex physics or instantly absorbing the structure of a computer program by seeing it's code in one shot.
- groestl 3y ago
  Also, there is no reason to believe that playing chess in our head is anything else but us pattern matching a mental process on a higher level, recognizing a simulation there, and feeding that info back into the loop below. Nature provided us with a complex, layered and circular architecture of the brain, but the rest is pretty much training that structure. And we know that different architectures with similar outcome are possible, since there are vast variations across our own species, and other species as well, with essentially the same capabilities.

krainboltgreene 3y ago

> it's "read" a significant fraction of everything we've ever written about chess I doubt heavily that a significant fraction of chess's writings are even available in digital format, much less inside of CommonCrawl and correctly trained on.

beardedwizard 3y ago
Why would common crawl NOT contain a chess manual? the rules are explained in detail on wikipedia. the simplest conclusion is that it has indeed been trained on a chess manual and is good at predicting what the next word in a chess manual is. it is not synthesizing anything.
- krainboltgreene 3y ago
  I never said that CommonCrawl doesn't have a chess manual, whatever the hell that might be. > the simplest conclusion is that it has indeed been trained on a chess manual That is an absurd conclusion that is based on literally fantasy.
- galaxyLogic 3y ago
  I haven't seen this mentioned but can LLMs actually play chess? I'm sure they have read rules of chess online, but if you ask them to play chess with you, what happens? Can they apply the rules? Can they apply them intelligently and win the game? My point is that even though LLMs "know" what the rules of chess are, they don't really "understand" them, unless they can use them to play the game and play it well.
  - anon84873628 3y ago
    You win chess by "applying the rules intelligently" the same way you launch rockets to space by applying the laws of physics intelligently. True, yes, but not as simple as it sounds.
  - consilient 3y ago
    GPT4 can play chess, but not especially well. For example: https://lichess.org/@/oopsallbots-gpt-4 https://lichess.org/@/oopsallbots-gpt-4

wombatpm 3y ago

So if we gathered all books, writings, games, etc on ever published on chess could we develop a grandmaster player without having to build a deep blue type system?

anon84873628 3y ago
In that scenario, I believe a current LLM would "understand" chess. It would know how to play the game, it would know what being a grandmaster means, and it would know the heuristics and algorithms for evaluating a given game state. But it would not be able to implement an optimized solver to actually play the game efficiently. To do that would require it to also have all the knowledge about computer science. Hence the dream of singularity. Can you teach ChatGPT to build AlphaZero?