3 ms·

That paper is famously misleading. It's all the same classic personification of LLMs. What an LLM can show is not the same as what it can do. The model was al

That paper is famously misleading. It's all the same classic personification of LLMs. What an LLM can show is not the same as what it can do. The model was already present: in the example game moves. The LLM modeled what it was given, and it was given none other than a valid series of Othello game states. Here's the problem with personification: A person who has modeled the game of Othello can use that model to strategize. An LLM cannot. An LLM can only take the whole model and repeat its parts with the most familiar patterns. It is stuck fuzzing around the strategies (or sections of strategy) it has been given. It cannot invent a new divergent strategy, even if the game rules require it to. It cannot choose the winning strategy unless that behavior is what was already recorded in the training corpus. An LLM does not play games, it plays plays.

archon1410 4y ago
> An LLM can only take the whole model and repeat its parts with the most familiar patterns. It is stuck fuzzing around the strategies (or sections of strategy) it has been given. It cannot invent a new divergent strategy, even if the game rules require it to. It cannot choose the winning strategy unless that behavior is what was already recorded in the training corpus. Where are you getting that from? My understanding is that you can get new, advanced, winning moves by starting a prompt with "total victory for the genius grandmaster player one who uses new and advanced winning techniques". If the model is capable and big enough, it'll give the correct completion by really inventing new strategies.
- Drew_ 4y ago
  Sounds like the type of prompt that would boldly give you a wrong/illegal answer.
  - archon1410 4y ago
    Perhaps. But the point is that some prompt will coax it into giving good answers that really make it win the game, if it has a good "world model" of how the game works. And there's no reason to think a language model cannot have such a world model. What exactly that prompt might be, the prompt engineers know best.
- thomastjeffery 4y ago
  It could give you a new strategy that is built from the parts of other known strategies. But would it give you the best one? Let's say the training corpus contains stories that compare example strategies. Each part of a strategy is explicitly weighed against another: one is called "superior". Now all you need is a prompt that asks for "a strategy containing all superior features". There are probably plenty of grammatical examples elsewhere in the model that make that transformation. All the work here is done by humans writing the training corpus. GPT never understood any of the steps. GPT just continued our story with the most obvious conclusion; and we made certain that conclusion would be correct. GPT doesn't play games, it plays plays.
  - archon1410 4y ago
    > GPT never understood any of the steps. GPT just continued our story with the most obvious conclusion; and we made certain that conclusion would be correct. Perhaps the earlier or current variations of GPT, for most games? But the idea that LLMs can never make anything novel, that it will never "generalise out of distribution" (if that's the correct term here) seems to be just an assertion, not backed by any theory with great evidence behind it. The "goal" of an LLM is to predict the next token. And the best way to do that is not brute force memorisation or regurgitating training data in various combinations, but to have a world model inside of it that will allow it to predict both the moves a bad player might make, and moves that a grandmaster might make.
    - thomastjeffery 4y ago
      > The "goal" of an LLM is to predict the next token That's another common misconception. That statement personifies GPT: GPT does not have goals or make predictions. Those are the effects of GPT: the behavior its authors hope will "emerge". None of that behavior comes from GPT itself. The behavior is defined by the patterns of tokens in the training corpus. GPT itself has two behaviors: modeling and presentation. GPT creates an implicit model of every pattern it can find between the tokens in its training corpus. It then expands that model to include the tokens of an arbitrary prompt. Finally, it presents the model to us by starting at the location it just added the prompt tokens to, and simply following the most obvious path forward until that path ends. The paths that GPT has available to present to us were already present in the training corpus. It isn't GPT that constructs the behavior, it is the people writing patterns into text. > not brute force memorisation or regurgitating training data in various combinations Not brute force: the combinations are not blindly assembled by GPT. GPT doesn't assemble combinations. The combinations were already assembled with patterns of grammar by the humans who wrote the valid progressions of game states. GPT found those patterns when it made its model. > to have a world model inside of it that will allow it to predict both the moves a bad player might make, and moves that a grandmaster might make. There is no prediction. A series of moves is a path carved into grammar. The path from one game state to the next involves several complex patterns that GPT has implicitly modeled. Depending on where GPT starts, the most obvious continuation may be to follow a more complex path. Even so, it's not GPT deciding where to go, it's the patterns that are already present that determine the path. Because we use the same grammatical/writing patterns to describe "good play" and "bad play", it's difficult to distinguish between the two. GPT alone can't categorize the skill level of games, but narrative surrounding those game examples potentially can.
fenomas 4y ago
Sorry, but what does anything you've said there have to do with the Othello paper? The point of that paper was that the AI was given nothing but sequences of move locations, and it nonetheless intuited the "world model" necessary to explain those locations. That is, it figured out that it needed to allocate 64 binary values and swap some of them after each move. The paper demonstrated that the AI was not just doing applied statistics on character strings - it had constructed a model to explain what the strings represented. "Strategy", meanwhile, has nothing to do with anything. The AI wasn't trained on competitive matches - it had no way of knowing that Othello has scoring, or even a win condition. It was simply trained to predict which moves are legal, not to strategize about anything.
- thomastjeffery 4y ago
  > The point of that paper was that the AI was given nothing but sequences of move locations, and it nonetheless intuited the "world model" necessary to explain those locations Yes... > That is, it figured out that it needed to allocate 64 binary values and swap some of them after each move. Yes, but "figured out" is misleading. It didn't invent or "figure out" the model. It discovered it, just like any other pattern it discovers. The pattern was already present in the example game. It was the "negative space" that the moves existed in. > "Strategy", meanwhile, has nothing to do with anything. The AI wasn't trained on competitive matches - it had no way of knowing that Othello has scoring, or even a win condition. It was simply trained to predict which moves are legal, not to strategize about anything. Yes, and that is critically important knowledge; yet dozens, if not hundreds, of comments here are missing that point. It found a model. That doesn't mean it can use the model. It can only repeat examples the of "uses" it has already seen. This is also the nature of the model itself: it was found by looking at the structural patterns of the example game. It was not magically constructed. > predict what moves are legal That looks like strategy, but it's still missing the point. We are the ones categorizing GPT's results as "legal". GPT never uses the word. It doesn't make that judgement anywhere. It just generates the continuation we told it to. What GPT was trained to do is emulate strategy. It modeled the example set of valid chronological game states. It can use that model to extrapolate any arbitrary valid game state into a hallucinated set of chronological game states. The model is so accurate that the hallucinated games usually follow the rules. Provided enough examples of edge cases, it could likely hallucinate a correct game every time; but that would still not be anything like a person playing the game intentionally. The more complete and exhaustive the example games are, the more "correctly" GPT's model will match the game rules. But even having a good model is not enough to generate novel strategy: GPT will repeat the moves it feels to be most familiar to a given game state. GPT does not play games, it plays plays.

archon1410 4y ago

> An LLM can only take the whole model and repeat its parts with the most familiar patterns. It is stuck fuzzing around the strategies (or sections of strategy) it has been given. It cannot invent a new divergent strategy, even if the game rules require it to. It cannot choose the winning strategy unless that behavior is what was already recorded in the training corpus. Where are you getting that from? My understanding is that you can get new, advanced, winning moves by starting a prompt with "total victory for the genius grandmaster player one who uses new and advanced winning techniques". If the model is capable and big enough, it'll give the correct completion by really inventing new strategies.

Drew_ 4y ago
Sounds like the type of prompt that would boldly give you a wrong/illegal answer.
- archon1410 4y ago
  Perhaps. But the point is that some prompt will coax it into giving good answers that really make it win the game, if it has a good "world model" of how the game works. And there's no reason to think a language model cannot have such a world model. What exactly that prompt might be, the prompt engineers know best.
thomastjeffery 4y ago
It could give you a new strategy that is built from the parts of other known strategies. But would it give you the best one? Let's say the training corpus contains stories that compare example strategies. Each part of a strategy is explicitly weighed against another: one is called "superior". Now all you need is a prompt that asks for "a strategy containing all superior features". There are probably plenty of grammatical examples elsewhere in the model that make that transformation. All the work here is done by humans writing the training corpus. GPT never understood any of the steps. GPT just continued our story with the most obvious conclusion; and we made certain that conclusion would be correct. GPT doesn't play games, it plays plays.
- archon1410 4y ago
  > GPT never understood any of the steps. GPT just continued our story with the most obvious conclusion; and we made certain that conclusion would be correct. Perhaps the earlier or current variations of GPT, for most games? But the idea that LLMs can never make anything novel, that it will never "generalise out of distribution" (if that's the correct term here) seems to be just an assertion, not backed by any theory with great evidence behind it. The "goal" of an LLM is to predict the next token. And the best way to do that is not brute force memorisation or regurgitating training data in various combinations, but to have a world model inside of it that will allow it to predict both the moves a bad player might make, and moves that a grandmaster might make.
  - thomastjeffery 4y ago
    > The "goal" of an LLM is to predict the next token That's another common misconception. That statement personifies GPT: GPT does not have goals or make predictions. Those are the effects of GPT: the behavior its authors hope will "emerge". None of that behavior comes from GPT itself. The behavior is defined by the patterns of tokens in the training corpus. GPT itself has two behaviors: modeling and presentation. GPT creates an implicit model of every pattern it can find between the tokens in its training corpus. It then expands that model to include the tokens of an arbitrary prompt. Finally, it presents the model to us by starting at the location it just added the prompt tokens to, and simply following the most obvious path forward until that path ends. The paths that GPT has available to present to us were already present in the training corpus. It isn't GPT that constructs the behavior, it is the people writing patterns into text. > not brute force memorisation or regurgitating training data in various combinations Not brute force: the combinations are not blindly assembled by GPT. GPT doesn't assemble combinations. The combinations were already assembled with patterns of grammar by the humans who wrote the valid progressions of game states. GPT found those patterns when it made its model. > to have a world model inside of it that will allow it to predict both the moves a bad player might make, and moves that a grandmaster might make. There is no prediction. A series of moves is a path carved into grammar. The path from one game state to the next involves several complex patterns that GPT has implicitly modeled. Depending on where GPT starts, the most obvious continuation may be to follow a more complex path. Even so, it's not GPT deciding where to go, it's the patterns that are already present that determine the path. Because we use the same grammatical/writing patterns to describe "good play" and "bad play", it's difficult to distinguish between the two. GPT alone can't categorize the skill level of games, but narrative surrounding those game examples potentially can.

fenomas 4y ago

Sorry, but what does anything you've said there have to do with the Othello paper? The point of that paper was that the AI was given nothing but sequences of move locations, and it nonetheless intuited the "world model" necessary to explain those locations. That is, it figured out that it needed to allocate 64 binary values and swap some of them after each move. The paper demonstrated that the AI was not just doing applied statistics on character strings - it had constructed a model to explain what the strings represented. "Strategy", meanwhile, has nothing to do with anything. The AI wasn't trained on competitive matches - it had no way of knowing that Othello has scoring, or even a win condition. It was simply trained to predict which moves are legal, not to strategize about anything.

thomastjeffery 4y ago
> The point of that paper was that the AI was given nothing but sequences of move locations, and it nonetheless intuited the "world model" necessary to explain those locations Yes... > That is, it figured out that it needed to allocate 64 binary values and swap some of them after each move. Yes, but "figured out" is misleading. It didn't invent or "figure out" the model. It discovered it, just like any other pattern it discovers. The pattern was already present in the example game. It was the "negative space" that the moves existed in. > "Strategy", meanwhile, has nothing to do with anything. The AI wasn't trained on competitive matches - it had no way of knowing that Othello has scoring, or even a win condition. It was simply trained to predict which moves are legal, not to strategize about anything. Yes, and that is critically important knowledge; yet dozens, if not hundreds, of comments here are missing that point. It found a model. That doesn't mean it can use the model. It can only repeat examples the of "uses" it has already seen. This is also the nature of the model itself: it was found by looking at the structural patterns of the example game. It was not magically constructed. > predict what moves are legal That looks like strategy, but it's still missing the point. We are the ones categorizing GPT's results as "legal". GPT never uses the word. It doesn't make that judgement anywhere. It just generates the continuation we told it to. What GPT was trained to do is emulate strategy. It modeled the example set of valid chronological game states. It can use that model to extrapolate any arbitrary valid game state into a hallucinated set of chronological game states. The model is so accurate that the hallucinated games usually follow the rules. Provided enough examples of edge cases, it could likely hallucinate a correct game every time; but that would still not be anything like a person playing the game intentionally. The more complete and exhaustive the example games are, the more "correctly" GPT's model will match the game rules. But even having a good model is not enough to generate novel strategy: GPT will repeat the moves it feels to be most familiar to a given game state. GPT does not play games, it plays plays.