3 ms·

> The Othello paper is hardly a counter example. Researchers created an Othello specific model that almost learned the grammar of Othello not how to play well.

> The Othello paper is hardly a counter example. Researchers created an Othello specific model that almost learned the grammar of Othello not how to play well. Yes, there was largely correct internal game state built up from past moves. No it didn’t actually learn the rules so it would make strictly legal moves nor did it learn to make good moves It is, though. Nobody said anything about playing well, or learning the rules. The very fact that it had a valid internal representation of the game state means it's extrapolated beyond token-level. Which is the point.

Retric 3y ago
> The very fact that it had a valid internal representation of the game state means it's extrapolated beyond token-level. Which is the point. The paper said it was making incorrect moves thus it has an invalid representation of the game. So an LLM when specifically trained on Othello, a game with very simple and completely mechanical rules, failed to abstract what those rules actually where. This means at a purely mechanical level it doesn’t understand the game when that was exclusively what it was trained to do. It’s a clear illustration that these things are really really bad at abstraction. But that should be obvious because they are simply manipulating arbitrary tokens from their perspective. It doesn’t intuit that the game should have simple rules and therefore it doesn’t find them. People on the other hand have a real bias regarding simple rules.
- regularfry 3y ago
  Except that incorrect moves don't imply an incorrect representation. The internal representation was a literal 8x8 grid of piece locations they could externally change and have it generate moves consistent with the changed position. It's about the clearest example of a learned higher-level internal representation I can think I've seen. The fact that it didn't also perfectly learn the rules while it was doing that is entirely uninteresting.

Retric 3y ago

> The very fact that it had a valid internal representation of the game state means it's extrapolated beyond token-level. Which is the point. The paper said it was making incorrect moves thus it has an invalid representation of the game. So an LLM when specifically trained on Othello, a game with very simple and completely mechanical rules, failed to abstract what those rules actually where. This means at a purely mechanical level it doesn’t understand the game when that was exclusively what it was trained to do. It’s a clear illustration that these things are really really bad at abstraction. But that should be obvious because they are simply manipulating arbitrary tokens from their perspective. It doesn’t intuit that the game should have simple rules and therefore it doesn’t find them. People on the other hand have a real bias regarding simple rules.

regularfry 3y ago
Except that incorrect moves don't imply an incorrect representation. The internal representation was a literal 8x8 grid of piece locations they could externally change and have it generate moves consistent with the changed position. It's about the clearest example of a learned higher-level internal representation I can think I've seen. The fact that it didn't also perfectly learn the rules while it was doing that is entirely uninteresting.