3 ms·

LLMs clearly develop internal representations, this is an empirical fact. For example, one paper cited is https://arxiv.org/abs/2210.13382 https://arxiv.org/abs

LLMs clearly develop internal representations, this is an empirical fact. For example, one paper cited is https://arxiv.org/abs/2210.13382 https://arxiv.org/abs/2210.13382, which trained an LLM on Othello games, where an 8 layer model is fed one of 60 tokens (there are 60 possible moves in Othello, 8*8 minus 4 initially occupied squares). You can probe this model at layer 5, and it computes Othello board state from moves, and you can be sure this board state is used, because if you patch activation so that board state is modified, the model generates moves for this modified board state.

PaulDavisThe1st 4y ago
> LLMs clearly develop internal representations, this is an empirical fact. you've offered an anecdote. Some LLMs (more generally, this type of neural network model) will generate configurations that can be understood as a representation; others will not. The fact that the authors were able to find an apparent model in a heavily rule-based system is not incredibly surprising but offers little clue about whether this is "clearly" true for similar architectures that represent very different (real) world systems. From your citation itself: > More broadly, it would be interesting to study how our results generalize to models trained on natural language. One stepping stone might be to look at language models whose training data has included game transcripts. Will we see similar representation of board state?
- sanxiyn 4y ago
  Okay, I think we are in agreement. Let's say current LLM architectures are clearly capable of developing internal representations, not just learning surface statistics, and it can execute algorithmic computation as complex as computing Othello board states, and it can completely generalize out of distribution thanks to such algorithmic computation. (One experiment was to completely eliminate any Othello games starting with move C5 from the training data. The model had no trouble generating only legal moves for C5 games.) I agree whether LLMs trained on natural language do develop world model is an interesting and open empirical question. They clearly can, but we don't know whether they do. I do consider the evidence pretty suggestive that they do. But this is clearly just an empirical question, not a question that can be answered by arguments as useless as "they are statistical models", because statistical models can and do learn world model. I actually consider your considering this result not incredibly surprising pretty surprising.
  - PaulDavisThe1st 4y ago
    The othello example explicitly was trained on the rules of the game - it isn't suprising that you can find a representation of game state. LLMs are trained on the "rules" of human written language, which are sufficiently distinct from anything in the actual world that find world modelling would, indeed, be a surprise to me.

PaulDavisThe1st 4y ago

> LLMs clearly develop internal representations, this is an empirical fact. you've offered an anecdote. Some LLMs (more generally, this type of neural network model) will generate configurations that can be understood as a representation; others will not. The fact that the authors were able to find an apparent model in a heavily rule-based system is not incredibly surprising but offers little clue about whether this is "clearly" true for similar architectures that represent very different (real) world systems. From your citation itself: > More broadly, it would be interesting to study how our results generalize to models trained on natural language. One stepping stone might be to look at language models whose training data has included game transcripts. Will we see similar representation of board state?

sanxiyn 4y ago
Okay, I think we are in agreement. Let's say current LLM architectures are clearly capable of developing internal representations, not just learning surface statistics, and it can execute algorithmic computation as complex as computing Othello board states, and it can completely generalize out of distribution thanks to such algorithmic computation. (One experiment was to completely eliminate any Othello games starting with move C5 from the training data. The model had no trouble generating only legal moves for C5 games.) I agree whether LLMs trained on natural language do develop world model is an interesting and open empirical question. They clearly can, but we don't know whether they do. I do consider the evidence pretty suggestive that they do. But this is clearly just an empirical question, not a question that can be answered by arguments as useless as "they are statistical models", because statistical models can and do learn world model. I actually consider your considering this result not incredibly surprising pretty surprising.
- PaulDavisThe1st 4y ago
  The othello example explicitly was trained on the rules of the game - it isn't suprising that you can find a representation of game state. LLMs are trained on the "rules" of human written language, which are sufficiently distinct from anything in the actual world that find world modelling would, indeed, be a surprise to me.