4 ms·

LLMs aren't simply modeling frequency distributions. Self-attention essentially searches the space of circuits to find which circuits help to model the training

LLMs aren't simply modeling frequency distributions. Self-attention essentially searches the space of circuits to find which circuits help to model the training data. This search process recovers the internal structure of the training data that isn't captured by naive frequency distribution models. The limit of arbitrarily complex frequency distribution models, i.e. P(xN|x1...x(N-1)) for large N is just memorizing the complete training data, which we know LLMs aren't doing due to space limitations. The abilities of LLMs aren't well explained by parroting or modeling frequency distributions.

mjburgess 3y ago
It's necessarily modelling freq. distribution; I'm not sure how you could think it's doing anything else. Self-attention is just a frequency distribution over a freq distribution. It's literally trained with P(B|A) as the objective.... The 'circuits' you're talking about are just sequences of `|`, ie., P(B|A1..|...n))
- hackinthebochs 3y ago
  >Self-attention is just a frequency distribution over a freq distribution. I don't know where you get that. But it's not my understanding. >It's literally trained with P(B|A) as the objective.... This is a description of the objective, not a model or an algorithm. The algorithm is not learning frequency data. The algorithm tries to maximize P(B|A), but within this constraint there are a vast range of algorithms.
  - AstralStorm 3y ago
    But can rhis algorithm ever produce reasoning without learning a whole universe of possible inputs? Given the evidence that it fails to learn arithmetic, skips inference steps, misassigns symbols, I'd say likely not.
    - hackinthebochs 3y ago
      Reasoning is abstracted from particulars. So in principle what it needs to learn is a finite set of rules. There are good reasons that explain why current LLMs don't learn arithmetic and has odd failure modes: it's processing is feed-forward (non-recursive) with a fixed computational budget. This means that it in principle cannot learn general rules for arithmetic which involve unbounded carrying. But this is not an in principle limitation for LLMs or gradient descent based ML in general.

mjburgess 3y ago

It's necessarily modelling freq. distribution; I'm not sure how you could think it's doing anything else. Self-attention is just a frequency distribution over a freq distribution. It's literally trained with P(B|A) as the objective.... The 'circuits' you're talking about are just sequences of `|`, ie., P(B|A1..|...n))

hackinthebochs 3y ago
>Self-attention is just a frequency distribution over a freq distribution. I don't know where you get that. But it's not my understanding. >It's literally trained with P(B|A) as the objective.... This is a description of the objective, not a model or an algorithm. The algorithm is not learning frequency data. The algorithm tries to maximize P(B|A), but within this constraint there are a vast range of algorithms.
- AstralStorm 3y ago
  But can rhis algorithm ever produce reasoning without learning a whole universe of possible inputs? Given the evidence that it fails to learn arithmetic, skips inference steps, misassigns symbols, I'd say likely not.
  - hackinthebochs 3y ago
    Reasoning is abstracted from particulars. So in principle what it needs to learn is a finite set of rules. There are good reasons that explain why current LLMs don't learn arithmetic and has odd failure modes: it's processing is feed-forward (non-recursive) with a fixed computational budget. This means that it in principle cannot learn general rules for arithmetic which involve unbounded carrying. But this is not an in principle limitation for LLMs or gradient descent based ML in general.